Honey, I shrunk the LLM! A beginner’s guide to quantization – and testing it

Just be careful not to shave off too many bits … These things are known to hallucinate as it is

Hands on  If you hop on Hugging Face and start browsing through large language models, you’ll quickly notice a trend: Most have been trained at 16-bit floating point of Brain-float precision. …