What is quantization

A model is billions of numbers, and quantization stores each number more coarsely so the whole file shrinks. The original weights use sixteen bits per number; a quantized copy might use four or five. The result is a model around a third of the size, at a small and usually unnoticeable cost in answer quality — which is why nearly everyone running models at home runs them quantized.

The names look like line noise but decode simply. Q4_K_M means roughly four bits per number, in a particular packing — the specific quant most people download, and the level our fit verdicts assume, because it is the sweet spot where the savings are large and the quality loss is small. Q8 keeps more precision at twice the size; Q2 and Q3 shrink further but the damage starts to show. GGUF is just the file format these come in.

What it means in practice: whether a model fits your machine is mostly a question of which quant. A 24-billion-parameter model is out of reach for a 16 GB card in its original precision, and comfortable at Q4. Every model page here shows the memory each quant needs against your hardware — and marks which sizes are measured from a real file and which are our own estimates, because the two deserve different trust.

The trade is real but lopsided: for chat and everyday work, Q4-class quants are hard to tell from the original. Precision matters more at the extremes — heavy maths, long chains of code edits — where stepping up to Q5 or Q6 is cheap insurance if the memory is there.

Where next: What is a context window · Check a model against your machine