VRAM and what fits

Whether a model runs on your machine is not about how fast the machine is. It is about one number: how much memory the graphics hardware can use. On a PC that is the graphics card's VRAM; on a Mac or a phone it is a share of the machine's unified memory. The model has to fit in it, whole, before a single word comes out.

Why "fit" is binary. A model is a large file of numbers, and to run it, essentially all of those numbers must be loaded at once. A model that needs 20 GB does not run slowly on a 16 GB card — it doesn't run, or it spills over into ordinary RAM (offloading), which is so much slower that it usually stops being worth using. This is why the catalogue gives fit verdicts rather than scores.

Why the same model has several sizes. Downloads come at different quantization levels — the same model stored more or less coarsely. When a page here says "VRAM @ Q4", it means: the memory needed at the most common quality-versus-size compromise, the one most people actually download. A model that misses at full precision often fits comfortably at Q4.

The file is not the whole bill. While the model works, it keeps notes on the conversation so far — the KV cache — and that grows with context length. A long chat or a big pasted document can add gigabytes on top of the file. The system also keeps some memory for itself. So the honest rule is: the model file, plus room for the context you'll actually use, plus headroom — a model that "fits" with nothing to spare doesn't fit.

After fit, bandwidth. Once a model fits, how fast the words arrive is mostly memory bandwidth — every generated token requires reading the whole model once. Two machines that both fit a model can differ several-fold in speed. Fit decides whether; bandwidth decides how fast — the fit checker estimates both for your machine.

Where next: Check your machine · What is quantization · What is a context window