What is inference

Training a model is done once, by a lab, on hardware you will never own. Inference is everything after that: taking the finished model and running it — every chat message, every transcription, every generated image is an act of inference. This whole site is about inference, which is why the choices on it are yours to make rather than a lab's.

Inference has a cost, and it is paid in one of two currencies. Rent it, and you pay per use — most text models are priced per million tokens, roughly word-pieces, with a rate for what you send in and a rate for what comes back. Run it yourself, and you pay in hardware instead: the model has to fit in your machine's memory, and how fast it answers depends on how quickly that memory can be read.

Two numbers describe how inference feels in practice. Throughput is how many tokens arrive per second once the answer is flowing — below ten feels like waiting, above fifty feels instant. Latency is the pause before the first word appears. Hosted providers publish the first; the second depends mostly on who else is using the machine at that moment.

The practical consequence: the same model can be a different product depending on where it runs. A 20-billion-parameter model rented from a fast host is an instant assistant; the same weights on a laptop with too little memory are a slideshow. That is why this site pairs every model with hardware fit verdicts and live provider prices — the model alone tells you almost nothing about what using it will be like.

Where next: What is meant by providers · When to consider running models locally