What Does Inference Mean?
Inference is the act of running a trained AI model to produce an output. Training is when a model learns from data, which happens once and is very expensive. Inference is every time someone uses it afterwards: each chat reply, each autocomplete suggestion and each step of an agent is an inference call. It runs on servers with specialized chips, such as GPUs, or on your own machine for smaller models. API providers charge for inference by the token, usually with separate prices for input tokens, the text you send, and output tokens, the text the model writes.
Why Does Inference Matter?
Inference is where the ongoing cost of using AI sits. Agent workflows make many inference calls per task, each re-sending the growing context, so costs add up faster than with single chat replies. Choosing a smaller model for simple steps, keeping context short and using prompt caching are the main ways to keep inference spend reasonable. Speed matters too: slow inference makes every agent turn feel sluggish.
What Does Inference Look Like in Practice?
An agent fixing a bug takes 12 turns. Each turn sends about 30,000 tokens of context and gets back about 800 tokens. That is roughly 360,000 input tokens and 10,000 output tokens of inference for one small fix. Run locally on an open-weight model, the cost is your hardware and electricity instead of a per-token bill. With prompt caching, most of those repeated input tokens are billed at a discount.
What Do People Get Wrong About Inference?
Running a model does not teach it anything. Inference uses the model as it is; your conversation does not update its weights. Any memory between sessions comes from the application saving and re-sending information.
Which Terms Are Related?
- Token: A small chunk of text, often part of a word, that models process; limits and prices are counted in tokens.
- Quantization: Storing model weights with fewer bits so a model needs less memory and runs faster, with some loss of quality.
- Prompt caching: Reusing the already processed start of a prompt across requests so repeated context is cheaper and faster.
- Open-weight model: A model whose trained weights you can download and run on your own hardware, under the terms of its license.
- Frontier model: One of the most capable AI models at a given moment, typically closed and served by a large lab through an API.
All 28 terms are in the AI glossary. For models side by side, see frontier vs open-weight.
Frequently Asked Questions
Why are output tokens more expensive than input tokens?
Input can be processed in parallel, but output is generated one token at a time, each step running the model again. That uses more compute per token, so providers usually charge several times more for output.
Can I run inference on my own computer?
Yes, for open-weight models that fit your hardware. Small and quantized models run on recent laptops; larger ones need a strong GPU or several. The very largest models still need server hardware.
Does inference send my data to the provider?
With an API or app, yes: your input is processed on their servers under their data policy. Running an open-weight model locally keeps the data on your machine.