QUANTIZATION

Storing model weights with fewer bits so a model needs less memory and runs faster, with some loss of quality.

Zero retention: everything you type stays in this browser. Verify it yourself in DevTools.
Plain-language definition

What Does Quantization Mean?

Quantization is a technique for shrinking an AI model by storing its weights with less precision. Models are usually trained with 16-bit numbers; quantization rounds them to 8, 4 or even fewer bits. A 4-bit version needs roughly a quarter of the memory of the 16-bit original and often runs faster, which is how large open-weight models fit on laptops and single GPUs. The cost is some loss of quality, which is usually small at 8 bits, often acceptable at 4 bits, and more noticeable below that. File formats such as GGUF and settings such as Q4 or Q8 in model names refer to quantization.

Why Does Quantization Matter?

If you run models yourself, quantization decides which models fit your hardware and how fast they go. It also explains why the same model name can behave differently from one host to another: a provider serving a heavily quantized version may give slightly worse answers than one serving full precision. When comparing hosts or local setups, check which quantization each one uses before blaming the model.

What Does Quantization Look Like in Practice?

A 30-billion-parameter model needs about 60 GB of memory at 16 bits, too much for most laptops. Quantized to 4 bits, it needs roughly 15 to 20 GB including overhead, which fits on a machine with 32 GB of RAM. Its answers are a little less precise on hard tasks but close enough for everyday coding help. An 8-bit version, at roughly 32 GB, would be closer to the original but no longer fit.

What Do People Get Wrong About Quantization?

Quantization is not the same as distillation. Quantizing keeps the same model with rounded numbers. Distillation trains a different, smaller model. Both make models cheaper to run, in different ways.

Which Terms Are Related?

  • Open-weight model: A model whose trained weights you can download and run on your own hardware, under the terms of its license.
  • Inference: Running an already trained model to produce output. Every reply you get from an AI is one inference call.
  • Distillation: Training a small model to imitate a larger model, so you get much of the quality at a fraction of the cost.
  • Mixture of experts: A model design that activates only a few specialist sub-networks per token, making huge models cheaper to run.

All 28 terms are in the AI glossary. For models side by side, see frontier vs open-weight.

Frequently Asked Questions

How much quality does 4-bit quantization lose?

It varies by model and method, but for many models 4-bit versions perform close to the original on everyday tasks, with larger drops on precise reasoning, math and long contexts. Test on your own tasks.

What do Q4 and Q8 mean in model file names?

They indicate roughly how many bits each weight uses after quantization, 4 or 8. Extra letters after them usually name the specific quantization method or variant.

Do API providers quantize models?

Some do, to save cost, and they do not always say so. If quality on the same model differs between hosts, quantization is a likely reason. Check the host documentation.

Was this helpful?