MIXTURE OF EXPERTS

A model design that activates only a few specialist sub-networks per token, making huge models cheaper to run.

Zero retention: everything you type stays in this browser. Verify it yourself in DevTools.
Plain-language definition

What Does Mixture Of Experts Mean?

Mixture of experts, or MoE, is a way of building a large AI model out of many smaller sub-networks, called experts, plus a router that picks which few experts handle each token. A model might have hundreds of billions of parameters in total but use only a small fraction of them for any single token. That makes it much cheaper and faster to run than a model of the same total size where every parameter works on every token. The experts are not neatly specialized by topic the way the name suggests; the routing is learned and often hard to interpret.

Why Does Mixture Of Experts Matter?

Many current open-weight and frontier models use MoE, which is why model names sometimes list two sizes, such as total and active parameters. For anyone running models locally, it matters because the whole model still has to fit in memory even though only part of it computes each token. So an MoE model can be fast but still need a lot of RAM or GPU memory.

What Does Mixture Of Experts Look Like in Practice?

A model described as "35B-A3B" has about 35 billion parameters in total, with about 3 billion active per token. It runs roughly at the speed of a 3-billion-parameter model but needs memory for all 35 billion, and its quality usually lands somewhere between a small and a large dense model. Quantizing it to 4 bits can bring that memory need down a lot further.

What Do People Get Wrong About Mixture Of Experts?

Experts are not separate specialists like a coding expert and a history expert. The router learns its own split, which usually does not line up with human topics, and many experts handle overlapping work.

Which Terms Are Related?

  • Open-weight model: A model whose trained weights you can download and run on your own hardware, under the terms of its license.
  • Inference: Running an already trained model to produce output. Every reply you get from an AI is one inference call.
  • Quantization: Storing model weights with fewer bits so a model needs less memory and runs faster, with some loss of quality.
  • Large language model: An AI model trained on vast amounts of text to predict the next token, which lets it write, answer questions and code.

All 28 terms are in the AI glossary. For models side by side, see frontier vs open-weight.

Frequently Asked Questions

Is a mixture-of-experts model better than a dense model?

Not automatically. MoE gives more total capacity for a given compute cost, which often helps, but quality depends on training. Compare models on your own tasks rather than by architecture.

Why do MoE models need so much memory?

All experts must be loaded because any of them might be chosen for the next token. Only the computation is reduced, not the storage, so memory needs follow the total parameter count.

How can I tell if a model uses mixture of experts?

Model cards usually say so and often list total and active parameter counts, sometimes in the name, such as A3B or 128E for the number of experts.

Was this helpful?