Is a mixture-of-experts model better than a dense model?
Not automatically. MoE gives more total capacity for a given compute cost, which often helps, but quality depends on training. Compare models on your own tasks rather than by architecture.
A model design that activates only a few specialist sub-networks per token, making huge models cheaper to run.
Mixture of experts, or MoE, is a way of building a large AI model out of many smaller sub-networks, called experts, plus a router that picks which few experts handle each token. A model might have hundreds of billions of parameters in total but use only a small fraction of them for any single token. That makes it much cheaper and faster to run than a model of the same total size where every parameter works on every token. The experts are not neatly specialized by topic the way the name suggests; the routing is learned and often hard to interpret.
Many current open-weight and frontier models use MoE, which is why model names sometimes list two sizes, such as total and active parameters. For anyone running models locally, it matters because the whole model still has to fit in memory even though only part of it computes each token. So an MoE model can be fast but still need a lot of RAM or GPU memory.
A model described as "35B-A3B" has about 35 billion parameters in total, with about 3 billion active per token. It runs roughly at the speed of a 3-billion-parameter model but needs memory for all 35 billion, and its quality usually lands somewhere between a small and a large dense model. Quantizing it to 4 bits can bring that memory need down a lot further.
Experts are not separate specialists like a coding expert and a history expert. The router learns its own split, which usually does not line up with human topics, and many experts handle overlapping work.
All 28 terms are in the AI glossary. For models side by side, see frontier vs open-weight.
Not automatically. MoE gives more total capacity for a given compute cost, which often helps, but quality depends on training. Compare models on your own tasks rather than by architecture.
All experts must be loaded because any of them might be chosen for the next token. Only the computation is reduced, not the storage, so memory needs follow the total parameter count.
Model cards usually say so and often list total and active parameter counts, sometimes in the name, such as A3B or 128E for the number of experts.