What is a mixture of experts?
How a model can have hundreds of billions of parameters and still answer quickly: it only wakes up a few of them for each token.
- → A mixture-of-experts model splits part of the network into many smaller experts and uses only a few of them per token.
- → That is why such models list two sizes: total parameters and active parameters.
- → Most open-weights models now use it, because it gives large-model knowledge at small-model speed.
Model names come with parameter counts, and lately they come with two: a large total and a much smaller number labeled active. gpt-oss-120b, for example, has about 117 billion parameters in total but uses only about 5 billion of them for any given token. That is a mixture-of-experts model, usually shortened to MoE.
The idea is simple. Instead of running every parameter for every token, the model keeps many specialized sub-networks, the experts, and a small router decides which few to consult each time. The rest sit idle for that step.
Dense versus sparse
A dense model uses all of its parameters on every token. Doubling its size roughly doubles the compute per token, which is why big dense models are slow and expensive to serve. A mixture-of-experts model is sparse: it stores far more parameters than it uses at once. Compute per token tracks the active count, not the total.
What an expert actually is
Inside a transformer, each layer has a block that does most of the heavy lifting on a token. In an MoE model that block is replaced by several copies, the experts, plus a router. For each token the router scores the experts and sends the token to the top one or few. The experts are not labeled by subject. They specialize on their own during training, in ways that rarely map neatly onto human categories.
Why total and active both matter
The total count tells you how much the model can know: all of those parameters were trained and all of them hold something. The active count tells you how much work it does per token, so it is the better guide to speed and serving cost. When you compare an MoE model to a dense one, compare on both numbers, or the comparison misleads.
The catch: memory
Idle parameters still have to live somewhere. Every expert must be loaded into GPU memory so the router can reach it, so an MoE model needs the memory of its total size even though it runs at the speed of its active size. This is why these models are cheap to serve at scale on big hardware and awkward to run on a laptop.
Why open-weights labs like it
Training and serving cost is where open-weights labs have to be efficient, and MoE stretches a compute budget further than a dense design. DeepSeek, Kimi, GLM, and Qwen all use it. It is a large part of how those models close the gap with far more expensive closed ones.
A large hospital has hundreds of specialists on staff, but a patient sees two or three of them. The building has to house everyone, which is the memory cost. Each visit is fast, which is the active cost. And the front desk, the router, has to send people to the right doors, which is what training teaches it.
Every model Berges AI runs is a mixture-of-experts model. It is a big part of why we can serve large open-weights reasoning models at everyday prices, and why even the largest ones answer at a readable pace.
Try Berges AIRelated concepts
Things people ask
Is a 120B MoE model as good as a 120B dense model?
Usually not quite, on capability per parameter, but it is far cheaper to run. The fairer comparison is with a dense model somewhere between its active and total size. In practice MoE wins on the trade-off, which is why it has spread.
Does the model pick one expert for a whole answer?
No. Routing happens per token and per layer, so a single sentence passes through many different experts. There is no "math expert" answering your math question.
Can I run an MoE model on my own computer?
Only if you have the memory for the total size. Speed is fine, since only the active parameters do work, but every expert has to be loaded. That makes MoE models better suited to hosted serving than to a single consumer machine.
Is Qwen 3.8 Max a mixture of experts?
Yes. It has about 2.4 trillion parameters in total and about 95 billion active per token. The total is why it knows so much; the active count is why each reply costs more units on Berges AI than one from a smaller model.