Mixture of experts: active parameters versus total parameters

01The layer

A Transformer block has two halves: attention, which moves information between tokens, and a feed-forward network, which transforms each token on its own. In a dense model every token passes through the same feed-forward network, and that network is where most of the parameters live, typically two thirds of the block.

dense:   output = FFN(x)

A mixture-of-experts layer keeps attention as it is and replaces the single feed-forward network with N of them, the experts, plus a router that decides which ones a token visits.

                       ┌──► expert 1 (FFN1) ──┐
                       ├──► expert 2 (FFN2) ──┤
token x ──► router G(x)  ├──► expert 3 (FFN3) ──┼──► weighted sum ──► output
                       ├──►    …             ──┤
                       └──► expert N (FFNN) ──┘

only the top-k arrows are actually computed for a given token

output = ∑_{i ∈ top-k} G(x)_i · FFN_i(x)

The experts have the same shape as the feed-forward network they replace. "8×7B" means a 7B-sized architecture whose feed-forward block has been copied into eight experts; the attention weights, embeddings and norms are shared and exist once.

02The router

The router is small: a linear layer that maps the token vector to one score per expert. The top k scores are kept, usually k = 1 or 2, and normalised with a softmax so the chosen experts' outputs can be combined as a weighted sum. Whether the softmax runs before or after the top-k cut varies by model; Mixtral applies it to the top-2 scores only.

Two things about this are easy to miss. The decision is per token and per layer, so one token may visit experts 3 and 7 in layer 4 and experts 1 and 5 in layer 5, and the next token makes its own choices. And the router is learned with everything else, by gradient, through the weights it assigns to the chosen experts. Nobody tells it that expert 3 is for code; whatever specialisation emerges comes from training.

03Total and active parameters

• Total parameters are every weight in the model: all N experts in every layer, plus the shared attention, embeddings and routers.
• Active parameters are the weights a single token actually passes through: the shared parts plus k experts per layer.

Mixtral is the clean example. Eight 7B-shaped experts would be 56B if everything were copied, but only the feed-forward blocks are, so the total is 47B. A token uses 2 of the 8 experts plus the shared weights, which is 13B. Newer designs push the ratio much further, with hundreds of small experts and a few percent active per token, and some add a shared expert that every token visits so that common knowledge is not duplicated across routed experts.

04Why memory follows the total

The router decides per token, at run time, and any expert can be chosen. So every expert has to be resident before the first token arrives. There is no way to load the right 13B on demand; the right 13B is different for every token in every layer.

Mixtral 8×7B therefore needs about 93 GB in 16-bit, the same as a dense 47B model, and it needs it even if most experts sit idle for a given request. Add the KV cache, which is paid per request and depends on the attention design rather than the experts. Add, for the larger models, the fact that 671B parameters do not fit on one device and the experts have to be spread across GPUs, which brings communication into every layer.

05Why compute follows the active count

A forward pass costs roughly two floating-point operations per parameter per token, one multiply and one add. The parameters a token does not pass through cost nothing. Mixtral's token passes through 13B, so it costs the arithmetic of a 13B dense model, about 26 GFLOPs, not the 94 GFLOPs a dense 47B model would spend.

This is the bargain. Capacity, in the sense of how much the weights can store, scales with the total. Cost per token scales with the active count. A mixture of experts gets the knowledge of a large model at the training and inference arithmetic of a small one, and in training it is the arithmetic that sets the budget: for a fixed compute budget, a sparse model reaches a lower loss than a dense one.

06Where the bargain weakens: batching and bandwidth

The outline usually stops at "less compute, faster generation". It is worth being precise, because decode is bound by memory bandwidth, not arithmetic, and the two scale differently with the batch.

• At batch size 1, a decode step reads only the weights it uses: the shared parts and the 2 chosen experts per layer. Bytes streamed scale with the active count, so the step is about as fast as a 13B dense model's. Both the FLOPs saving and the bandwidth saving hold.
• At a large batch, the tokens in the batch between them choose every expert in every layer, so the whole 47B streams from memory on every step. Bytes per step now match a dense 47B model. The FLOPs per token are still those of a 13B model, so the arithmetic saving holds, but the bandwidth saving is gone, and bandwidth is the bottleneck.

So at high throughput a mixture of experts behaves like a large model on memory traffic and a small model on arithmetic. It is still the better trade, because the arithmetic saving lets the server batch much deeper before it becomes compute-bound, but "faster generation" is a batch-size-dependent claim.

Spreading experts across devices adds a second cost. Each token's hidden state has to travel to the GPUs holding its chosen experts and back, an all-to-all exchange in every MoE layer. The routing therefore depends on the whole batch, which is one of the reasons greedy decoding is not reproducible on these models.

07What routing costs in training

A learned router has a failure mode built in. Early in training a few experts are slightly better, so the router sends them more tokens, so they get more gradient and improve further, so they are chosen even more. Left alone, the model collapses onto a handful of experts and the rest never learn; the total parameter count is then a fiction.

• Load-balancing loss. An auxiliary term that penalises uneven expert usage across a batch, pushing the router toward spreading tokens. It works, and it fights the main objective slightly. DeepSeek-V3 replaces it with a per-expert bias adjusted outside the gradient.
• Expert capacity. A cap on how many tokens an expert may take per batch, so that one overloaded expert does not stall the step. Tokens beyond the cap are dropped from that layer, passing through the residual connection unchanged, or rerouted.
• Router noise and instability. Top-k is not differentiable; the router only gets gradient through the weights of the chosen experts. Training is less stable than dense, and small routers in large models have needed care, such as computing router logits in higher precision.
• Fine-tuning is harder. With little data, the router overfits and expert usage skews. Many fine-tuning recipes freeze the router.

08Dense versus mixture of experts

The last row is the honest summary. A 47B mixture is not as good as a 47B dense model; it is better than a 13B dense model and costs about as much to run. Whether that is the right trade depends on whether you are short of memory or short of compute, and for most training runs and most high-throughput serving, compute is the scarcer one.

09Takeaways

1. An MoE layer is several feed-forward networks and a router. Attention is shared and unchanged.
2. Total parameters set capacity and memory. Every expert must be resident, because any token may choose it.
3. Active parameters set arithmetic. A token pays only for the experts it visits.
4. The bandwidth saving depends on batch size. At batch 1 only the chosen experts stream; at high batch the whole model does.
5. Routing is learned and prone to collapse. Load-balancing losses, capacity limits and careful fine-tuning are the price.
6. Read "8×7B" as 47B of knowledge at 13B of cost. Not as eight 7B models, and not as a 56B model.