Mixture of Experts (MoE) is a neural network design in which a model is split into many specialized sub-networks, called experts, and a small router (or gating network) decides which experts process each piece of input. Instead of running every parameter for every token, an MoE model activates only a few experts at a time. The result is a model with a very large total number of parameters but a much smaller amount of compute per token.

Why it matters now: several widely used open-weight and frontier large language models rely on MoE to grow their capacity without growing inference cost at the same rate. If you build AI applications, MoE explains why a model can be described with two parameter counts, why it may need lots of memory even though it runs fast, and why some models are cheaper per token than their size suggests. Hugging Face’s explainer sums up the trade-off: MoE models pretrain faster and infer faster than dense models of the same total size, but need high memory because all experts must be loaded (Hugging Face: Mixture of Experts Explained).

How Mixture of Experts works

Dense vs. sparse models

In a standard dense transformer, every token passes through every layer and every weight. In an MoE transformer, some layers (typically the feed-forward blocks) are replaced by a set of experts. Each token is sent to only a few of them, so the model is sparse: most parameters sit idle for any given token.

The router

For every token, the router scores all experts and picks the top-k, often one or two. Their outputs are combined using the router’s weights. This idea was scaled up in the sparsely gated MoE layer introduced by Shazeer and colleagues, which used thousands of experts to increase model capacity with only minor losses in computational efficiency (Shazeer et al., Outrageously Large Neural Networks).

Total vs. active parameters

A well-known example is Mixtral 8x7B: each layer has eight experts and the router selects two per token, so each token has access to about 47B parameters while only about 13B are active during inference (Jiang et al., Mixtral of Experts). Speed and cost track the active count; memory needs track the total count.

Load balancing

If the router keeps choosing the same few experts, the others never learn and the hardware is used unevenly. Training therefore adds a load-balancing loss that pushes the router to spread tokens across experts. The Switch Transformer simplified routing to a single expert per token and showed that this kind of sparse model could scale to over a trillion parameters (Fedus et al., Switch Transformers).

Advantages of MoE

• More capacity for the same compute: the model can store more knowledge and skills without making every token more expensive.

• Faster training to a given quality: sparse models often reach a target loss with less compute than dense models.

• Lower cost per token: providers can serve large models at prices closer to those of smaller dense models.

Trade-offs and limitations

• High memory requirements: every expert must be loaded into GPU memory, even if only a few run per token, which makes MoE models harder to self-host.

• Harder training: routing can be unstable, and balancing the load across experts takes extra care.

• Fine-tuning quirks: MoE models can overfit more easily when fine-tuned on small datasets.

• Serving complexity: spreading experts across several GPUs adds communication overhead, so infrastructure matters.

Teams that want to run MoE models on smaller hardware often combine them with quantization to reduce memory use.

What MoE means for AI engineers

You rarely build an MoE model from scratch, but you will choose between models that use it. When comparing options, look at both total and active parameters, check the memory needed if you plan to self-host, and measure latency and cost on your own workload rather than relying on parameter counts alone. If you are still getting comfortable with the basics, start with how large language models work, then come back to architecture choices like MoE.

FAQ

Are the experts specialized in topics like math or law?

Not in a human sense. Research suggests experts often specialize in token types or syntactic patterns rather than clear subject areas.

Is an MoE model smarter than a dense model?

Not automatically. MoE gives more capacity per unit of compute; final quality still depends on data, training, and tuning.

Why does an MoE model need so much memory if it only uses a few experts?

Because the router can pick any expert for the next token, all of them must be available in memory at inference time.

Do I need to know MoE to build LLM apps?

You do not need to implement it, but understanding it helps you pick models, estimate hosting costs, and explain performance trade-offs to your team.

Want to understand the models behind today’s AI products and build on top of them? Explore these Coderhouse courses:

• AI Engineering Course — learn to choose, evaluate, and deploy LLMs in real applications and agents.

• Data Analytics Course — build the SQL and analysis skills to measure what your AI systems actually deliver.