What Is MoE (Mixture of Experts)? How Giant AIs Stay Fast

Among the latest large language models, there are more and more that seem contradictory at first glance: their total parameters run into the hundreds of billions, yet they actually run fast. The representative mechanism that makes this possible is MoE (Mixture of Experts). Many recent leading models—including Mixtral, Gemini, Grok, and DeepSeek—are said to adopt this architecture ².

MoE is one answer to a dilemma that ever-growing AI faces: “the bigger you make it, the more computational cost explodes.” This article explains, for non-specialists and drawing on the original paper and technical sources, what MoE is and why it can be huge yet fast. Reading it alongside how the Transformer works will deepen your understanding.

The Limit of Dense Models: Even Easy Questions Make “Everyone” Work

To understand MoE, it helps to first grasp how conventional models operate.

Conventional models like GPT-3 and Llama 2 are called dense models. A Qiita explainer describes their characteristic as “no matter how simple the incoming question, all the neurons in the model (everyone) participate in the computation” ². In other words, whatever the input, all of the model’s parameters are mobilized, so the larger you make the model, the more the computation for a single pass increases in step.

This is the dilemma of dense models. To raise performance, you want to add parameters. But the more parameters you add, the heavier the cost and time of inference (actually using it) become—you hit this wall.

The MoE Idea: Activate Only Some of the “Experts”

MoE overcomes this dilemma by stopping the practice of “using everything every time.” The Qiita explainer defines MoE as “a technology that achieves both a huge number of parameters and fast inference by activating only some of the ‘experts’ depending on the input” ².

The image is closer to a general hospital with many specialists than a single doctor who handles everything. Many small networks called “experts” are prepared inside the model, and depending on the input, “only some of the neurons (experts) that are knowledgeable about it participate in the computation” ². Because only the assigned members work rather than everyone, the per-pass computation stays light even when the total number is enormous.

The idea itself is not new. The 2017 original paper “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” (Shazeer et al., Google) introduced “a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks,” and reported achieving “greater than 1000x improvements in model capacity with only minor losses in computational efficiency” ¹. Increasing capacity (the model’s size) by orders of magnitude while keeping the increase in computation in check—that is the heart of MoE.

The Router Decides “Which Expert to Assign”

So who decides which experts to activate? That role is played by the router (gating network).

In the original paper’s wording, “a trainable gating network determines a sparse combination of these experts to use for each example” ¹. The Qiita explainer puts it more concretely: for each incoming token (a fragment of a word), it is a mechanism that “instantly decides which expert to assign it to” ².

A widely used approach here is the Top-K method. The router assigns a score to each expert and “sends data only to the top K experts with the highest scores (usually 2)” ². For example, even if there are eight experts, only the top two actually process a given token. The remaining experts rest, so they consume no computational resources. This property of “activating only some” is called sparsity, and it is why MoE is described as a sparse architecture.

”47B’s Knowledge at 13B’s Speed”—Seeing the Effect Through an Example

A clear illustration of this effect is Mixtral 8x7B, known as an open model. According to the Qiita explainer, this model has 47 billion parameters in total, yet “while holding the knowledge of a 47B model, it runs at the blazing speed of a 13B model” ².

In other words, the breadth of its knowledge drawers (total parameters) is on par with a large model, while the portion actually run each time is held to the level of a mid-size model. Besides Mixtral 8x7B, models said to adopt MoE include Gemini 1.5, Grok-1, and DeepSeek-V3 ². GPT-4 is also frequently reported to use MoE, but there is no official confirmation from OpenAI.

Between the Transformer and Model Compression

MoE is not a technology that replaces AI with something entirely new. In many cases, within the basic structure of the Transformer, the spot where all inputs would normally pass through the same part is replaced by multiple experts and a router. It helps to see it as an evolution that turns part of the Transformer foundation into a “division of labor.”

For the goal of running giant models at practical cost, the problem also overlaps with model compression (distillation and quantization). Whereas compression improves efficiency by making the model itself smaller, MoE improves efficiency by keeping it large while “narrowing down the parts used”—a difference in approach.

You will likely keep seeing the phrase “adopts MoE” every time a new large-scale model appears. Behind it lies a practical demand that AI development has long carried: “for the sake of intelligence we want models large, yet we don’t want to use all of it every time.” MoE is, for now, one of the most practical answers to that demand.