Typorithm · hard

Scaling & Mixture-of-Experts

scaling-moetransformersscalingmoesparse

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Scaling & Mixture-of-Experts

Training & Alignment

Scaling laws push models larger, but Mixture-of-Experts grows capacity efficiently: a router sends each token to only the top-k expert feed-forward networks, so total parameters rise sharply while per-token compute stays near constant — sparse activation buys more capacity without proportional cost.

Scaling laws say bigger helps, but making every parameter active for every token gets expensive fast. Mixture-of-Experts (MoE) breaks that link: a layer holds many expert feed-forward networks, and a lightweight router picks just a few experts per token.

The router scores the experts for the current token, softmax turns those scores into weights, and top-k keeps only the highest few (commonly 1 or 2). Those experts process the token and their outputs are combined. The vast majority of experts sit idle for any given token — this is sparse activation.

The payoff: total parameter count (and therefore capacity) can be many times larger, while the compute per token stays close to that of a much smaller dense model. The costs are added memory to hold all experts and the engineering of load-balancing tokens across them.

A router sends each token to only the top-k experts.

Total params grow; per-token compute stays near constant.

Sparse activation = capacity without proportional cost.

// MoE: route each token to a few experts
const gate = softmax(router(x));
const experts = topK(gate, 2);
const y = combine(experts, x);
// more params, near-constant compute/token

Read the implementation

Connect each part of the code to the concept before you type.

1. MoE idea

Mixture-of-Experts replaces one big feed-forward block with many experts, but each token visits only a few.

// MoE: route each token to a few experts

2. Router gate

A small router scores each expert for this token; softmax turns the scores into routing weights.

const gate = softmax(router(x));

3. Top-k routing

Only the top-k experts (here 2) are activated for the token — this sparsity keeps compute low.

const experts = topK(gate, 2);

4. Combine outputs

The chosen experts process the token and their outputs are weighted and combined into the result.

const y = combine(experts, x);

5. The trade-off

Total parameters grow with the number of experts, but per-token compute stays near constant because only a few run.

// more params, near-constant compute/token

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.