Typorithm · hard

Multi-Head Attention

multi-headtransformersparallelprojection

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Multi-Head Attention

Attention & Transformers

Multi-head attention runs several attention heads in parallel, each on its own projected subspace, so different heads can capture different relationships. The heads' outputs are concatenated and projected back to the model dimension with a learned matrix Wo.

A single attention computation can only focus one way at a time. Multi-head attention splits the projections into h smaller heads and runs attention in all of them in parallel.

Each head operates on its own lower-dimensional subspace, so one head might track subject-verb agreement while another tracks coreference or positional patterns. This gives the model several independent 'views' of the sequence.

After the heads finish, their outputs are concatenated and passed through a final learned projection Wo, which combines the per-head results back into a single vector of the model's dimension.

h heads run attention in parallel on projected subspaces.

Different heads capture different relationships.

Heads are concatenated, then projected by Wo.

// run attention h times in parallel
const heads = range(h).map((i) => attention(Q[i], K[i], V[i]));
const concat = concatenate(heads);
const output = concat.matmul(Wo);
// each head learns a different relation

Read the implementation

Connect each part of the code to the concept before you type.

1. h parallel heads

Instead of one attention, multi-head attention runs h attention computations side by side.

// run attention h times in parallel

2. Per-head attention

Each head i runs the same attention on its own projected slice of the queries, keys, and values.

range(h).map((i) => attention(Q[i], K[i], V[i]))

3. Concatenate heads

The per-head outputs are concatenated back into a single wide vector per token.

const concat = concatenate(heads);

4. Output projection Wo

A final learned matrix Wo mixes the concatenated heads back to the model dimension.

const output = concat.matmul(Wo);

5. Different relations

Because each head sees a different subspace, heads specialize in different relationships like syntax or coreference.

// each head learns a different relation

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.