Typorithm · hard

The Transformer Block

transformer-blocktransformersresiduallayernormffn

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn The Transformer Block

Attention & Transformers

A transformer block stacks an attention sublayer and a feed-forward network, each wrapped in a residual connection and layer normalization (pre-norm is standard). Stacking N such blocks builds depth, and a causal mask keeps decoder tokens from attending to future positions.

The transformer block is the repeating unit of the model. It has two sublayers: a multi-head attention layer and a position-wise feed-forward network (FFN) applied to each token independently.

Each sublayer is wrapped in a residual connection — its output is added back to its input — and a layer normalization. In the now-standard pre-norm arrangement, layerNorm is applied before the sublayer, which keeps gradients stable in very deep stacks.

Stacking N identical blocks gives the model its depth. In a decoder, attention uses a causal mask so that each position can only attend to itself and earlier tokens, never to future ones, which is what makes left-to-right generation possible.

Two sublayers: attention then a feed-forward network.

Residual + layer norm wrap each sublayer (pre-norm).

Stack N blocks; a causal mask hides future tokens.

// pre-norm residual block
x = x.add(attention(layerNorm(x)));
x = x.add(feedForward(layerNorm(x)));
// stack N blocks; causal mask for decoding

Read the implementation

Connect each part of the code to the concept before you type.

1. Pre-norm block

Pre-norm applies layer normalization before each sublayer, which is the standard for stable deep transformers.

// pre-norm residual block

2. Attention sublayer

The attention output is added back to x through a residual connection, with layerNorm applied to the input.

x = x.add(attention(layerNorm(x)));

3. Feed-forward sublayer

A position-wise feed-forward network follows attention, again wrapped in a residual and layer norm.

x = x.add(feedForward(layerNorm(x)));

4. Layer normalization

Layer norm rescales each token's features to stabilize training across deep stacks.

layerNorm(x)

5. Stack and mask

Many blocks are stacked for depth, and a causal mask stops decoder tokens from attending to the future.

// stack N blocks; causal mask for decoding

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.