Typorithm · medium

Pretraining

pretrainingtransformersnext-tokenscaling

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Pretraining

Training & Alignment

Pretraining is self-supervised next-token prediction over trillions of tokens: the model produces logits, cross-entropy loss measures the error against the true next token, and gradient descent slowly improves it — building a general base model whose capability scales with parameters, data, and compute.

A language model is pretrained on a huge corpus by next-token prediction: given a stretch of text, predict the token that comes next. No human labels are needed — the next token is its own label, which is why this is called self-supervised.

Each step runs a forward pass to get logits, scores cross-entropy loss against the real next token, backpropagates to get gradients, and takes an optimizer step. Repeated over trillions of tokens, this yields a general base model that has absorbed broad knowledge of language.

Scaling laws describe how loss falls as you increase model parameters, training data, and compute together. They let teams predict the payoff of a bigger run before committing the resources, and explain why frontier models keep growing.

Pretraining = self-supervised next-token prediction.

Cross-entropy loss is minimized by gradient descent.

Scaling laws link params, data, and compute to capability.

// next-token prediction on huge corpora
const logits = model(tokens);
const loss = crossEntropy(logits, nextTokens);
loss.backward();
optimizer.step(); // scale: params x data x compute

Read the implementation

Connect each part of the code to the concept before you type.

1. Forward pass

The model maps a batch of tokens to logits — an unnormalized score for every possible next token.

const logits = model(tokens);

2. Cross-entropy loss

Cross-entropy compares the predicted distribution to the actual next token; minimizing it teaches the model to predict text.

const loss = crossEntropy(logits, nextTokens);

3. Backprop

Backpropagation computes the gradient of the loss with respect to every parameter.

loss.backward();

4. Optimizer step

The optimizer nudges parameters down the gradient — one tiny step of gradient descent.

optimizer.step();

5. Scaling laws

Capability grows predictably with more parameters, more training tokens, and more compute — the scaling laws.

// scale: params x data x compute

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.