1. Raw logits
The model emits one raw score (logit) per vocabulary token for the next position.
const logits = model(context);Typorithm · medium
Sampling: Temperature & Top-p
samplingtransformerstemperaturetop-p
Saved on this device
Full lesson · 8 learning steps Open any exercise for focused practice
10%
Step 1 of 8 · intro card
Learn Sampling: Temperature & Top-p
Inference & Sampling
The model outputs a probability distribution over the vocabulary. Temperature scales randomness — low is near-deterministic, high is diverse. Top-p (nucleus) sampling draws from the smallest set of tokens covering probability p; top-k caps the candidate count and greedy takes the argmax.
At each step the model produces a logit for every token in the vocabulary. softmax converts those logits into a probability distribution, and a sampling strategy picks the next token from it.
Temperature rescales the logits before softmax. A temperature below 1 sharpens the distribution toward the most likely tokens (more deterministic); above 1 flattens it (more diverse and surprising). At temperature 0 you effectively pick the argmax.
Top-p (nucleus) sampling keeps the smallest set of top tokens whose probabilities add up to p, then samples within that set. Top-k instead keeps a fixed number k of the highest-probability tokens, and greedy decoding always takes the single most likely token.
01
softmax over logits = a distribution over the vocabulary.
02
Temperature <1 sharpens, >1 diversifies.
03
Top-p samples the smallest set covering probability p.
const logits = model(context);
const scaled = logits.div(temperature); // >1 random, <1 sharp
const probs = softmax(scaled);
const next = nucleusSample(probs, { topP: 0.9 });Read the implementation
Connect each part of the code to the concept before you type.
2. Temperature
Dividing logits by temperature scales randomness: below 1 sharpens the distribution, above 1 flattens it.
logits.div(temperature)3. Probabilities
softmax turns the scaled logits into a probability distribution over the whole vocabulary.
const probs = softmax(scaled);4. Top-p
Nucleus sampling draws from the smallest set of tokens whose probabilities sum to p (0.9 here).
nucleusSample(probs, { topP: 0.9 })5. p threshold
p is the cumulative-probability cutoff: only tokens inside the top 90% mass are candidates.
topP: 0.9A complete learning loop
01
Understand
Read the concept map and follow the code breakdown.
02
Reconstruct
Type the implementation, repair mistakes and work through every recall exercise.
03
Verify
Use the final check without hints; revisit weak concepts in short practice sessions.