Typorithm · hard

Fine-Tuning & RLHF

fine-tuning-rlhftransformerssftrlhfalignment

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Fine-Tuning & RLHF

Training & Alignment

Alignment turns a base model into a helpful assistant in stages: supervised fine-tuning (SFT) on instruction/response pairs, then RLHF — a reward model trained on human preference comparisons guides reinforcement learning (PPO) to maximize reward, with DPO as a simpler, reward-model-free alternative.

A base model predicts text but does not reliably follow instructions or stay helpful and safe. Supervised fine-tuning (SFT) is the first alignment step: continue training on curated instruction/response pairs so the model learns the assistant style. It is still cross-entropy, just on cleaner, task-shaped data.

RLHF (reinforcement learning from human feedback) goes further. Humans rank competing responses; a reward model learns to predict those preferences as a score. Reinforcement learning — usually PPO — then optimizes the model to produce responses the reward model rates highly.

DPO (direct preference optimization) reaches a similar goal more simply: it optimizes the model directly on the preference pairs without training a separate reward model or running full RL, which makes it cheaper and more stable to run.

SFT imitates instruction/response pairs (cross-entropy).

RLHF: reward model + PPO maximize human preference.

DPO optimizes preferences directly, skipping the reward model.

// 1. SFT on instruction/response pairs
sftLoss = crossEntropy(model(prompt), response);
// 2. train a reward model on human prefs
// 3. RL (PPO/DPO) to maximize reward
policyLoss = -reward(model(prompt));

Read the implementation

Connect each part of the code to the concept before you type.

1. Supervised fine-tuning

Step one teaches the base model to follow instructions by imitating curated prompt/response examples.

// 1. SFT on instruction/response pairs

2. SFT loss

SFT is still next-token cross-entropy, but on instruction/response pairs rather than raw web text.

sftLoss = crossEntropy(model(prompt), response);

3. Reward model

Humans compare answers; a reward model learns to score how much a response is preferred.

// 2. train a reward model on human prefs

4. RL stage

Reinforcement learning (PPO) — or the simpler DPO — updates the model to produce higher-reward responses.

// 3. RL (PPO/DPO) to maximize reward

5. Policy objective

Minimizing the negative reward is the same as maximizing reward — the model is steered toward preferred answers.

policyLoss = -reward(model(prompt));

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.