Fine-Tuning vs RLHF and DPO
Pretraining decides what a model can say. Supervised fine-tuning decides what it usually says. Preference optimisation decides which of two plausible answers it prefers — and, pushed too hard, which one it degenerates into.
Stage 2 — Fine-Tuning
how hard the demonstrations pull
Stage 3 — Preferences
high keeps the model near its SFT self; low lets it chase reward
a real and well-documented flaw, not a hypothetical one
"Explain why the sky is blue to a 10-year-old"
—The Three Stages
Scores
Which Stage Fixes What
Pretraining — knowledge and fluency. Enormously expensive, done once.
SFT — format, task, register. Cheap, needs written examples.
RLHF / DPO — taste between plausible answers. Needs comparisons, not examples.
Retrieval — facts. None of the above.
Fine-Tuning, RLHF and DPO: A Practical Guide
Three techniques that all get called "training", doing three different jobs.
Quick Context
A pretrained model is a text predictor. Ask it a question and it produces something a document containing that question might plausibly continue with — which is not the same as an answer, and certainly not a helpful one.
Two more stages turn that into an assistant. Supervised fine-tuning shows it thousands of written demonstrations of the behaviour you want. Preference optimisation then shows it pairs of answers with a human verdict on which is better, for all the cases where nobody could write the ideal answer but anyone can pick the better of two.
The maths both preference methods share
Every preference method optimises the same objective: get more reward without drifting far from the model you started with. That constraint has a closed form,
π(y) ∝ πref(y) · exp( r(y) / β )
which is exactly what this page computes. The reference policy is the SFT model, r is the reward, and β is the leash: large β keeps the tuned model close to its reference, small β lets it chase reward wherever that leads. The drift readout is the KL divergence between the two, which is the quantity the penalty is penalising.
RLHF gets r from a reward model trained on human comparisons, then optimises against it with PPO. DPO skips the reward model entirely: the same optimum can be reached by a supervised-looking loss on the preference pairs themselves. Same target, far less machinery, which is why DPO and its relatives took over so quickly.
Interactive Exploration Guide
- Start at pretraining. With SFT at 0 and no preference method, the dense jargon answer is the most likely one. That is what "most probable continuation" looks like when the text you trained on is the internet.
- Apply SFT. Raise Supervised Fine-Tuning to 2. The demonstrated answer — short, clear, aimed at a child — takes over. Nothing has been rewarded yet; it is just imitation.
- Add RLHF with a tight leash. Set the method to RLHF with beta at 3. The good answer gains a little, the drift stays small, and human ratings go up.
- Loosen the leash. Drag beta down towards 0.1. The model collapses onto the long, hedging, flattering answer — because the reward model scores it highest. Reward goes up, the human rating goes down, and the drift readout goes through the roof. That is reward hacking, and it is why the KL penalty is not optional.
- Fix the reward model instead. Uncheck Reward Model Likes Long Answers and the collapse stops happening, at the same beta. The KL penalty was never the real fix; it was a guard rail against a proxy that was wrong.
- Compare DPO. Switch the method to DPO. It moves probability towards the preferred answer and away from the rejected one, and it has no reward model to hack — but it also has no opinion at all about answers nobody compared.
Choosing between them
- Wrong facts? None of these. Use retrieval. Fine-tuning teaches behaviour, and treating it as a way to insert knowledge is the most expensive mistake in this area.
- Wrong format, tone or task? SFT, and often LoRA so you are not moving every weight.
- Right content, wrong judgement? Preferences. If you can rank two outputs but cannot write the perfect one, that is exactly the signal these methods consume.
- Nothing but a few examples? Try prompting first. A great deal of what people fine-tune for is a system prompt away.
What usually goes wrong
- Reward hacking. The reward model is a proxy for human judgement, and optimisers find every gap between the two — length, confidence, flattery, formatting.
- Alignment tax. Heavy preference tuning can make a model more agreeable and less capable, hedging where it used to answer.
- Catastrophic forgetting. Fine-tune hard on a narrow task and unrelated abilities degrade. Low learning rates, adapters and mixed data all exist for this.
- Preference data that disagrees with itself. Two annotators, two answers, no shared rubric. The resulting reward model learns the noise.
- Evaluating on the reward model. If the reward model is also your evaluation, you have measured the thing you optimised and learned nothing.
Key Takeaway
Pretraining supplies knowledge and fluency, supervised fine-tuning supplies behaviour by imitation, and preference optimisation supplies judgement between answers that are all plausible. RLHF and DPO reach the same optimum — a policy proportional to the reference times the exponentiated reward — with DPO skipping the reward model that RLHF has to train and then defend. The beta term is the leash, and slackening it produces exactly the failure you would predict: reward climbs, human ratings fall, and the model drifts into whatever the proxy overrates. None of these stages is a way to teach a model new facts; that is what retrieval is for.