Fine-Tuning vs RLHF and DPO

Quick Context

A pretrained model is a text predictor. Ask it a question and it gives something a document holding that question might plausibly continue with —. This is not the same as an answer, and certainly not a helpful one.

Two more stages turn that into an assistant. Supervised fine-tuning shows it thousands of written demonstrations of the behaviour you want. Choice optimisation then shows it pairs of answers with a human verdict on which is better, for all the cases where nobody could write the ideal answer but anyone can pick the better of two. The division of labour matters: SFT teaches format and basic cue below. While choice optimisation teaches the subtler qualities. Helpfulness, safety, tone. That humans can spot easily but cannot show perfectly in a single gold-standard case. RLHF was the first choice method: train a reward model on human checks. Then optimise the language model against that reward with PPO while penalising drift from the SFT checkpoint. DPO simplified the pipeline by showing that the same optimum can be reached with a supervised loss right on the choice pairs. This skips the reward model and the reinforcement learning loop fully. Both methods solve the same constrained optimisation problem. Boost reward subject to a KL cost that keeps the model close to its guide. And the temperature parameter beta controls how tight the leash is. This page computes the reward, the KL divergence. And the policy shift for a toy vocabulary so you can see just what the cost does.

The maths both choice methods share

Each choice method optimises the same objective: get more reward without drifting far from the model you started with. That constraint has a closed form,

π(y) ∝ π_ref(y) · exp( r(y) / β )

which is just what this page computes. The guide policy is the SFT model, r is the reward, and β is the leash. Large β keeps the tuned model close to its guide, small β lets it chase reward wherever that leads. The drift readout is the KL divergence between the two. This is the quantity the cost is penalising.

Three stages, three different jobs

A usable assistant is not gave in one training run. Three stages follow one another, each teaching something the last one cannot.

1. Pretraining — next-token prediction over enormous quantities of text. This is where knowledge and language competence come from. It costs millions of pounds and gives a base model that completes text rather than answering questions.

2. Supervised fine-tuning (SFT) — training on curated cases of cues and good responses. This teaches the format: that a question should be answered, that output should be structured a given way, that a conversation has turns.

3. Choice optimisation — RLHF, DPO or close. Humans compare pairs of responses and the model is tuned towards the preferred one. This teaches the qualities that are easy to spot and hard to show. Helpfulness, right caution, tone, refusing what should be refused.

Why choice learning exists at all

The natural question: if SFT can learn from good cases, why add a stage?

Because for many qualities, writing an ideal case is harder than judging between two candidates. Asked to give the perfect response to a sensitive question, most annotators struggle. Shown two responses and asked which is better, they answer quickly and always.

Choice data is so cheaper and more reliable to collect for just the qualities that matter most in an assistant.

There is a second reason, which is about the loss function. SFT boosts the likelihood of one target response, so anything not in the target is implicitly wrong — even if it is equally good. Choice training instead pushes the model's distribution towards a whole region of preferred behaviour. This suits problems with many valid answers.

RLHF, and how DPO simplified it

RLHF has three moving parts:

1. Collect human checks of response pairs.
2. Train a reward model to predict which response a human would prefer.
3. Optimise the language model against that reward with reinforcement learning (PPO), including a KL cost keeping it close to the SFT model.

The KL cost is not optional. Without it the policy drifts into degenerate text that scores highly on the reward model and is useless — reward hacking. And it is the trait failure of the method.

Practically, RLHF is complicated: two models in memory, an RL loop that is sensitive to hyperparameters. And a reward model that can be gamed.

DPO (direct choice optimisation) removes the reward model and the RL loop. It shows that the same objective can be optimised right on the choice pairs with a simple classification-style loss. Increase the likelihood of the preferred response, decrease the likelihood of the rejected one, next to a frozen guide model.

DPO's ease is why it became the default for open models. Variants — IPO, KTO, ORPO, SimPO — adjust the loss or remove the guide model; the family is still moving.

The one equation both choice methods share

RLHF and DPO look like fully different machinery, and they optimise the same objective. Writing that objective out is the shortest route to grasp what choice training does. And what neither method can fix.

import numpy as np

print("A PREFERENCE PAIR is the whole training signal. no correct answer,")
print("just an ordering:")
print("      prompt:   'explain gradient descent'")
print("      chosen:   a clear two-paragraph explanation")
print("      rejected: a correct but impenetrable one")
print("   note what is NOT here: a target string. supervised fine-tuning")
print("   needs someone to WRITE the ideal answer; preference training")
print("   needs someone to PICK between two, which is enormously cheaper")
print("   and much more reliable -- people are bad at writing the best")
print("   answer and good at spotting the better one.")
print()

print("THE OBJECTIVE BOTH METHODS OPTIMISE. write the reward of an answer")
print("as r, and the probability that a human prefers A over B is modelled")
print("by the Bradley-Terry law:")
print("      P(A preferred) = sigmoid(r_A - r_B)")
print("   which says only the DIFFERENCE of rewards matters:")
def sig(x):
    return 1.0 / (1.0 + np.exp(-x))
print("%-16s %-16s %14s %16s" % ("r_A", "r_B", "difference", "P(A wins)"))
for a, b in ((1.0, 0.0), (5.0, 4.0), (0.5, 0.0), (3.0, 0.0), (0.0, 0.0)):
    print("%-16.1f %-16.1f %14.1f %16.4f" % (a, b, a - b, sig(a - b)))
print("   rows 1 and 2 have completely different rewards and identical")
print("   preferences, because the difference is the same. a reward model")
print("   is therefore only ever defined up to an additive constant --")
print("   which is why you cannot read an absolute 'quality' off one.")
print()

The next block goes on with the left demonstrations.

import numpy as np

print("RLHF DOES IT IN TWO STEPS:")
print("%-10s %-34s %s" % ("step", "what it trains", "on what"))
print("%-10s %-34s %s" % ("1", "a reward model r(x, y)", "the preference pairs"))
print("%-10s %-34s %s" % ("2", "the policy, by RL (PPO)", "the reward model"))
print("   the second step needs a reference model held in memory too, to")
print("   keep the policy from drifting:")
print("      maximise  r(x, y)  -  beta * KL(policy || reference)")
print("%-34s %s" % ("models in memory during PPO", "policy, reference, reward,"))
print("%-34s %s" % ("", "and the value head -- four"))
print("   the KL term is not optional. without it the policy walks off to")
print("   whatever the reward model scores highly, which is not the same")
print("   as what humans like -- that is reward hacking, and it looks like")
print("   a model that has learned to write in whatever style the reward")
print("   model over-scores.")
print()

The next block goes on with the left demonstrations.

import numpy as np


print("DPO SKIPS STEP 1 ENTIRELY. the insight is that the optimal policy")
print("for that KL-constrained objective has a closed form, so the reward")
print("can be written IN TERMS OF the policy:")
print("      r(x, y) = beta * log( policy(y|x) / reference(y|x) )")
print("   substitute that back into Bradley-Terry and the reward model")
print("   cancels out. what is left is a loss you can compute from four")
print("   numbers, all of them log-probabilities:")
print("%-34s %s" % ("policy log-prob of the chosen", "pc"))
print("%-34s %s" % ("policy log-prob of the rejected", "pr"))
print("%-34s %s" % ("reference log-prob of the chosen", "rc"))
print("%-34s %s" % ("reference log-prob of the rejected", "rr"))
print("      loss = -log sigmoid( beta * ((pc - rc) - (pr - rr)) )")
print()
print("   work through what that quantity means:")
BETA = 0.1
CASES = [
    ("the policy already prefers the chosen", -1.0, -3.0, -1.5, -2.0),
    ("the policy prefers the REJECTED", -3.0, -1.0, -1.5, -2.0),
    ("policy and reference agree exactly", -1.5, -2.0, -1.5, -2.0),
    ("the policy strongly prefers chosen", -0.5, -6.0, -1.5, -2.0),
]
# d(loss)/d(margin) = -beta * sigmoid(-beta * margin), so the gradient
# is largest exactly where the policy is most wrong
print("%-40s %10s %10s %16s"
      % ("situation", "margin", "loss", "|d loss/d margin|"))
for name, pc, pr, rc, rr in CASES:
    margin = (pc - rc) - (pr - rr)

The next block goes on with the left demonstrations.

import numpy as np

    grad = BETA * sig(-BETA * margin)
    print("%-40s %10.2f %10.4f %16.5f" % (name, margin, loss, grad))
print("   the margin is 'how much more the policy prefers the chosen")
print("   answer than the reference model did'. positive means the policy")
print("   has already moved in the right direction and the loss is small.")
print("   row 3 is the starting point: policy equals reference, margin 0,")
print("   loss = -log(0.5) = %.4f. training begins from exactly there."
      % -np.log(0.5))
print("   and read the gradient column: it is largest where the policy is")
print("   most wrong and smallest where it is already right -- but only by")
print("   %.1fx, which is barely any self-regulation at all."
      % ((BETA * sig(BETA * 2.5)) / (BETA * sig(-BETA * 5.0))))
print("   THAT is what beta controls. it scales the margin before the")
print("   sigmoid, so it decides how far into the sigmoid's flat regions")
print("   the training ever gets:")
print("%-10s %18s %18s %14s"
      % ("beta", "grad, most wrong", "grad, already right", "ratio"))
for bt in (0.05, 0.1, 0.5, 1.0, 4.0):
    hi, lo = bt * sig(bt * 2.5), bt * sig(-bt * 5.0)
    print("%-10.2f %18.5f %18.5f %13.1fx" % (bt, hi, lo, hi / lo))
print("   at beta = 0.05 every pair contributes about equally and the")
print("   policy barely moves from the reference. at beta = 4 the pairs")
print("   the model already gets right stop mattering entirely and it")
print("   trains almost only on its mistakes -- which is faster and drifts")
print("   further from the reference model.")
print("   that is the whole tuning story for DPO: beta is a leash, and")
print("   0.1 to 0.5 is where most people end up.")
print()

The next block goes on with the left demonstrations.

import numpy as np

print("SO WHAT DID DPO REMOVE?")
print("%-30s %-18s %s" % ("", "RLHF", "DPO"))
for row in (("models in memory", "4", "2"),
            ("a separate reward model", "yes", "no"),
            ("an RL loop", "yes", "no -- plain SGD"),
            ("hyperparameters to tune", "many", "in effect beta"),
            ("what it optimises", "the same objective", "the same objective")):
    print("%-30s %-18s %s" % row)
print("   the last row is the point. DPO is not a different goal reached")
print("   more cheaply; it is the SAME goal with the intermediate model")
print("   algebraically eliminated.")
print("   RLHF keeps one real advantage: a reward model can score answers")
print("   that were never in the preference data, so it can be reused for")
print("   sampling, filtering and evaluation. DPO gives you a policy and")
print("   nothing else.")
print()

print("AND WHAT NEITHER OF THEM FIXES. the three stages do different jobs:")
print("%-24s %-30s %s" % ("stage", "teaches", "cannot fix"))
for row in (("pretraining", "language and world knowledge", "-"),
            ("supervised fine-tuning", "the format of a reply", "missing knowledge"),
            ("preference training", "which reply is preferred", "missing knowledge")):
    print("%-24s %-30s %s" % row)
print("   preference training changes which of the answers a model could")
print("   already produce it prefers to produce. if the right answer was")
print("   never in that set, no amount of preference data puts it there.")
print("   that is why a model that does not know your company's refund")
print("   policy cannot be fine-tuned into knowing it reliably, and why")
print("   the answer to that problem is retrieval rather than training.")

Guided tour

1. Start at pretraining. With SFT at 0 and no choice method, the dense jargon answer is the most likely one. That is what "most probable follow-up" looks like when the text you trained on is the internet.
2. Apply SFT. Raise Supervised Fine-Tuning to 2. The showed answer — short, clear, aimed at a child — takes over. Nothing has been rewarded yet; it is just imitation.
3. Add RLHF with a tight leash. Set the method to RLHF with beta at 3. The good answer gains a little, the drift stays small, and human ratings go up.
4. Loosen the leash. Drag beta down towards 0.1. The model collapses onto the long, hedging, flattering answer — because the reward model scores it highest. Reward goes up, the human rating goes down, and the drift readout goes through the roof. That is reward hacking, and it is why the KL cost is not optional.
5. Fix the reward model instead. Uncheck Reward Model Likes Long Answers and the collapse stops happening, at the same beta. The KL cost was never the real fix; it was a guard rail against a proxy that was wrong.
6. Compare DPO. Switch the method to DPO. It moves chance towards the preferred answer and away from the rejected one. And it has no reward model to hack. But it also has no opinion at all about answers nobody compared.

Choosing between them

• Wrong facts? None of these. Use retrieval. Fine-tuning teaches behaviour, and treating it as a way to insert knowledge is the most costly mistake in this area.
• Wrong format, tone or task? SFT, and often LoRA so you are not moving each weight.
• Right content, wrong judgement? Choices. If you can rank two outputs but cannot write the perfect one, that is just the sign these methods consume.
• Nothing but a few cases? Try prompting first. A great deal of what people fine-tune for is a system prompt away.

Failure modes

• Reward hacking. The reward model is a proxy for human judgement, and optimisers find each gap between the two. Length, confidence, flattery, formatting.
• Alignment tax. Heavy choice tuning can make a model more agreeable and less capable, hedging where it used to answer.
• Catastrophic forgetting. Fine-tune hard on a narrow task and unrelated abilities degrade. Low learning rates, adapters and mixed data all exist for this.
• Choice data that disagrees with itself. Two annotators, two answers, no shared rubric. The coming reward model learns the noise.
• Evaluating on the reward model. If the reward model is also your evaluation, you have measured the thing you optimised and learned nothing.

Worth recalling

Pretraining supplies knowledge and fluency, supervised fine-tuning supplies behaviour by imitation, and choice optimisation supplies judgement between answers that are all plausible. RLHF and DPO reach the same optimum. A policy tied to to the guide times the exponentiated reward. With DPO skipping the reward model that RLHF has to train and then defend. The beta term is the leash, and slackening it gives just the failure you would predict. Reward climbs, human ratings fall. And the model drifts into any the proxy overrates. None of these stages is a way to teach a model new facts; that is what retrieval is for.

What each stage cannot fix

This is the practically useful part, because it decides which stage to reach for.

Fine-tuning does not reliably add facts. It can be made to memorise, and doing so is slow, hard to update, and gives confident errors on anything nearby that was not memorised. Facts belong in retrieval.

Choice training does not add power. It shapes which of the model's existing behaviours are expressed. A model that cannot do arithmetic will not learn to from choice data.

Neither fixes a knowledge cut-off. Only retrieval or a tool does.

SFT on a narrow task can cause forgetting. Heavy fine-tuning on one domain degrades general ability. The alignment tax. LoRA reduces it, as does mixing general data into the fine-tuning set.

So the decision table most projects need:

Practical guidance

Try prompting first. With few-shot cases and a clear system prompt, a strong model handles a great many tasks without any training. Fine-tuning is a commitment to a pipeline, a dataset and a retraining cadence.

If fine-tuning, use LoRA. A few hundred to a few thousand well-curated cases, adapters. Rather than full weights, and evaluation against a held-out set that existed before you started.

Data quality rules quantity. A thousand carefully written cases beat ten thousand scraped ones. The well-known result here is that a small, high-quality cue set can give a surprisingly capable assistant.

Choice optimisation is rarely the first move for an use team. It matters when you are shipping a model whose tone and refusal behaviour you own.

Evaluate on what you care about. Fine-tuning improves the target metric and can degrade others. Keep a general power check alongside the task-set one.

Questions people ask

RAG or fine-tuning? RAG for facts, currency and citations. Fine-tuning for format, style and task behaviour. They are complementary and often used together.

How much data do I need for SFT? A few hundred cases can shift format clearly; a few thousand for a large task change.

Is DPO strictly better than RLHF? Simpler and comparable in most published checks. Well-executed PPO still leads on some benchmarks.

What is the alignment tax? The power lost when tuning for helpfulness and safety. Real, and often small next to the gain.

Can I fine-tune a hosted model? A few providers offer it, often LoRA-based, with your data and their infrastructure.

Does fine-tuning make a model smaller or faster? No — that is distillation or quantisation. Fine-tuning changes behaviour, not size.

Recap in one screen

• Pretraining gives knowledge, SFT gives format, choice optimisation gives quality and tone.
• Choice data exists because judging two responses is easier and more reliable than writing an ideal one.
• RLHF needs a reward model and an RL loop; DPO reaches comparable results with a supervised loss.
• Fine-tuning does not reliably add facts — use retrieval for that.
• Try prompting first, then LoRA on a few thousand carefully curated cases.