Fine-Tuning vs RLHF and DPO

The maths both preference methods share

Every preference method optimises the same objective: get more reward without drifting far from the model you started with. That constraint has a closed form,

π(y) ∝ π_ref(y) · exp( r(y) / β )

which is exactly what this page computes. The reference policy is the SFT model, r is the reward, and β is the leash: large β keeps the tuned model close to its reference, small β lets it chase reward wherever that leads. The drift readout is the KL divergence between the two, which is the quantity the penalty is penalising.

Three stages, three different jobs

A usable assistant is not produced in one training run. Three stages follow one another, each teaching something the previous one cannot.

1. Pretraining — next-token prediction over enormous quantities of text. This is where knowledge and language competence come from. It costs millions of pounds and produces a base model that completes text rather than answering questions.

2. Supervised fine-tuning (SFT) — training on curated examples of instructions and good responses. This teaches the format: that a question should be answered, that output should be structured a particular way, that a conversation has turns.

3. Preference optimisation — RLHF, DPO or similar. Humans compare pairs of responses and the model is tuned towards the preferred one. This teaches the qualities that are easy to recognise and hard to demonstrate: helpfulness, appropriate caution, tone, refusing what should be refused.

Why preference learning exists at all

The natural question: if SFT can learn from good examples, why add a stage?

Because for many qualities, writing an ideal example is harder than judging between two candidates. Asked to produce the perfect response to a sensitive question, most annotators struggle. Shown two responses and asked which is better, they answer quickly and consistently.

Preference data is therefore cheaper and more reliable to collect for exactly the qualities that matter most in an assistant.

There is a second reason, which is about the loss function. SFT maximises the likelihood of one target response, so anything not in the target is implicitly wrong — even if it is equally good. Preference training instead pushes the model's distribution towards a whole region of preferred behaviour, which suits problems with many valid answers.

RLHF, and how DPO simplified it

RLHF has three moving parts:

1. Collect human comparisons of response pairs.
2. Train a reward model to predict which response a human would prefer.
3. Optimise the language model against that reward with reinforcement learning (PPO), including a KL penalty keeping it close to the SFT model.

The KL penalty is not optional. Without it the policy drifts into degenerate text that scores highly on the reward model and is useless — reward hacking, and it is the characteristic failure of the method.

Practically, RLHF is complicated: two models in memory, an RL loop that is sensitive to hyperparameters, and a reward model that can be gamed.

DPO (direct preference optimisation) removes the reward model and the RL loop. It shows that the same objective can be optimised directly on the preference pairs with a simple classification-style loss — increase the likelihood of the preferred response, decrease the likelihood of the rejected one, relative to a frozen reference model.

DPO's simplicity is why it became the default for open models. Variants — IPO, KTO, ORPO, SimPO — adjust the loss or remove the reference model; the family is still moving.

The one equation both preference methods share

RLHF and DPO look like completely different machinery, and they optimise the same objective. Writing that objective out is the shortest route to understanding what preference training does -- and what neither method can fix.

import numpy as np

print("A PREFERENCE PAIR is the whole training signal. no correct answer,")
print("just an ordering:")
print("      prompt:   'explain gradient descent'")
print("      chosen:   a clear two-paragraph explanation")
print("      rejected: a correct but impenetrable one")
print("   note what is NOT here: a target string. supervised fine-tuning")
print("   needs someone to WRITE the ideal answer; preference training")
print("   needs someone to PICK between two, which is enormously cheaper")
print("   and much more reliable -- people are bad at writing the best")
print("   answer and good at spotting the better one.")
print()

print("THE OBJECTIVE BOTH METHODS OPTIMISE. write the reward of an answer")
print("as r, and the probability that a human prefers A over B is modelled")
print("by the Bradley-Terry law:")
print("      P(A preferred) = sigmoid(r_A - r_B)")
print("   which says only the DIFFERENCE of rewards matters:")
def sig(x):
    return 1.0 / (1.0 + np.exp(-x))
print("%-16s %-16s %14s %16s" % ("r_A", "r_B", "difference", "P(A wins)"))
for a, b in ((1.0, 0.0), (5.0, 4.0), (0.5, 0.0), (3.0, 0.0), (0.0, 0.0)):
    print("%-16.1f %-16.1f %14.1f %16.4f" % (a, b, a - b, sig(a - b)))
print("   rows 1 and 2 have completely different rewards and identical")
print("   preferences, because the difference is the same. a reward model")
print("   is therefore only ever defined up to an additive constant --")
print("   which is why you cannot read an absolute 'quality' off one.")
print()

print("RLHF DOES IT IN TWO STEPS:")
print("%-10s %-34s %s" % ("step", "what it trains", "on what"))
print("%-10s %-34s %s" % ("1", "a reward model r(x, y)", "the preference pairs"))
print("%-10s %-34s %s" % ("2", "the policy, by RL (PPO)", "the reward model"))
print("   the second step needs a reference model held in memory too, to")
print("   keep the policy from drifting:")
print("      maximise  r(x, y)  -  beta * KL(policy || reference)")
print("%-34s %s" % ("models in memory during PPO", "policy, reference, reward,"))
print("%-34s %s" % ("", "and the value head -- four"))
print("   the KL term is not optional. without it the policy walks off to")
print("   whatever the reward model scores highly, which is not the same")
print("   as what humans like -- that is reward hacking, and it looks like")
print("   a model that has learned to write in whatever style the reward")
print("   model over-scores.")
print()

print("DPO SKIPS STEP 1 ENTIRELY. the insight is that the optimal policy")
print("for that KL-constrained objective has a closed form, so the reward")
print("can be written IN TERMS OF the policy:")
print("      r(x, y) = beta * log( policy(y|x) / reference(y|x) )")
print("   substitute that back into Bradley-Terry and the reward model")
print("   cancels out. what is left is a loss you can compute from four")
print("   numbers, all of them log-probabilities:")
print("%-34s %s" % ("policy log-prob of the chosen", "pc"))
print("%-34s %s" % ("policy log-prob of the rejected", "pr"))
print("%-34s %s" % ("reference log-prob of the chosen", "rc"))
print("%-34s %s" % ("reference log-prob of the rejected", "rr"))
print("      loss = -log sigmoid( beta * ((pc - rc) - (pr - rr)) )")
print()
print("   work through what that quantity means:")
BETA = 0.1
CASES = [
    ("the policy already prefers the chosen", -1.0, -3.0, -1.5, -2.0),
    ("the policy prefers the REJECTED", -3.0, -1.0, -1.5, -2.0),
    ("policy and reference agree exactly", -1.5, -2.0, -1.5, -2.0),
    ("the policy strongly prefers chosen", -0.5, -6.0, -1.5, -2.0),
]
# d(loss)/d(margin) = -beta * sigmoid(-beta * margin), so the gradient
# is largest exactly where the policy is most wrong
print("%-40s %10s %10s %16s"
      % ("situation", "margin", "loss", "|d loss/d margin|"))
for name, pc, pr, rc, rr in CASES:
    margin = (pc - rc) - (pr - rr)
    loss = -np.log(sig(BETA * margin))
    grad = BETA * sig(-BETA * margin)
    print("%-40s %10.2f %10.4f %16.5f" % (name, margin, loss, grad))
print("   the margin is 'how much more the policy prefers the chosen")
print("   answer than the reference model did'. positive means the policy")
print("   has already moved in the right direction and the loss is small.")
print("   row 3 is the starting point: policy equals reference, margin 0,")
print("   loss = -log(0.5) = %.4f. training begins from exactly there."
      % -np.log(0.5))
print("   and read the gradient column: it is largest where the policy is")
print("   most wrong and smallest where it is already right -- but only by")
print("   %.1fx, which is barely any self-regulation at all."
      % ((BETA * sig(BETA * 2.5)) / (BETA * sig(-BETA * 5.0))))
print("   THAT is what beta controls. it scales the margin before the")
print("   sigmoid, so it decides how far into the sigmoid's flat regions")
print("   the training ever gets:")
print("%-10s %18s %18s %14s"
      % ("beta", "grad, most wrong", "grad, already right", "ratio"))
for bt in (0.05, 0.1, 0.5, 1.0, 4.0):
    hi, lo = bt * sig(bt * 2.5), bt * sig(-bt * 5.0)
    print("%-10.2f %18.5f %18.5f %13.1fx" % (bt, hi, lo, hi / lo))
print("   at beta = 0.05 every pair contributes about equally and the")
print("   policy barely moves from the reference. at beta = 4 the pairs")
print("   the model already gets right stop mattering entirely and it")
print("   trains almost only on its mistakes -- which is faster and drifts")
print("   further from the reference model.")
print("   that is the whole tuning story for DPO: beta is a leash, and")
print("   0.1 to 0.5 is where most people end up.")
print()

print("SO WHAT DID DPO REMOVE?")
print("%-30s %-18s %s" % ("", "RLHF", "DPO"))
for row in (("models in memory", "4", "2"),
            ("a separate reward model", "yes", "no"),
            ("an RL loop", "yes", "no -- plain SGD"),
            ("hyperparameters to tune", "many", "essentially beta"),
            ("what it optimises", "the same objective", "the same objective")):
    print("%-30s %-18s %s" % row)
print("   the last row is the point. DPO is not a different goal reached")
print("   more cheaply; it is the SAME goal with the intermediate model")
print("   algebraically eliminated.")
print("   RLHF keeps one real advantage: a reward model can score answers")
print("   that were never in the preference data, so it can be reused for")
print("   sampling, filtering and evaluation. DPO gives you a policy and")
print("   nothing else.")
print()

print("AND WHAT NEITHER OF THEM FIXES. the three stages do different jobs:")
print("%-24s %-30s %s" % ("stage", "teaches", "cannot fix"))
for row in (("pretraining", "language and world knowledge", "-"),
            ("supervised fine-tuning", "the format of a reply", "missing knowledge"),
            ("preference training", "which reply is preferred", "missing knowledge")):
    print("%-24s %-30s %s" % row)
print("   preference training changes which of the answers a model could")
print("   already produce it prefers to produce. if the right answer was")
print("   never in that set, no amount of preference data puts it there.")
print("   that is why a model that does not know your company's refund")
print("   policy cannot be fine-tuned into knowing it reliably, and why")
print("   the answer to that problem is retrieval rather than training.")

Guided tour

1. Start at pretraining. With SFT at 0 and no preference method, the dense jargon answer is the most likely one. That is what "most probable continuation" looks like when the text you trained on is the internet.
2. Apply SFT. Raise Supervised Fine-Tuning to 2. The demonstrated answer — short, clear, aimed at a child — takes over. Nothing has been rewarded yet; it is just imitation.
3. Add RLHF with a tight leash. Set the method to RLHF with beta at 3. The good answer gains a little, the drift stays small, and human ratings go up.
4. Loosen the leash. Drag beta down towards 0.1. The model collapses onto the long, hedging, flattering answer — because the reward model scores it highest. Reward goes up, the human rating goes down, and the drift readout goes through the roof. That is reward hacking, and it is why the KL penalty is not optional.
5. Fix the reward model instead. Uncheck Reward Model Likes Long Answers and the collapse stops happening, at the same beta. The KL penalty was never the real fix; it was a guard rail against a proxy that was wrong.
6. Compare DPO. Switch the method to DPO. It moves probability towards the preferred answer and away from the rejected one, and it has no reward model to hack — but it also has no opinion at all about answers nobody compared.

Choosing between them

• Wrong facts? None of these. Use retrieval. Fine-tuning teaches behaviour, and treating it as a way to insert knowledge is the most expensive mistake in this area.
• Wrong format, tone or task? SFT, and often LoRA so you are not moving every weight.
• Right content, wrong judgement? Preferences. If you can rank two outputs but cannot write the perfect one, that is exactly the signal these methods consume.
• Nothing but a few examples? Try prompting first. A great deal of what people fine-tune for is a system prompt away.

Failure modes

• Reward hacking. The reward model is a proxy for human judgement, and optimisers find every gap between the two — length, confidence, flattery, formatting.
• Alignment tax. Heavy preference tuning can make a model more agreeable and less capable, hedging where it used to answer.
• Catastrophic forgetting. Fine-tune hard on a narrow task and unrelated abilities degrade. Low learning rates, adapters and mixed data all exist for this.
• Preference data that disagrees with itself. Two annotators, two answers, no shared rubric. The resulting reward model learns the noise.
• Evaluating on the reward model. If the reward model is also your evaluation, you have measured the thing you optimised and learned nothing.

Worth remembering

Pretraining supplies knowledge and fluency, supervised fine-tuning supplies behaviour by imitation, and preference optimisation supplies judgement between answers that are all plausible. RLHF and DPO reach the same optimum — a policy proportional to the reference times the exponentiated reward — with DPO skipping the reward model that RLHF has to train and then defend. The beta term is the leash, and slackening it produces exactly the failure you would predict: reward climbs, human ratings fall, and the model drifts into whatever the proxy overrates. None of these stages is a way to teach a model new facts; that is what retrieval is for.

What each stage cannot fix

This is the practically useful part, because it determines which stage to reach for.

Fine-tuning does not reliably add facts. It can be made to memorise, and doing so is inefficient, hard to update, and produces confident errors on anything nearby that was not memorised. Facts belong in retrieval.

Preference training does not add capability. It shapes which of the model's existing behaviours are expressed. A model that cannot do arithmetic will not learn to from preference data.

Neither fixes a knowledge cut-off. Only retrieval or a tool does.

SFT on a narrow task can cause forgetting. Heavy fine-tuning on one domain degrades general ability — the alignment tax. LoRA reduces it, as does mixing general data into the fine-tuning set.

So the decision table most projects need:

Practical guidance

Try prompting first. With few-shot examples and a clear system prompt, a strong model handles a great many tasks without any training. Fine-tuning is a commitment to a pipeline, a dataset and a retraining cadence.

If fine-tuning, use LoRA. A few hundred to a few thousand well-curated examples, adapters rather than full weights, and evaluation against a held-out set that existed before you started.

Data quality dominates quantity. A thousand carefully written examples beat ten thousand scraped ones. The well-known result here is that a small, high-quality instruction set can produce a surprisingly capable assistant.

Preference optimisation is rarely the first move for an application team. It matters when you are shipping a model whose tone and refusal behaviour you own.

Evaluate on what you care about. Fine-tuning improves the target metric and can degrade others. Keep a general capability check alongside the task-specific one.

Questions people ask

RAG or fine-tuning? RAG for facts, currency and citations. Fine-tuning for format, style and task behaviour. They are complementary and frequently used together.

How much data do I need for SFT? A few hundred examples can shift format noticeably; a few thousand for a substantial task change.

Is DPO strictly better than RLHF? Simpler and comparable in most published comparisons. Well-executed PPO still leads on some benchmarks.

What is the alignment tax? The capability lost when tuning for helpfulness and safety. Real, and usually small relative to the gain.

Can I fine-tune a hosted model? Several providers offer it, usually LoRA-based, with your data and their infrastructure.

Does fine-tuning make a model smaller or faster? No — that is distillation or quantisation. Fine-tuning changes behaviour, not size.

Recap in one screen

• Pretraining gives knowledge, SFT gives format, preference optimisation gives quality and tone.
• Preference data exists because judging two responses is easier and more reliable than writing an ideal one.
• RLHF needs a reward model and an RL loop; DPO achieves comparable results with a supervised loss.
• Fine-tuning does not reliably add facts — use retrieval for that.
• Try prompting first, then LoRA on a few thousand carefully curated examples.