Modules / Gen AI / Decoding Lab

How LLMs Predict the Next Word?

A model does not output words — it outputs a score for every token in its vocabulary. Reshape those scores with temperature, top-k and top-p, and watch the same model turn from deterministic to wildly creative.

1. Context

These raw logits are illustrative values for a plausible model — every transformation applied to them below is computed exactly.

2. Decoding Controls

Temperature 1.00

0.05 sharp2.0 flat

Top-k off

Top-p (nucleus) 1.00

Distribution Metrics

Top-1 Prob0%

Entropy0

Candidates Kept0

Last Pick–

Vocabulary Scores

raw logit final probability

Generated so far

From Logits to Words: How the Next Token Is Chosen

Ask an LLM to continue a sentence and it does not pick a word. It emits one raw score — a logit — for every token in its vocabulary, often 50,000 or more. Everything that feels like personality, creativity or repetitiveness happens in the small amount of maths applied after that.

Step 1: Softmax Turns Scores into Probabilities

Logits are unbounded and can be negative, so they are pushed through softmax, which exponentiates each one and normalises so they sum to 1:

pᵢ = exp(zᵢ / T) / Σ⫺ exp(z⫺ / T)

Because of the exponential, small gaps in logits become large gaps in probability. A token scoring 2 points higher is not slightly more likely — it is roughly seven times more likely.

Step 2: Temperature Reshapes Confidence

That T in the formula divides every logit before the exponential, which stretches or compresses the gaps between them:

T < 1 — gaps widen, the leader runs away with it. Output becomes focused and repetitive. As T approaches 0 this is identical to always taking the argmax.
T = 1 — the model's own distribution, untouched.
T > 1 — gaps compress toward uniform. Unlikely tokens become genuinely reachable, which reads as creativity right up until it reads as nonsense.

Temperature never changes the ranking, only the confidence spread. Watch the entropy readout as you drag it — that number is the distribution's uncertainty in bits.

Step 3: Truncation — Top-k and Top-p

Even after temperature, the long tail of thousands of near-zero tokens still holds meaningful total probability. Sample from it often enough and you eventually draw something absurd. Two filters prevent this:

Top-k keeps the k highest-scoring tokens and discards the rest. Simple, but rigid: k=5 is far too narrow when the model is genuinely uncertain and far too wide when only one answer is correct.
Top-p (nucleus) keeps the smallest set whose probabilities sum to p. The cutoff adapts: for "The capital of France is" the nucleus may hold one token, while for an open-ended prompt it may hold forty.

After truncating, the survivors are renormalised so they sum to 1 again — which is why the surviving bars grow when you tighten the filter.

Interactive Exploration Guide

Keep the France prompt and press Sample repeatedly at T = 1. The correct answer dominates, so you nearly always get it — but not quite always.
Push temperature to 2.0 and sample again. The distribution flattens and the model starts confidently naming the wrong city. This is exactly how hallucination looks at the token level.
Drop temperature to 0.05. One bar takes essentially all the probability mass; sampling and greedy become the same thing.
Set top-p to 0.9 and switch prompts. Note how many candidates survive on the factual prompt versus the open-ended one — same setting, completely different cutoff. That adaptivity is why nucleus sampling largely replaced top-k.
Set top-k to 1. Every filter collapses to greedy decoding regardless of temperature — proof that truncation, not temperature, is what removes randomness entirely.

Key Takeaway

The model supplies a distribution; the decoding strategy supplies the behaviour. Identical weights can produce rigid factual answers or freewheeling prose depending purely on temperature and truncation. And because each pick is appended and fed back in, one unlucky sample early on steers everything that follows — which is why these knobs matter far more than their obscurity suggests.

Cheat sheet