vizlearn
Modules / Gen AI / Decoding Lab

How LLMs Predict the Next Word?

Generative AI & LLMsBy Updated

A model does not output words — it outputs a score for every token in its vocabulary. Reshape those scores with temperature, top-k and top-p, and watch the same model turn from deterministic to wildly creative.

Overview

Step 1: Softmax Turns Scores into Probabilities

Logits are unbounded and can be negative, so they are pushed through softmax, which exponentiates each one and normalises so they sum to 1:

pᵢ = exp(zᵢ / T) / Σ⫺ exp(z⫺ / T)

Because of the exponential, small gaps in logits become large gaps in probability. A token scoring 2 points higher is not slightly more likely — it is roughly seven times more likely.

Vocabulary Scores

raw logit final probability

Generated so far

From Logits to Words: How the Next Token Is Chosen

Step 2: Temperature Reshapes Confidence

That T in the formula divides every logit before the exponential, which stretches or compresses the gaps between them:

  • T < 1 — gaps widen, the leader runs away with it. Output becomes focused and repetitive. As T approaches 0 this is identical to always taking the argmax.
  • T = 1 — the model's own distribution, untouched.
  • T > 1 — gaps compress toward uniform. Unlikely tokens become genuinely reachable, which reads as creativity right up until it reads as nonsense.

Temperature never changes the ranking, only the confidence spread. Watch the entropy readout as you drag it — that number is the distribution's uncertainty in bits.

Step 3: Truncation — Top-k and Top-p

Even after temperature, the long tail of thousands of near-zero tokens still holds meaningful total probability. Sample from it often enough and you eventually draw something absurd. Two filters prevent this:

  • Top-k keeps the k highest-scoring tokens and discards the rest. Simple, but rigid: k=5 is far too narrow when the model is genuinely uncertain and far too wide when only one answer is correct.
  • Top-p (nucleus) keeps the smallest set whose probabilities sum to p. The cutoff adapts: for "The capital of France is" the nucleus may hold one token, while for an open-ended prompt it may hold forty.

After truncating, the survivors are renormalised so they sum to 1 again — which is why the surviving bars grow when you tighten the filter.

One token at a time

A language model does exactly one thing: given a sequence of tokens, produce a probability distribution over what comes next. Everything else — conversation, code, summarisation — is that operation repeated.

"The capital of France is" → { " Paris": 0.89, " a": 0.03, " located": 0.02, … }

Concretely, the final layer produces one logit per token in the vocabulary — an unbounded score, perhaps 100,000 of them. Softmax converts those into probabilities summing to 1. Then one token is chosen, appended to the sequence, and the whole process repeats.

That loop is called autoregressive generation, and it has a direct consequence: the model cannot revise. Once a token is emitted it becomes part of the input, and a poor early choice constrains everything after it.

How the token is chosen

The distribution is not the answer — something must pick from it, and the choice of how is what the sampling parameters control.

MethodBehaviour
GreedyAlways the highest-probability token
TemperatureFlatten or sharpen the distribution before sampling
Top-kSample only from the k most likely
Top-p (nucleus)Sample from the smallest set summing to p
Beam searchTrack several candidate sequences

Temperature divides the logits before the softmax. Below 1 sharpens the distribution towards the most likely token; above 1 flattens it. At 0 it is greedy.

TemperatureUse
0Extraction, classification, code — anything needing determinism
0.3–0.7Factual answering with slight variation
0.8–1.0Creative writing
>1.2Usually incoherent

Top-p is the more robust of the truncation methods: it adapts to the distribution's shape, taking few tokens when the model is confident and many when it is not. Top-k takes a fixed number regardless, which is too restrictive on uncertain steps and too permissive on confident ones.

Greedy decoding is not the best answer. Picking the most likely token at each step does not produce the most likely sequence, which is why beam search exists — and why greedy output tends towards repetition.

Why generation gets slower with length, and how caching fixes it

Each new token attends to every previous token. Naively, generating token 1,000 means recomputing keys and values for all 999 before it, so generation is quadratic in output length.

The KV cache removes that. The keys and values for previous tokens do not change, so they are computed once and stored. Each new token computes only its own, appends them, and attends against the cache. Generation becomes linear.

The cost is memory: two tensors per layer per token. For a long context and a large model that runs to many gigabytes, and it is why grouped-query attention exists — sharing key and value projections across heads shrinks the cache several-fold.

This also explains a visible asymmetry in API pricing and latency. Processing the prompt is one parallel pass over all its tokens; generating the output is one sequential step per token. Input is cheap and fast, output is expensive and slow.

One distribution over the vocabulary, sampled

A language model outputs one number per token in its vocabulary and nothing else. Everything that looks like personality, creativity or caution is a choice about how that list of numbers is turned into a single pick -- which this builds knob by knob.

example_01.pyNumPy
Output

                

                

                    
Try it yourself

                    
  1. Keep the France prompt and press Sample repeatedly at T = 1. The correct answer dominates, so you nearly always get it — but not quite always.
  2. Push temperature to 2.0 and sample again. The distribution flattens and the model starts confidently naming the wrong city. This is exactly how hallucination looks at the token level.
  3. Drop temperature to 0.05. One bar takes essentially all the probability mass; sampling and greedy become the same thing.
  4. Set top-p to 0.9 and switch prompts. Note how many candidates survive on the factual prompt versus the open-ended one — same setting, completely different cutoff. That adaptivity is why nucleus sampling largely replaced top-k.
  5. Set top-k to 1. Every filter collapses to greedy decoding regardless of temperature — proof that truncation, not temperature, is what removes randomness entirely.

                

                

                    
In one line

                    
The model supplies a distribution; the decoding strategy supplies the behaviour. Identical weights can produce rigid factual answers or freewheeling prose depending purely on temperature and truncation. And because each pick is appended and fed back in, one unlucky sample early on steers everything that follows — which is why these knobs matter far more than their obscurity suggests.

                

                

                    
What next-token prediction turns out to buy

                    
Training on "predict the next token" over enormous quantities of text produces capabilities nobody put in explicitly.
To predict the next word well, a model must in effect track syntax, resolve references, hold facts, follow instructions embedded in text, and continue patterns. All of those emerge because they reduce prediction error on real text.
In-context learning is the most striking. Show three examples in the prompt and the model continues the pattern, having learned nothing — no weights changed. It is pattern continuation, which is exactly what next-token prediction rewards. Research on "induction heads" traces this to specific attention circuits that copy and complete patterns seen earlier in the context.
Two things it does not buy, and both follow from the objective:
Calibrated uncertainty. The model produces a distribution over tokens, not a belief about truth. High probability means "this is what such text looks like", which is why confident falsehoods are the characteristic failure.
Arithmetic and character-level reasoning. Tokenisation fragments numbers inconsistently, and letters are not visible as separate units. Counting the r's in "strawberry" asks the model to reason about something it cannot see.

                

                

                    
From base model to assistant

                    
A raw next-token predictor continues text; it does not answer questions. Turning one into an assistant takes two further stages.
Instruction tuning — supervised fine-tuning on examples of instructions and good responses. This is what teaches the model that a question should be answered rather than continued with more questions.
Preference optimisation — RLHF, DPO or similar. Humans rank pairs of responses, and the model is tuned towards the preferred ones. This shapes helpfulness, tone and refusal behaviour.
Neither changes the underlying mechanism. The model is still predicting the next token; what changed is the distribution it was tuned to predict.

                

                

                    
Questions people ask

                    
Does the model plan ahead? Architecturally it produces one token at a time, and there is evidence that internal states encode information about tokens several steps ahead. Whether that constitutes planning is an open question.
Why does it repeat itself? Low temperature and greedy decoding favour high-probability continuations, and repetition is high-probability. Raise the temperature, use top-p, or apply a repetition penalty.
What temperature should I use? 0 for anything requiring a determinate answer; 0.7 for general use; higher for creative work.
Why is output slower than input? The prompt is processed in one parallel pass; output is one sequential step per token.
Can it see the whole conversation? Everything within the context window, which is re-sent with every request. It has no memory between calls beyond what you include.
Why do the same inputs give different answers? Sampling, unless temperature is 0 — and even then, batching and hardware non-determinism can produce small variations.

                

                

                    
Recap in one screen

                    
  • The model outputs a probability distribution over the next token; generation appends one and repeats.
  • Temperature, top-k and top-p decide how that distribution is sampled — 0 for determinism, higher for variety.
  • Greedy decoding is not optimal for the sequence, and it tends to repeat.
  • The KV cache makes generation linear rather than quadratic, at a large memory cost.
  • In-context learning is pattern continuation; confident falsehoods follow from optimising likelihood rather than truth.

                

            


        

    

    


        
    



Check yourself
0 of 3
Answer without scrolling back up.
  1. At each step, a language model produces:
  2. Raising the sampling temperature does what?
  3. Why does a model produce fluent text that is confidently wrong?





 Cheat sheet
How LLMs Predict the Next Word?
A model does not output words — it outputs a score for every token in its vocabulary. Reshape those scores with temperature, top-k and top-p, and watch the same model turn from deterministic to wildly creative.
GEN AI · vizlearn.in/gen_ai/how_llms_predict_next_word.html
More in Gen AI
Gen AIWhat are Queries, Keys and Values in an LLM?Gen AICausal Language ModelingGen AIWhat is Masked Language Modeling?Gen AIQuantization in LLMs
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact