A model does not output words — it outputs a score for every token in its vocabulary. Reshape those scores with temperature, top-k and top-p, and watch the same model turn from deterministic to wildly creative.
Logits are unbounded and can be negative, so they are pushed through softmax. This exponentiates each one and normalises so they sum to 1:
pᵢ = exp(zᵢ / T) / Σ⫺ exp(z⫺ / T)
Because of the rapid, small gaps in logits turn large gaps in chance. A token scoring 2 points higher is not slightly more likely. It is roughly seven times more likely. The temperature T controls how peaked the distribution is: T=1 is the model's native confidence, T<1 sharpens the distribution toward the top token (more deterministic), and T>1 flattens it (more random). After softmax gives chances, a sampling plan picks the next token. Greedy decoding always takes the highest-chance token, which is fast but repetitive. Top-k sampling restricts the choice to the k most likely tokens and samples among them. Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative chance exceeds p. This adapts the candidate set size to the model's confidence. All three plans can be combined with temperature. The model then appends the chosen token to the input and runs the entire sequence through the transformer again to give the next token's logit distribution. This autoregressive loop goes on until a stop token is made or a length limit is reached. This page shows the logit-to-chance conversion, shows temperature scaling, and compares greedy, top-k, and top-p sampling on the same prompt.
A model does not output words — it outputs a score for every token in its vocabulary.
Logits are unbounded and can be negative, so they are pushed through softmax. This exponentiates each one and normalises so they sum to 1:
pᵢ = exp(zᵢ / T) / Σ⫺ exp(z⫺ / T)
Because of the rapid, small gaps in logits turn large gaps in chance. A token scoring 2 points higher is not slightly more likely. It is roughly seven times more likely. The temperature T controls how peaked the distribution is: T=1 is the model's native confidence, T<1 sharpens the distribution toward the top token (more deterministic), and T>1 flattens it (more random). After softmax gives chances, a sampling plan picks the next token. Greedy decoding always takes the highest-chance token, which is fast but repetitive. Top-k sampling restricts the choice to the k most likely tokens and samples among them. Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative chance exceeds p. This adapts the candidate set size to the model's confidence. All three plans can be combined with temperature. The model then appends the chosen token to the input and runs the entire sequence through the transformer again to give the next token's logit distribution. This autoregressive loop goes on until a stop token is made or a length limit is reached. This page shows the logit-to-chance conversion, shows temperature scaling, and compares greedy, top-k, and top-p sampling on the same prompt.
That T in the formula divides each logit before the rapid. This stretches or compresses the gaps between them:
Temperature never changes the ranking, only the confidence spread. Watch the entropy readout as you drag it — that number is the distribution's doubt in bits.
Even after temperature, the long tail of thousands of near-zero tokens still holds meaningful total chance. Sample from it often enough and you in time draw a thing absurd. Two filters prevent this:
After truncating, the survivors are renormalised so they sum to 1 again —. This is why the surviving bars grow when you tighten the filter.
A language model does just one thing: given a sequence of tokens, give a chance distribution over what comes next. All of it else — conversation, code, summarisation — is that step repeated.
"The capital of France is" → { " Paris": 0.89, " a": 0.03, " located": 0.02, … }
Concretely, the final layer gives one logit per token in the vocabulary. An unbounded score, perhaps 100,000 of them. Softmax converts those into chances summing to 1. Then one token is chosen, appended to the sequence, and the whole process repeats.
That loop is called autoregressive generation, and it has a direct result: the model cannot revise. Once a token is emitted it turns part of the input. And a poor early choice constrains all of it after it.
The distribution is not the answer — a thing must pick from it. And the choice of how is what the sampling parameters control.
| Method | Behaviour |
|---|---|
| Greedy | Always the highest-probability token |
| Temperature | Flatten or sharpen the distribution before sampling |
| Top-k | Sample only from the k most likely |
| Top-p (nucleus) | Sample from the smallest set summing to p |
| Beam search | Track several candidate sequences |
Temperature divides the logits before the softmax. Below 1 sharpens the distribution towards the most likely token; above 1 flattens it. At 0 it is greedy.
| Temperature | Use |
|---|---|
| 0 | Extraction, classification, code — anything needing determinism |
| 0.3–0.7 | Factual answering with slight variation |
| 0.8–1.0 | Creative writing |
| >1.2 | Usually incoherent |
Top-p is the more reliable of the truncation methods: it adapts to the distribution's shape. This takes few tokens when the model is confident and many when it is not. Top-k takes a fixed number regardless, which is too restrictive on unclear steps and too permissive on confident ones.
Greedy decoding is not the best answer. Picking the most likely token at each step does not give the most likely sequence. This is why beam search exists. And why greedy output tends towards repetition.
Each new token attends to each last token. Naively, making token 1,000 means recomputing keys and values for all 999 before it. So generation is quadratic in output length.
The KV cache removes that. The keys and values for last tokens do not change. So they are computed once and stored. Each new token computes only its own, appends them, and attends against the cache. Generation turns linear.
The cost is memory: two tensors per layer per token. For a long context and a large model that runs to many gigabytes. And it is why grouped-query attention exists. Sharing key and value projections across heads shrinks the cache a few-fold.
This also explains a visible asymmetry in API pricing and latency. Handling the prompt is one parallel pass over all its tokens; making the output is one sequential step per token. Input is cheap and fast, output is costly and slow.
A language model outputs one number per token in its vocabulary and nothing else. All of it that looks like personality, creativity or caution is a choice about how that list of numbers is turned into a single pick --. This this builds knob by knob.
Those raw logits are not a distribution yet. The next block runs them through softmax to see why a small gap in logits turns a large gap in chance.
Temperature is the knob that reshapes that spread before softmax runs. The next block divides the same logits by a few temperatures and reads off the entropy.
Temperature alone has a limit: it cannot tell a sensible unlikely word from a nonsense one. The next block measures how much wrong-word mass a high temperature lets through.
Cutting the tail off fully is the fix temperature cannot make on its own. The next block applies top-k truncation, then shows why a single fixed k is wrong for both a confident and an unclear step.
Top-p (nucleus) sampling fixes that by adapting the cutoff to the model's own confidence rather than fixing a count. The next block compares it right against the confident and unclear distributions above.
None of those knobs touch the model itself. They all post-process the same fixed list of logits. The final block summarizes what each setting is really for.
The model supplies a distribution; the decoding plan supplies the behaviour. The same weights can give rigid factual answers or freewheeling prose depending purely on temperature and truncation. And because each pick is appended and fed back in, one unlucky sample early on steers all of it that follows —. This is why these knobs matter far more than their obscurity suggests.
Training on "predict the next token" over enormous quantities of text gives powers nobody put in by name.
To predict the next word well, a model must in effect track syntax, resolve guides, hold facts, follow cues embedded in text, and continue patterns. All of those emerge because they reduce prediction error on real text.
In-context learning is the most striking. Show three cases in the prompt and the model goes on the pattern, having learned nothing — no weights changed. It is pattern follow-up, which is just what next-token prediction rewards. Research on "induction heads" traces this to set attention circuits that copy and complete patterns seen earlier in the context.
Two things it does not buy, and both follow from the objective:
Calibrated doubt. The model gives a distribution over tokens, not a belief about truth. High chance means "this is what such text looks like". This is why confident falsehoods are the trait failure.
Arithmetic and character-level reasoning. Tokenisation fragments numbers inconsistently, and letters are not visible as separate units. Counting the r's in "strawberry" asks the model to reason about a thing it cannot see.
A raw next-token predictor goes on text; it does not answer questions. Turning one into an assistant takes two further stages.
Cue tuning — supervised fine-tuning on cases of cues and good responses. This is what teaches the model that a question should be answered rather than continued with more questions.
Choice optimisation — RLHF, DPO or close. Humans rank pairs of responses, and the model is tuned towards the preferred ones. This shapes helpfulness, tone and refusal behaviour.
Neither changes the underlying method. The model is still predicting the next token; what changed is the distribution it was tuned to predict.
Does the model plan ahead? Architecturally it gives one token at a time, and there is evidence that internal states encode data about tokens a few steps ahead. Whether that constitutes planning is an open question.
Why does it repeat itself? Low temperature and greedy decoding favour high-chance continuations, and repetition is high-chance. Raise the temperature, use top-p, or apply a repetition cost.
What temperature should I use? 0 for a thing needing a determinate answer; 0.7 for general use; higher for creative work.
Why is output slower than input? The prompt is handled in one parallel pass; output is one sequential step per token.
Can it see the whole conversation? All of it within the context window. This is re-sent with each request. It has no memory between calls beyond what you include.
Why do the same inputs give different answers? Sampling, unless temperature is 0. And even then, batching and hardware non-determinism can give small changes.
Answer without scrolling back up.
At each step, a language model produces:
The model scores every token it knows. Which one actually gets emitted is a separate sampling decision made on top of that distribution.
Raising the sampling temperature does what?
Low temperature sharpens toward the top token and reads as safe and repetitive. High temperature flattens the odds and reads as creative, or as incoherent once pushed too far.
Why does a model produce fluent text that is confidently wrong?
Next-token prediction rewards text that looks like its training data. A fluent falsehood satisfies that objective just as well as a fluent fact - there is no separate check for truth.
See logits become probabilities through softmax, and watch temperature, top-k and top-p sampling reshape which word the model picks.