Train a model on custom data and visualize how it predicts the next token based on learned probability distributions.
Overview
Overview
Causal Language Modeling (CLM) is the fundamental principle behind how generative AI models like GPT work. At its core, it's a simple yet powerful task: predicting the next word (or "token") in a sequence given the words that came before it. This interactive lab demystifies that process, allowing you to train your own mini-model and see exactly how it makes decisions.
Generated Output
LIVE INFERENCE
|
Next Token learned Distribution
Generate next token to see probabilities
Distribution ScrollView
Understanding Causal Language Modeling
Causal Language Modeling (CLM) is the fundamental principle behind how generative AI models like GPT work. At its core, it's a simple yet powerful task: predicting the next word (or "token") in a sequence given the words that came before it. This interactive lab demystifies that process, allowing you to train your own mini-model and see exactly how it makes decisions.
Quick Context: Learning from Data
A language model learns by analyzing vast amounts of text. It doesn't understand text like a human does; instead, it learns statistical patterns. The process, which you can simulate in this lab, involves:
Tokenization: The input text is broken down into smaller units called tokens (often words or sub-words).
Building a Probability Distribution: The model analyzes which tokens tend to follow other tokens. For any given sequence of tokens (the "context"), it builds a list of all possible next tokens and assigns a probability to each one. For example, after the context "The sky is", the token "blue" will have a very high probability.
When you click the "Update Learned Probabilities" button, you are running this training process on the text you've provided. The model builds its statistical understanding based only on that data.
Core Idea: Probabilistic Prediction
Once trained, the model generates text one token at a time. Given your input prompt, it looks at the sequence and calculates the probability distribution for the very next token. This is what you see in the "Next Token Predictions" list. Each bar represents the model's "confidence" that a particular token should come next.
Greedy Decoding
The simplest method is to always choose the token with the highest probability. This is called "greedy decoding." It's predictable but can lead to repetitive and boring text.
The Role of Temperature
To add creativity, we introduce randomness via a "temperature" setting. A low temperature makes the model's choices more conservative (closer to greedy), while a high temperature increases randomness, allowing it to pick less likely tokens and produce more surprising (but potentially less coherent) text.
Predicting forward, and only forward
Causal language modelling is the training objective behind every generative model: predict the next token from the tokens before it, and never look ahead.
P(w₁, w₂, …, wₙ) = ∏ᵗ P(wᵗ | w₁ … wᵗ₋₁)
The word "causal" refers to that one-directional flow: token t may depend on its past and never on its future. (It is often written "casual" by accident, including in some published code.)
The mechanism enforcing it is a mask. Attention scores for future positions are set to negative infinity before the softmax, so their weights become exactly zero:
the cat sat on
the . -inf -inf -inf
cat . . -inf -inf
sat . . . -inf
on . . . .
Without that mask, predicting "cat" from a context including "cat" would be trivial at training time and impossible at inference, where the future does not exist.
Why one pass trains every position
The masking has an efficiency consequence that is easy to miss and is central to why this objective scales.
For a sequence of 1,000 tokens, a single forward pass produces predictions at all 1,000 positions simultaneously, each correctly conditioned only on its own past. The loss is averaged over all of them.
So one pass over a document yields as many training signals as the document has tokens. No labelling, no annotation — any text at all is training data, and the supervision is free.
That property is the reason next-token prediction won. The available training set is the written output of humanity, and every token in it is a labelled example.
Causal LM
Masked LM
Sees
Past only
Both directions
Predictions per pass
Every position
Only the masked 15%
Can generate
Yes
No
Natural at
Generation
Understanding
Models
GPT, Llama, Claude
BERT, RoBERTa
The second row is a real efficiency difference: masked language modelling extracts a training signal from roughly 15% of tokens per pass, causal modelling from all of them.
What emerges from it
Trained at sufficient scale, next-token prediction produces capabilities nobody specified.
To predict text well a model must track syntax, resolve pronouns, hold facts, maintain consistency over long passages, and continue patterns. All of those reduce prediction error, so all of them are learned.
In-context learning is the most striking. Show three worked examples in the prompt and the model continues the pattern with no weight updates at all. That is pattern continuation, which is precisely what the objective rewards — and research on "induction heads" traces the ability to identifiable attention circuits that copy and complete earlier patterns.
Two things it does not produce, and both follow from the objective. Truthfulness, because likelihood is not truth: a plausible falsehood scores well. And calibrated uncertainty, because the model outputs a distribution over tokens rather than a belief about the world.
One pass, every position, and the mask that makes it work
The claim that a single forward pass trains every position at once is the reason causal language modelling scales. This builds the mask that makes it true, checks that it actually prevents the model from reading ahead, and counts what the alternative would have cost.
example_01.pyNumPy
import numpy as np
TOKENS = ["the", "cat", "sat", "on", "the", "mat"]
T = len(TOKENS)
print("A SEQUENCE OF %d TOKENS. training means predicting each one from" % T)
print("everything before it -- which is %d separate prediction problems:" % (T - 1))
for i in range(1, T):
print(" given %-26s predict %r"
% ("'" + " ".join(TOKENS[:i]) + "'", TOKENS[i]))
print()
print("THE NAIVE READING is that these are %d forward passes. they are"
% (T - 1))
print("not -- they are ONE, and the causal mask is what makes that")
print("legitimate.")
print()
mask = np.tril(np.ones((T, T)))
print("THE MASK: position i may attend to positions 0..i and no further.")
print(" %s" % " ".join("%5s" % t for t in TOKENS))
for i in range(T):
print("%-6s %s" % (TOKENS[i],
" ".join("%5s" % ("YES" if mask[i, j] else ".")
for j in range(T))))
print(" the upper triangle is the future, and it is switched off. in")
print(" the implementation the masked scores are set to -infinity")
print(" BEFORE the softmax, so they receive exactly zero weight rather")
print(" than a small one:")
scores = np.array([2.0, 1.0, 3.0, 0.5])
for name, m in (("no mask", np.array([1, 1, 1, 1])),
("masked after softmax", None),
("masked before (correct)", np.array([1, 1, 0, 0]))):
if m is None:
e = np.exp(scores - scores.max()); p = e / e.sum()
p = p * np.array([1, 1, 0, 0])
print(" %-26s %s sums to %.4f"
% (name, " ".join("%.4f" % v for v in p), p.sum()))
else:
s = np.where(m == 1, scores, -np.inf)
e = np.exp(s - np.nanmax(s[np.isfinite(s)])); p = e / e.sum()
print(" %-26s %s sums to %.4f"
% (name, " ".join("%.4f" % v for v in p), p.sum()))
print(" zeroing after the softmax leaves the row summing to less than 1,")
print(" so the surviving weights are wrong as well as incomplete. the")
print(" -infinity has to go in first.")
print()
print("NOW CHECK THAT IT ACTUALLY WORKS. run a toy attention twice --")
print("once on the real sequence, once with a LATER token replaced --")
print("and see which outputs move:")
rng = np.random.default_rng(2)
D = 8
E = rng.normal(0, 1, (T, D))
def attend(X, causal):
s = X @ X.T / np.sqrt(D)
if causal:
s = np.where(np.tril(np.ones((T, T))) == 1, s, -np.inf)
s = s - s.max(axis=1, keepdims=True)
w = np.exp(s)
w = w / w.sum(axis=1, keepdims=True)
return w @ X
E2 = E.copy()
E2[4] = rng.normal(0, 1, D) # change token 4 only
print("%-10s %20s %20s" % ("position", "causal: output moved", "no mask: moved"))
a1, a2 = attend(E, True), attend(E2, True)
b1, b2 = attend(E, False), attend(E2, False)
for i in range(T):
print("%-10d %20.2e %20.2e"
% (i, np.abs(a1[i] - a2[i]).max(), np.abs(b1[i] - b2[i]).max()))
print(" with the causal mask, positions 0 to 3 are BIT-IDENTICAL after a")
print(" later token changed. without it, every position moved.")
print(" that is the property the whole scheme rests on: position i's")
print(" prediction cannot have seen token i+1, so all %d predictions can"
% (T - 1))
print(" be computed in the same pass and scored against the same")
print(" sequence, honestly.")
print()
print("WHAT THAT SAVES. suppose you did it the naive way instead:")
print("%-16s %18s %18s %14s"
% ("sequence length", "naive passes", "with the mask", "speedup"))
for n in (6, 128, 1024, 8192):
print("%-16s %18s %18d %13dx"
% ("{:,}".format(n), "{:,}".format(n - 1), 1, n - 1))
print(" the mask turns training from O(sequence length) forward passes")
print(" into ONE. that is not an optimisation -- at %s tokens per"
% "{:,}".format(8192))
print(" sequence it is the difference between possible and not.")
print()
print("AND THE LOSS IS AN AVERAGE OVER ALL OF THEM. one number per")
print("sequence, from %d predictions:" % (T - 1))
probs = [0.62, 0.31, 0.88, 0.45, 0.07]
print("%-30s %14s %14s" % ("predicting", "P(correct)", "-log P"))
for (i, p) in zip(range(1, T), probs):
print("%-30s %14.4f %14.4f" % (repr(TOKENS[i]), p, -np.log(p)))
loss = float(np.mean([-np.log(p) for p in probs]))
print("%-30s %14s %14.4f" % ("mean cross-entropy loss", "", loss))
print("%-30s %14s %14.4f" % ("perplexity = exp(loss)", "", np.exp(loss)))
print(" perplexity %.2f means the model was, on average, as uncertain as"
% np.exp(loss))
print(" if it had been choosing uniformly among %.2f options. that is"
% np.exp(loss))
print(" the whole interpretation, and it is why perplexity is quoted")
print(" rather than the raw loss -- it has units you can picture.")
print(" note also that the WORST prediction here (%r at %.2f) dominates"
% (TOKENS[5], probs[-1]))
print(" the average: -log(%.2f) = %.2f against -log(%.2f) = %.2f."
% (probs[-1], -np.log(probs[-1]), max(probs), -np.log(max(probs))))
print(" cross-entropy punishes confident mistakes far harder than it")
print(" rewards confident successes, which is exactly what you want")
print(" from a training signal.")
Output
Things to try
Use the controls to build intuition for how these models behave.
Train on Different Data: Clear the training data and use one of the samples. Train on the "Nature Description" text. Now, give it the prompt "The ocean is". The predictions will likely be words like "vast", "blue", or "deep". Now, load and train on the "Tech Support Logs" and use the same prompt. The predictions will be completely different, demonstrating that the model's knowledge is entirely dependent on its training data.
Experiment with Temperature: Set the temperature to its lowest value (0.1). Generate a few tokens. The output will be very predictable. Now, slide the temperature to a high value (like 1.2) and generate again. Notice how the model starts making more creative and sometimes nonsensical choices. This is the trade-off between coherence and creativity.
The "Stuck in a Loop" Problem: With a simple model and low temperature, you might see the model get stuck in a repetitive loop (e.g., "is a is a is a..."). This is a classic problem in language generation that more advanced techniques (like top-k or nucleus sampling) help solve.
Worth remembering
Models are Statistical Parrots: They learn patterns, not meaning. Their output is based on the probability of what token should come next based on the data they were trained on.
Generation is One Token at a Time: Text is generated sequentially. The model predicts the next token, adds it to the sequence, and then uses that new, longer sequence to predict the token after that.
Temperature Controls Creativity: Temperature is a key parameter for controlling the randomness and "creativity" of the generated output.
Context is King: The model's predictions are entirely dependent on the preceding tokens (the context). Changing even one word in the prompt can drastically alter the probability distribution for the next token.
From base model to assistant
A causal language model trained only on this objective is a base model. It continues text. Ask it a question and it may produce more questions, because that is what a list of questions looks like in the training data.
Two further stages make it an assistant:
Supervised fine-tuning on instruction-response pairs teaches the format — that a question should be answered, that output has a structure, that a conversation has turns.
Preference optimisation (RLHF or DPO) tunes helpfulness, tone and refusal behaviour from human comparisons.
Neither changes the mechanism. The model is still predicting the next token; the distribution it has been tuned to predict is different.
This distinction matters practically: base models and instruction-tuned models behave very differently on the same prompt, and a base model is usually the wrong thing to build a product on.
Sampling: turning a distribution into text
The model gives probabilities; something must choose.
Setting
Effect
Temperature 0
Always the most likely token — deterministic
Temperature 0.7
Moderate variety
Temperature >1.2
Usually incoherent
Top-p 0.9
Sample from the smallest set summing to 0.9
Repetition penalty
Down-weight tokens already used
Two facts worth carrying. Greedy decoding is not optimal — picking the most likely token at each step does not give the most likely sequence, which is why beam search exists and why greedy output tends to repeat. And temperature 0 is the right default for anything factual: variety in a factual answer is error.
Questions people ask
Why "causal"? Because information flows in one direction only, past to future — the same sense as in causal systems. It is frequently misspelled "casual".
Is this the same as autoregressive? Effectively yes. "Causal" describes the masking; "autoregressive" describes feeding outputs back as inputs.
Why not train bidirectionally and generate anyway? A bidirectional model has no notion of "next", and its training task assumes the surrounding context exists. Attempts to generate by iterative filling work poorly.
Does the model plan the whole sentence? Architecturally it emits one token at a time, and there is evidence internal states encode information about upcoming tokens. Whether that is planning is an open question.
Why does it repeat itself? Low temperature and greedy decoding favour high-probability continuations, and repetition is high-probability. Raise temperature, use top-p, or apply a repetition penalty.
Can a causal model do classification? Yes — by prompting, or by reading the final hidden state. At scale it matches encoders on most understanding benchmarks despite the one-directional constraint.
Recap in one screen
Predict the next token from the past only; a causal mask sets future attention scores to negative infinity.
One forward pass trains every position at once, and any text is training data — which is why the objective scales.
Generation is the same model applied repeatedly, appending its own output.
Capabilities including in-context learning emerge from the objective rather than being specified.
A base model continues text; instruction tuning and preference optimisation make it an assistant.
Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Models are Statistical Parrots” here?
They learn patterns, not meaning. Their output is based on the probability of what token should come next based on the data they were trained on.
What is meant by “Generation is One Token at a Time” here?
Text is generated sequentially. The model predicts the next token, adds it to the sequence, and then uses that new, longer sequence to predict the token after that.
What is meant by “Temperature Controls Creativity” here?
Temperature is a key parameter for controlling the randomness and "creativity" of the generated output.
What is meant by “Context is King” here?
The model's predictions are entirely dependent on the preceding tokens (the context). Changing even one word in the prompt can drastically alter the probability distribution for the next token.
Cheat sheet
Causal Language Modeling
Causal Language Modeling (CLM) is the fundamental principle behind how generative AI models like GPT work. At its core, it's a simple yet powerful task: predicting the next word (or "token") in a sequence given the words that came before it. This interactive lab demystifies that process, allowing you to train your own mini-model and see exactly how it makes decisions.
GEN AI · vizlearn.in/gen_ai/casual_language_modeling.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.