Train a model on custom data and visualize how it predicts the next token based on learned probability distributions.
Generated Output
LIVE INFERENCE
|
Next Token learned Distribution
Generate next token to see probabilities
Distribution ScrollView
Overview
How GPT-style models learn to write
Causal Language Modeling (CLM) is the basic principle behind how generative AI models like GPT work. At its core, it's a simple yet powerful task: predicting the next word (or "token") in a sequence given the words that came before it. This interactive lab explains that process. This allows you to train your own mini-model and see just how it makes decisions. The "causal" in the name means the model can just look left. When predicting token t, it sees tokens 1 through t-1 but never t+1 or beyond. This is enforced by a causal mask in the attention layer that sets future spots to negative infinity before the softmax. Contrast this with masked language modelling, where the model sees both sides of a blank and fills it in — that is the pretraining objective behind BERT and is better for grasp. But it cannot generate text since it has no notion of left-to-right sorting. CLM training is self-supervised: the training data is raw text with no labels. And each token's label is simply the next token in the sequence. The loss is cross-entropy averaged over all spots. And the model learns a chance distribution over the full vocabulary at each step. At generation time, a sampling plan (temperature, top-k, top-p) picks from that distribution to give the next token. And the process repeats autoregressively.
Train a model on custom data and visualize how it predicts the next token based on learned probability distributions.
Causal Language Modeling (CLM) is the fundamental principle behind how generative AI models like GPT work. At its core, it's a simple yet powerful task: predicting the next word (or "token") in a sequence given the words that came before it. This interactive lab explains that process, allowing you to train your own mini-model and see exactly how it makes decisions.
How GPT-style models learn to write
Causal Language Modeling (CLM) is the basic principle behind how generative AI models like GPT work. At its core, it's a simple yet powerful task: predicting the next word (or "token") in a sequence given the words that came before it. This interactive lab explains that process. This allows you to train your own mini-model and see just how it makes decisions. The "causal" in the name means the model can just look left. When predicting token t, it sees tokens 1 through t-1 but never t+1 or beyond. This is enforced by a causal mask in the attention layer that sets future spots to negative infinity before the softmax. Contrast this with masked language modelling, where the model sees both sides of a blank and fills it in — that is the pretraining objective behind BERT and is better for grasp. But it cannot generate text since it has no notion of left-to-right sorting. CLM training is self-supervised: the training data is raw text with no labels. And each token's label is simply the next token in the sequence. The loss is cross-entropy averaged over all spots. And the model learns a chance distribution over the full vocabulary at each step. At generation time, a sampling plan (temperature, top-k, top-p) picks from that distribution to give the next token. And the process repeats autoregressively.
Quick Context: Learning from Data
A language model learns by studying vast amounts of text. It doesn't understand text like a human does; instead, it learns numeric patterns. The process, which you can simulate in this lab, involves:
Tokenization: The input text is broken down into smaller units called tokens (often words or sub-words).
Building a Chance Distribution: The model studies which tokens tend to follow other tokens. For any given sequence of tokens (the "context"), it builds a list of all doable next tokens and assigns a chance to each one. For case, after the context "The sky is", the token "blue" will have a very high chance.
When you click the "Update Learned Chances" button, you are running this training process on the text you've provided. The model builds its numeric grasp based just on that data.
Core Idea: Probabilistic Prediction
Once trained, the model makes text one token at a time. Given your input prompt, it looks at the sequence and finds the chance distribution for the very next token. This is what you see in the "Next Token Predictions" list. Each bar represents the model's "confidence" that a given token should come next.
Greedy Decoding
The simplest method is to always choose the token with the highest chance. This is called "greedy decoding." It's predictable but can lead to repetitive and boring text.
The Role of Temperature
To add creativity, we add randomness via a "temperature" setting. A low temperature makes the model's choices more conservative (closer to greedy). While a high temperature raises randomness. This allows it to pick less likely tokens and give more surprising (but perhaps less coherent) text.
Predicting forward, and just forward
Causal language modelling is the training objective behind each generative model. Predict the next token from the tokens before it, and never look ahead.
P(w₁, w₂, …, wₙ) = ∏ᵗ P(wᵗ | w₁ … wᵗ₋₁)
The word "causal" means that one-one-way flow: token t may depend on its past and never on its future. (It is often written "casual" by accident, with in some published code.)
The method enforcing it is a mask. Attention scores for future spots are set to negative infinity before the softmax. So their weights turn just zero:
the cat sat on
the . -inf -inf -inf
cat . . -inf -inf
sat . . . -inf
on . . . .
Without that mask, predicting "cat" from a context with "cat" would be trivial at training time and not doable at inference. Where the future does not exist.
Why one pass trains each spot
The masking has an speed result that is easy to miss and is central to why this objective scales.
For a sequence of 1,000 tokens, a single forward pass gives predictions at all 1,000 spots at once, each correctly conditioned just on its own past. The loss is averaged over all of them.
So one pass over a document yields as many training signs as the document has tokens. No labelling, no annotation — any text at all is training data. And the supervision is free.
That property is the reason next-token prediction won. The available training set is the written output of humanity. And each token in it is a labelled case.
Causal LM
Masked LM
Sees
Past only
Both directions
Predictions per pass
Every position
Only the masked 15%
Can generate
Yes
No
Natural at
Generation
Understanding
Models
GPT, Llama, Claude
BERT, RoBERTa
The second row is a real speed gap: masked language modelling extracts a training sign from roughly 15% of tokens per pass, causal modelling from all of them.
What emerges from it
Trained at enough scale, next-token prediction gives powers nobody specified.
To predict text well a model must track syntax, resolve pronouns, hold facts, maintain consistency over long passages, and continue patterns. All of those reduce prediction error, so all of them are learned.
In-context learning is the most striking. Show three worked cases in the prompt and the model goes on the pattern with no weight updates at all. That is pattern follow-up, which is just what the objective rewards. And research on "induction heads" traces the ability to identifiable attention circuits that copy and complete earlier patterns.
Two things it does not give, and both follow from the objective. Truthfulness, since likelihood is not truth: a plausible falsehood scores well. And calibrated doubt, since the model outputs a distribution over tokens rather than a belief about the world.
One pass, each spot, and the mask that makes it work
The claim that a single forward pass trains each spot at once is the reason causal language modelling scales. This builds the mask that makes it true, checks that it really prevents the model from reading ahead, and counts what the other would have cost.
example_01.pyNumPy
import numpy as np
TOKENS = ["the", "cat", "sat", "on", "the", "mat"]
T = len(TOKENS)
print("A SEQUENCE OF %d TOKENS. training means predicting each one from" % T)
print("everything before it -- which is %d separate prediction problems:" % (T - 1))
for i in range(1, T):
print(" given %-26s predict %r"
% ("'" + " ".join(TOKENS[:i]) + "'", TOKENS[i]))
print()
print("THE NAIVE READING is that these are %d forward passes. they are"
% (T - 1))
print("not -- they are ONE, and the causal mask is what makes that")
print("legitimate.")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
TOKENS = ["the", "cat", "sat", "on", "the", "mat"]
T = len(TOKENS)
mask = np.tril(np.ones((T, T)))
print("THE MASK: position i may attend to positions 0..i and no further.")
print(" %s" % " ".join("%5s" % t for t in TOKENS))
for i in range(T):
print("%-6s %s" % (TOKENS[i],
" ".join("%5s" % ("YES" if mask[i, j] else ".")
for j in range(T))))
print(" the upper triangle is the future, and it is switched off. in")
print(" the implementation the masked scores are set to -infinity")
print(" BEFORE the softmax, so they receive exactly zero weight rather")
print(" than a small one:")
scores = np.array([2.0, 1.0, 3.0, 0.5])
for name, m in (("no mask", np.array([1, 1, 1, 1])),
("masked after softmax", None),
("masked before (correct)", np.array([1, 1, 0, 0]))):
if m is None:
e = np.exp(scores - scores.max()); p = e / e.sum()
p = p * np.array([1, 1, 0, 0])
print(" %-26s %s sums to %.4f"
% (name, " ".join("%.4f" % v for v in p), p.sum()))
else:
s = np.where(m == 1, scores, -np.inf)
e = np.exp(s - np.nanmax(s[np.isfinite(s)])); p = e / e.sum()
print(" %-26s %s sums to %.4f"
% (name, " ".join("%.4f" % v for v in p), p.sum()))
print(" zeroing after the softmax leaves the row summing to less than 1,")
print(" so the surviving weights are wrong as well as incomplete. the")
print(" -infinity has to go in first.")
print()
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
TOKENS = ["the", "cat", "sat", "on", "the", "mat"]
T = len(TOKENS)
print("NOW CHECK THAT IT ACTUALLY WORKS. run a toy attention twice --")
print("once on the real sequence, once with a LATER token replaced --")
print("and see which outputs move:")
rng = np.random.default_rng(2)
D = 8
E = rng.normal(0, 1, (T, D))
def attend(X, causal):
s = X @ X.T / np.sqrt(D)
if causal:
s = np.where(np.tril(np.ones((T, T))) == 1, s, -np.inf)
s = s - s.max(axis=1, keepdims=True)
w = np.exp(s)
w = w / w.sum(axis=1, keepdims=True)
return w @ X
E2 = E.copy()
E2[4] = rng.normal(0, 1, D) # change token 4 only
print("%-10s %20s %20s" % ("position", "causal: output moved", "no mask: moved"))
a1, a2 = attend(E, True), attend(E2, True)
b1, b2 = attend(E, False), attend(E2, False)
for i in range(T):
print("%-10d %20.2e %20.2e"
% (i, np.abs(a1[i] - a2[i]).max(), np.abs(b1[i] - b2[i]).max()))
print(" with the causal mask, positions 0 to 3 are BIT-IDENTICAL after a")
print(" later token changed. without it, every position moved.")
print(" that is the property the whole scheme rests on: position i's")
print(" prediction cannot have seen token i+1, so all %d predictions can"
% (T - 1))
print(" be computed in the same pass and scored against the same")
print(" sequence, honestly.")
print()
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
import numpy as np
TOKENS = ["the", "cat", "sat", "on", "the", "mat"]
T = len(TOKENS)
print("WHAT THAT SAVES. suppose you did it the naive way instead:")
print("%-16s %18s %18s %14s"
% ("sequence length", "naive passes", "with the mask", "speedup"))
for n in (6, 128, 1024, 8192):
print("%-16s %18s %18d %13dx"
% ("{:,}".format(n), "{:,}".format(n - 1), 1, n - 1))
print(" the mask turns training from O(sequence length) forward passes")
print(" into ONE. that is not an optimisation -- at %s tokens per"
% "{:,}".format(8192))
print(" sequence it is the difference between possible and not.")
print()
print("AND THE LOSS IS AN AVERAGE OVER ALL OF THEM. one number per")
print("sequence, from %d predictions:" % (T - 1))
probs = [0.62, 0.31, 0.88, 0.45, 0.07]
print("%-30s %14s %14s" % ("predicting", "P(correct)", "-log P"))
for (i, p) in zip(range(1, T), probs):
print("%-30s %14.4f %14.4f" % (repr(TOKENS[i]), p, -np.log(p)))
loss = float(np.mean([-np.log(p) for p in probs]))
print("%-30s %14s %14.4f" % ("mean cross-entropy loss", "", loss))
print("%-30s %14s %14.4f" % ("perplexity = exp(loss)", "", np.exp(loss)))
print(" perplexity %.2f means the model was, on average, as uncertain as"
% np.exp(loss))
print(" if it had been choosing uniformly among %.2f options. that is"
% np.exp(loss))
print(" the whole interpretation, and it is why perplexity is quoted")
print(" rather than the raw loss -- it has units you can picture.")
print(" note also that the WORST prediction here (%r at %.2f) dominates"
% (TOKENS[5], probs[-1]))
print(" the average: -log(%.2f) = %.2f against -log(%.2f) = %.2f."
% (probs[-1], -np.log(probs[-1]), max(probs), -np.log(max(probs))))
print(" cross-entropy punishes confident mistakes far harder than it")
print(" rewards confident successes, which is exactly what you want")
print(" from a training signal.")
Output
Things to try
Use the controls to build intuition for how these models behave.
Train on Other Data: Clear the training data and use one of the samples. Train on the "Nature Description" text. Now, give it the prompt "The ocean is". The predictions will likely be words like "vast", "blue", or "deep". Now, load and train on the "Tech Support Logs" and use the same prompt. The predictions will be fully other. This shows that the model's knowledge is fully dependent on its training data.
Test with Temperature: Set the temperature to its lowest value (0.1). Generate a few tokens. The output will be very predictable. Now, slide the temperature to a high value (like 1.2) and generate again. Notice how the model starts making more creative and sometimes nonsensical choices. This is the trade-off between coherence and creativity.
The "Stuck in a Loop" Problem: With a simple model and low temperature, you might see the model get stuck in a repetitive loop (e.g., "is a is a is a..."). This is a classic problem in language generation that more advanced methods (like top-k or nucleus sampling) help solve.
Worth recalling
Models are Numeric Parrots: They learn patterns, not meaning. Their output is based on the chance of what token should come next based on the data they were trained on.
Generation is One Token at a Time: Text is made sequentially. The model predicts the next token, adds it to the sequence. Then uses that new, longer sequence to predict the token after that.
Temperature Controls Creativity: Temperature is a key parameter for controlling the randomness and "creativity" of the made output.
Context is King: The model's predictions are fully dependent on the preceding tokens (the context). Changing even one word in the prompt can drastically alter the chance distribution for the next token.
From base model to assistant
A causal language model trained just on this objective is a base model. It goes on text. Ask it a question and it may give more questions, since that is what a list of questions looks like in the training data.
Two further stages make it an assistant:
Supervised fine-tuning on cue-response pairs teaches the format — that a question should be answered, that output has a structure, that a conversation has turns.
Choice optimisation (RLHF or DPO) tunes helpfulness, tone and refusal behaviour from human checks.
Neither changes the method. The model is still predicting the next token; the distribution it has been tuned to predict is other.
This distinction matters nearly: base models and cue-tuned models behave very else on the same prompt. And a base model is often the wrong thing to build a product on.
Sampling: turning a distribution into text
The model gives chances; a thing must choose.
Setting
Effect
Temperature 0
Always the most likely token — deterministic
Temperature 0.7
Moderate variety
Temperature >1.2
Usually incoherent
Top-p 0.9
Sample from the smallest set summing to 0.9
Repetition penalty
Down-weight tokens already used
Two facts worth carrying. Greedy decoding is not best — picking the most likely token at each step does not give the most likely sequence. This is why beam search exists and why greedy output tends to repeat. And temperature 0 is the right default for a thing factual: variety in a factual answer is error.
Questions people ask
Why "causal"? Since data flows in one way just, past to future. The same sense as in causal systems. It is often misspelled "casual".
Is this the same as autoregressive? In effect yes. "Causal" describes the masking; "autoregressive" describes feeding outputs back as inputs.
Why not train bidirectionally and generate anyway? A bidirectional model has no notion of "next". And its training task assumes the close context exists. Tries to generate by iterative filling work poorly.
Does the model plan the whole sentence? Architecturally it emits one token at a time, and there is evidence internal states encode data about upcoming tokens. Whether that is planning is an open question.
Why does it repeat it? Low temperature and greedy decoding favour high-chance continuations, and repetition is high-chance. Raise temperature, use top-p, or apply a repetition cost.
Can a causal model do classification? Yes — by prompting, or by reading the final hidden state. At scale it matches encoders on most grasp benchmarks despite the one-one-way constraint.
Recap in one screen
Predict the next token from the past just; a causal mask sets future attention scores to negative infinity.
One forward pass trains each spot at once. And any text is training data — which is why the objective scales.
Generation is the same model applied repeatedly, appending its own output.
Powers with in-context learning emerge from the objective rather than being specified.
A base model goes on text; cue tuning and choice optimisation make it an assistant.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “How GPT-style models learn to write”?
What does this module say about “Quick Context: Learning from Data”?
What does this module say about “Greedy Decoding”?
Cheat sheet
Causal Language Modeling
See how predicting the next token, over and over, is all it takes for a model like GPT to produce fluent text.
GEN AI · vizlearn.in/gen_ai/casual_language_modeling.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.