Follow a sentence through every stage a transformer puts it through — tokens, IDs, embeddings, positional encoding, self-attention and output logits. Real sinusoidal encodings and a real attention matrix, computed live.
Overview
Stages 1–3: Text Becomes Integers
Tokenization splits the string into subword pieces, and each piece is looked up in the vocabulary to get an integer ID. That is the model's entire input: a list of integers. Nothing about the meaning of a word has entered yet — ID 4021 is no closer to ID 4022 than to ID 9. The IDs are arbitrary addresses, not measurements.
Raw text
–
The Journey from Characters to Prediction
A language model never sees text. It sees numbers — and the transformation from your sentence to those numbers, and back to a prediction, is a fixed pipeline of stages. Stepping through them is the fastest way to demystify what "the model read my prompt" actually means.
Stage 4: Embeddings Give IDs Meaning
Each ID indexes into a giant lookup table, pulling out a learned vector of d numbers. These vectors are where meaning lives. Words used in similar contexts end up with similar vectors, and this table is trained along with everything else.
Notice the shape change: a list of n integers becomes an n × d matrix. Every stage from here on preserves that shape — the transformer transforms these vectors repeatedly without ever changing the grid size.
Stage 5: Position Must Be Injected
Self-attention has no inherent sense of order — it sees a set of vectors, so "dog bites man" and "man bites dog" would be identical. Position is therefore added directly into the vectors, classically with sinusoids of geometrically increasing wavelength:
The values in this lab are the genuine formula. Look along a row and you will see fast oscillation in the early dimensions and near-constant values in the later ones — that mix gives every position a unique fingerprint the model can decode.
Stage 6: Attention Mixes the Tokens
Now every token looks at every other token. Each position produces a query and a key; their dot product, scaled by √d, scores how relevant one token is to another, and a softmax over each row turns those scores into weights that sum to 1:
Attention(Q, K, V) = softmax(QKᵀ / √d) V
The heatmap in this stage is a real computation from the vectors above it. Each row shows where one token sent its attention. This is the step where a token stops being an isolated word and becomes a contextual representation.
Stages 7–8: Depth, Then a Single Prediction
One attention block is repeated dozens of times, each with a feed-forward network, residual connections and normalisation. Early layers capture syntax; deeper layers capture semantics and task structure.
At the very end only the last position's vector is used for the next token. It is multiplied by the vocabulary matrix to produce one logit per possible token — and we are back to a distribution over words. The chosen token is appended to the input and the entire pipeline runs again, from stage 1, for every single word generated.
The full path, once
Text goes in, text comes out, and five stages sit between them. Knowing which stage a problem lives in resolves most confusion about model behaviour.
Tokenise — the string becomes integer ids via a subword vocabulary.
Embed — each id indexes a learned matrix, giving a vector per token.
Add position — attention is order-blind, so positional information is injected.
N transformer blocks — attention moves information between positions, feed-forward layers process each position.
Project and sample — one logit per vocabulary entry, softmax, pick a token.
"The capital of France is" → [464, 3139, 286, 4881, 318] → 5×4096 → blocks → " Paris"
Then the chosen token is appended to the input and the whole thing runs again. That loop is generation.
Inside one block
Every block is the same two sublayers, each wrapped in a residual connection and a normalisation:
Attention lets each token gather information from others. Each token produces a query, compares it against every token's key, and takes a weighted mixture of their values. In a decoder, future positions are masked out so generation is possible.
The feed-forward network processes each position independently, expanding to roughly four times the model dimension and back. This is where most of the parameters live — typically two thirds of the model — which is worth knowing because attention gets all the attention.
Residual connections give gradients a clean path back through dozens of blocks. Without them, deep stacks do not train.
The tensor shape through all of this is constant: (batch, sequence, model_dim). That constancy is what makes blocks stackable, and why the same architecture works at 12 layers and at 80.
Two phases, very different costs
Generation splits into two workloads, and the split explains most latency and pricing behaviour.
Prefill processes the prompt. Every token is known, so all positions are computed in parallel in one pass. Compute-bound, efficient, cheap per token.
Decode produces the output. Each token depends on the last, so it is strictly sequential, and every step must read the entire model's weights plus the KV cache from memory. Bandwidth-bound, and expensive per token.
Prefill
Decode
Parallel
Yes
No
Bottleneck
Compute
Memory bandwidth
Cost per token
Low
High
User-visible metric
Time to first token
Tokens per second
That is why input tokens are priced below output tokens, why a long prompt delays the first word rather than slowing the stream, and why generating 500 tokens takes longer than reading 5,000.
The KV cache is what keeps decode linear: previous tokens' keys and values are stored rather than recomputed. It costs memory — several gigabytes for a long context — and it is the dominant memory consumer in serving.
The shapes, end to end
The stages are described above. Following the SHAPES through them is what makes the pipeline concrete -- where the parameters actually are, why prefill and decode cost such different amounts, and what the model does not have that you might assume it does.
example_01.pyNumPy
import numpy as np
VOCAB, D, LAYERS, HEADS, T = 32000, 4096, 32, 32, 6
TOKENS = ["The", "cat", "sat", "on", "the", "mat"]
print("SIX TOKENS THROUGH A MODEL WITH %d LAYERS, %d HEADS AND"
% (LAYERS, HEADS))
print("d_model = %d. the shape at every stage:" % D)
print("%-34s %-18s %s" % ("stage", "shape", "what changed"))
rows = [
("the text", "a string", "-"),
("1-3. tokenise", "(%d,) ints" % T, "text becomes integers"),
("4. embedding lookup", "(%d, %d)" % (T, D), "integers become vectors"),
("5. + positional encoding", "(%d, %d)" % (T, D), "order is injected"),
("6. attention, per layer", "(%d, %d)" % (T, D), "tokens mix"),
(" feed-forward, per layer", "(%d, %d)" % (T, D), "each token alone"),
("7. after %d layers" % LAYERS, "(%d, %d)" % (T, D), "nothing, shape-wise"),
("8. the output head", "(%d, %s)" % (T, "{:,}".format(VOCAB)),
"back to vocabulary"),
(" take the last row", "(%s,)" % "{:,}".format(VOCAB), "one distribution"),
]
for name, shape, note in rows:
print("%-34s %-18s %s" % (name, shape, note))
print(" note what does NOT change: from stage 4 to stage 7 the shape is")
print(" (%d, %d) throughout. every layer is a function from that shape"
% (T, D))
print(" to itself, which is why you can stack as many as you like and")
print(" why they can share a residual stream.")
print()
print("WHERE THE PARAMETERS ARE. this is the table that explains the")
print("model's size:")
emb = VOCAB * D
attn_per_layer = 4 * D * D
ff_per_layer = 3 * D * (4 * D) # SwiGLU-ish: 3 matrices
per_layer = attn_per_layer + ff_per_layer
total = emb + LAYERS * per_layer + emb
print("%-34s %18s %10s" % ("component", "parameters", "share"))
for name, n in (("embedding table", emb),
("attention, all %d layers" % LAYERS, LAYERS * attn_per_layer),
("feed-forward, all %d layers" % LAYERS, LAYERS * ff_per_layer),
("output head", emb)):
print("%-34s %18s %9.1f%%"
% (name, "{:,}".format(n), 100.0 * n / total))
print("%-34s %18s" % ("total", "{:,}".format(total)))
print(" %.1f billion, which is what this configuration comes to -- the"
% (total / 1e9))
print(" round numbers in model names are the configuration, not the")
print(" parameter count.")
print(" the FEED-FORWARD layers hold about %.0f%% of the parameters, not"
% (100.0 * LAYERS * ff_per_layer / total))
print(" attention. attention gets the attention, and the majority of the")
print(" weights are in the boring per-token part.")
print(" note also that the embedding table and the output head are the")
print(" same size, and in many models they are the SAME MATRIX -- tied")
print(" weights, saving %s parameters." % "{:,}".format(emb))
print()
print("PREFILL AND DECODE ARE THE SAME ARITHMETIC AT DIFFERENT SHAPES:")
print("%-12s %-14s %-26s %s"
% ("phase", "input shape", "matmul shape", "bound by"))
print("%-12s %-14s %-26s %s"
% ("prefill", "(%d, %d)" % (T, D), "(%d, %d) x (%d, %d)" % (T, D, D, D),
"compute"))
print("%-12s %-14s %-26s %s"
% ("decode", "(1, %d)" % D, "(1, %d) x (%d, %d)" % (D, D, D),
"memory bandwidth"))
print(" the WEIGHTS are identical in both. what changes is how many rows")
print(" are pushed through them -- and a matrix multiply with one row is")
print(" a very poor use of hardware built for thousands.")
BYTES = 2
print(" weights read per token, decode: %s bytes"
% "{:,}".format(int(total * BYTES)))
print(" weights read per token, prefill at %d tokens: %s bytes"
% (T, "{:,}".format(int(total * BYTES / T))))
print(" the same read, amortised over %d tokens instead of 1. that ratio"
% T)
print(" is why a long prompt is cheap per token and generation is not,")
print(" and why batching helps decode enormously and prefill barely.")
print()
print("AND WHAT THE MODEL DOES NOT HAVE, which the shapes make obvious:")
print("%-38s %s" % ("no memory between calls", "the (%d, %d) is rebuilt each time" % (T, D)))
print("%-38s %s" % ("no access to its own weights", "it cannot inspect itself"))
print("%-38s %s" % ("no notion of 'now'", "unless a token says so"))
print("%-38s %s" % ("no arithmetic unit", "digits are just tokens"))
print(" the last one is worth a demonstration. a number is tokenised")
print(" like any other text, and the split is not what you would")
print(" choose. these are the kinds of split a BPE vocabulary produces:")
for n, pieces in (("7", ["7"]), ("42", ["42"]), ("1234", ["123", "4"]),
("12345", ["123", "45"]), ("3.14159", ["3", ".", "141", "59"])):
print(" %-10s -> %-24s %d token%s"
% (n, " | ".join(pieces), len(pieces), "" if len(pieces) == 1 else "s"))
print(" '1234' arriving as '123' + '4' means there is no representation")
print(" of 'the units digit' anywhere -- there is a token '123' and a")
print(" token '4', and the model's only handle on their relationship is")
print(" what it has learned about which tokens follow which.")
print(" arithmetic then has to be learned as a")
print(" pattern over those pieces rather than computed, which is exactly")
print(" why models are unreliable at long multiplication and reliable at")
print(" calling a calculator.")
Output
Try it yourself
Walk from stage 1 to 8 with Next and keep an eye on the shape tag. Watch n integers become n × d and stay there until the final projection.
At stage 5, compare the two rows of "the". Identical embeddings, different positional vectors — which is the only reason the model can tell the two occurrences apart.
At stage 6, read the attention rows. Repeated words attend to each other strongly, because attention is driven by vector similarity.
Change the sentence to "man bites dog", then "dog bites man". Same tokens, different attention pattern — order genuinely changes the computation once positions are added.
Raise the model dimension. Every downstream matrix widens, which is the crudest possible illustration of why parameter counts explode with d.
What to remember
Text in, integers, vectors, mixed vectors, logits, text out. The embeddings and attention weights in this lab are deterministic stand-ins for learned parameters, but the operations — sinusoidal positions, scaled dot-product attention, softmax — are the real ones. Understanding this pipeline makes context windows, tokenizer quirks and attention costs stop feeling arbitrary.
What the model does and does not have
No memory between calls. A chat interface re-sends the whole conversation every time. The model reads the history; it does not remember it. That is why long conversations get slower and more expensive, and why "remember what I said earlier" only works within the window.
No access to anything outside the prompt. No files, no internet, no database — unless a tool call provides it. Retrieval exists precisely to put external information into the context.
No view of characters. The model sees tokens. "Strawberry" may be two or three tokens, none of which is a letter, which is why letter-counting and spelling questions are unreliable.
No calculator. Numbers tokenise inconsistently, so arithmetic is pattern-matched rather than computed. Give it a tool.
No calibrated confidence. It produces a distribution over tokens, not a belief about truth. A confident tone is a property of the text distribution.
Those five limitations account for a large share of the surprising failures people report, and all five are structural rather than fixable by prompting.
Where the parameters are
For a 7B-parameter model with 32 layers and a model dimension of 4096:
Component
Approximate share
Feed-forward layers
~65%
Attention projections
~30%
Embedding and output
~5%
LayerNorm
Negligible
The embedding and output matrices are frequently tied — the same weights used in both directions — which saves a substantial number of parameters (a 128,000-token vocabulary times 4,096 dimensions is 524 million) and typically helps slightly.
Memory at inference is roughly the parameter count times the bytes per parameter, plus the KV cache. That is the arithmetic behind every quantisation decision: 7 billion parameters is 14GB at 16-bit and 3.5GB at 4-bit.
Practical implications
Count tokens with the real tokeniser when budgeting a prompt. Word-count estimates are unreliable, especially for code and non-Latin scripts.
Put important material at the ends of a long prompt. Models attend less reliably to the middle of a long context.
Shorter prompts are cheaper and often better. Irrelevant context degrades answers as well as costing money.
Temperature 0 for anything that must be determinate — extraction, classification, code. Higher only where variety is wanted.
Stream the output if a user is waiting. Time to first token is what feels like latency, and it is dominated by prefill.
Questions people ask
Does the model understand what it reads? It models statistical structure in text extremely well. Whether that constitutes understanding is a genuine open question rather than a settled one.
Why is the same question answered differently each time? Sampling, unless temperature is 0 — and even then, batching and hardware non-determinism produce small variations.
Can it read a file I mention? Only if something puts the contents into the context. Naming a path does nothing.
Why does it fail at simple arithmetic? Tokenisation fragments numbers, and the model pattern-matches rather than calculating. Use a tool.
What is the context window shared between? Prompt, system instructions, retrieved documents, conversation history and generated output — one budget for all of them.
Why are output tokens more expensive? Prefill is parallel; decode is sequential and bandwidth-bound.
Recap in one screen
Tokenise, embed, add position, pass through N identical blocks, project to the vocabulary, sample — then repeat.
Attention moves information between positions; the feed-forward layers hold most of the parameters.
Prefill is parallel and cheap; decode is sequential and expensive, which explains the pricing and latency asymmetry.
The KV cache makes decode linear at a large memory cost.
No memory between calls, no view of characters, no calculator, no calibrated confidence.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Text in, integers, vectors, mixed vectors, logits, text out. The embeddings and attention weights in this lab are deterministic stand-ins for learned parameters, but the operations — sinusoidal positions, scaled dot-product attention, softmax — are the real ones. Understanding this pipeline makes context windows, tokenizer quirks and attention costs stop feeling arbitrary.
What does this module say about “Stages 1–3: Text Becomes Integers”?
Tokenization splits the string into subword pieces, and each piece is looked up in the vocabulary to get an integer ID. That is the model's entire input: a list of integers. Nothing about the meaning of a word has entered yet — ID 4021 is no closer to ID 4022 than to ID 9. The IDs are arbitrary addresses, not measurements.
What does this module say about “Stage 4: Embeddings Give IDs Meaning”?
Each ID indexes into a giant lookup table, pulling out a learned vector of d numbers. These vectors are where meaning lives. Words used in similar contexts end up with similar vectors, and this table is trained along with everything else.
Cheat sheet
How LLMs Process Text?
Follow a sentence through every stage a transformer puts it through — tokens, IDs, embeddings, positional encoding, self-attention and output logits. Real sinusoidal encodings and a real attention matrix, computed live.
GEN AI · vizlearn.in/gen_ai/how_llms_process_text.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.