Follow a sentence through every stage a transformer puts it through — tokens, IDs, embeddings, positional encoding, self-attention and output logits. Real sinusoidal encodings and a real attention matrix, computed live.
Tokenization splits the string into subword pieces, and each piece is looked up in the vocabulary to get an integer ID. That is the model's entire input: a list of integers. Nothing about the meaning of a word has entered yet — ID 4021 is no closer to ID 4022 than to ID 9. The IDs are arbitrary fixes, not measurements.
The embedding layer converts each ID into a dense vector of learned real numbers. Words used in close contexts end up with close vectors. So meaning enters here. Spot is added next — either as sinusoidal encodings or as learned spot embeddings —. This is because self-attention treats its input as a set and would if not be blind to word order.
The transformer layers are the core of the model. Each layer applies self-attention (each token looks at each other token to decide what data to include) followed by a feed-forward network (two linear layers with a nonlinearity). A typical LLM stacks 32 to 96 of these layers. At each layer the forms are refined: early layers learn syntax and local patterns, middle layers learn semantic links, and later layers learn task-set features.
The output layer projects the final hidden state of each spot back to the vocabulary size. This gives a score (logit) for each token in the vocabulary. Softmax converts those logits into a chance distribution. And the model samples or takes the argmax to give the next token. This process repeats token by token — each made token is appended to the input and the model runs again —. This is why generation is called autoregressive and why it is slower than encoding.
The entire pipeline — tokenise, embed, add spot, transform through N layers, project, sample — is what happens each time the model makes a single token. Grasp it explains why context length matters (attention is quadratic in sequence length), why the first token is slow (the full input must be encoded) and why later tokens are faster (the KV cache stores the attention keys and values from last spots).
Follow a sentence through every stage a transformer puts it through — tokens, IDs, embeddings, positional encoding, self-attention and output logits.
A language model never sees text. It sees numbers — and the transformation from your sentence to those numbers, and back to a prediction, is a fixed pipeline of stages. Stepping through them is the fastest way to explain what "the model read my prompt" actually means.
Tokenization splits the string into subword pieces, and each piece is looked up in the vocabulary to get an integer ID. That is the model's entire input: a list of integers. Nothing about the meaning of a word has entered yet — ID 4021 is no closer to ID 4022 than to ID 9. The IDs are arbitrary fixes, not measurements.
The embedding layer converts each ID into a dense vector of learned real numbers. Words used in close contexts end up with close vectors. So meaning enters here. Spot is added next — either as sinusoidal encodings or as learned spot embeddings —. This is because self-attention treats its input as a set and would if not be blind to word order.
The transformer layers are the core of the model. Each layer applies self-attention (each token looks at each other token to decide what data to include) followed by a feed-forward network (two linear layers with a nonlinearity). A typical LLM stacks 32 to 96 of these layers. At each layer the forms are refined: early layers learn syntax and local patterns, middle layers learn semantic links, and later layers learn task-set features.
The output layer projects the final hidden state of each spot back to the vocabulary size. This gives a score (logit) for each token in the vocabulary. Softmax converts those logits into a chance distribution. And the model samples or takes the argmax to give the next token. This process repeats token by token — each made token is appended to the input and the model runs again —. This is why generation is called autoregressive and why it is slower than encoding.
The entire pipeline — tokenise, embed, add spot, transform through N layers, project, sample — is what happens each time the model makes a single token. Grasp it explains why context length matters (attention is quadratic in sequence length), why the first token is slow (the full input must be encoded) and why later tokens are faster (the KV cache stores the attention keys and values from last spots).
Each ID indexes into a giant lookup table. This pulls out a learned vector of d numbers. These vectors are where meaning lives. Words used in close contexts end up with close vectors. And this table is trained along with all of it else.
Notice the shape change: a list of n integers becomes an n × d matrix. Each stage from here on preserves that shape. The transformer transforms these vectors repeatedly without altering the grid size.
Self-attention has no inherent sense of order — it sees a set of vectors. So "dog bites man" and "man bites dog" would be the same. Spot is so added right into the vectors, classically with sinusoids of geometrically raising wavelength:
PE(pos, 2i) = sin(pos / 100002i/d) PE(pos, 2i+1) = cos(pos / 100002i/d)
The values in this lab are the genuine formula. Look along a row and you will see fast oscillation in the early dimensions and near-constant values in the later ones — that mix gives each spot a unique fingerprint the model can decode.
Now each token looks at each other token. Each spot gives a query and a key; their dot product, scaled by √d, scores how apt one token is to another. And a softmax over each row turns those scores into weights that sum to 1:
Attention(Q, K, V) = softmax(QKᵀ / √d) V
The heatmap in this stage is a real work from the vectors above it. Each row shows where one token sent its attention. This is the step where a token stops being an isolated word and becomes a context form.
One attention block is repeated dozens of times, each with a feed-forward network, residual connections and normalisation. Early layers capture syntax; deeper layers capture semantics and task structure.
At the very end only the last spot's vector is used for the next token. It is times by the vocabulary matrix to give one logit per doable token. And we are back to a distribution over words. The chosen token is appended to the input and the entire pipeline runs again, from stage 1, for each single word made.
Text goes in, text comes out, and five stages sit between them. Knowing which stage a problem lives in resolves most confusion about model behaviour.
"The capital of France is" → [464, 3139, 286, 4881, 318] → 5×4096 → blocks → " Paris"
Then the chosen token is appended to the input and the whole thing runs again. That loop is generation.
Each block is the same two sublayers, each wrapped in a residual connection and a normalisation:
x → LayerNorm → Attention → (+x) → LayerNorm → FFN → (+x)
Attention lets each token gather data from others. Each token gives a query, compares it against each token's key, and takes a weighted mixture of their values. In a decoder, future spots are masked out so generation is doable.
The feed-forward network processes each spot on its own, expanding to roughly four times the model dimension and back. This is where most of the parameters live. Often two thirds of the model. Which is worth knowing. This is because attention gets all the attention.
Residual connections give gradients a clean path back through dozens of blocks. Without them, deep stacks do not train.
The tensor shape through all of this is constant: (batch, sequence, model_dim). That constancy is what makes blocks stackable, and why the same design works at 12 layers and at 80.
Generation splits into two workloads, and the split explains most latency and pricing behaviour.
Prefill processes the prompt. Each token is known, so all spots are computed in parallel in one pass. Compute-bound, fast, cheap per token.
Decode gives the output. Each token depends on the last, so it is strictly sequential. And each step must read the entire model's weights plus the KV cache from memory. Bandwidth-bound, and costly per token.
| Prefill | Decode | |
|---|---|---|
| Parallel | Yes | No |
| Bottleneck | Compute | Memory bandwidth |
| Cost per token | Low | High |
| User-visible metric | Time to first token | Tokens per second |
That is why input tokens are priced below output tokens, why a long prompt delays the first word. Rather than slowing the stream, and why making 500 tokens takes longer than reading 5,000.
The KV cache is what keeps decode linear: last tokens' keys and values are stored rather than recomputed. It costs memory — a few gigabytes for a long context — and it is the dominant memory consumer in serving.
The stages are described above. Below the SHAPES through them is what makes the pipeline concrete -- where the parameters really are, why prefill and decode cost such different amounts, and what the model does not have that you might assume it does.
That constant shape hides where the parameters really live. The next block totals them up by part.
Those same weights are read very differently depending on how many rows pass through them at once. The next block compares prefill against decode.
The shapes also make it obvious what the model does not have. Including any real handle on digits. The next block shows how a BPE vocabulary splits numbers.
Text in, integers, vectors, mixed vectors, logits, text out. The embeddings and attention weights in this lab are deterministic stand-ins for learned parameters. But the steps — sinusoidal spots, scaled dot-product attention, softmax — are the real ones. Grasp this pipeline makes context windows, tokenizer quirks and attention costs stop feeling arbitrary.
No memory between calls. A chat interface re-sends the whole conversation each time. The model reads the history; it does not recall it. That is why long conversations get slower and more costly, and why "recall what I said earlier" only works within the window.
No access to anything outside the prompt. No files, no internet, no database — unless a tool call gives it. Retrieval exists just to put external data into the context.
No view of characters. The model sees tokens. "Strawberry" may be two or three tokens, none of which is a letter. This is why letter-counting and spelling questions are unreliable.
No calculator. Numbers tokenise inconsistently, so arithmetic is pattern-matched rather than computed. Give it a tool.
No calibrated confidence. It gives a distribution over tokens, not a belief about truth. A confident tone is a property of the text distribution.
Those five limitations account for a large share of the surprising failures people report. And all five are structural rather than fixable by prompting.
For a 7B-parameter model with 32 layers and a model dimension of 4096:
| Component | Approximate share |
|---|---|
| Feed-forward layers | ~65% |
| Attention projections | ~30% |
| Embedding and output | ~5% |
| LayerNorm | Negligible |
The embedding and output matrices are often tied. The same weights used in both directions. Which saves a large number of parameters (a 128,000-token vocabulary times 4,096 dimensions is 524 million) and often helps slightly.
Memory at inference is roughly the parameter count times the bytes per parameter, plus the KV cache. That is the arithmetic behind each quantisation decision: 7 billion parameters is 14GB at 16-bit and 3.5GB at 4-bit.
Count tokens with the real tokeniser when budgeting a prompt. Word-count estimates are unreliable, especially for code and non-Latin scripts.
Put key material at the ends of a long prompt. Models attend less reliably to the middle of a long context.
Shorter prompts are cheaper and often better. Irrelevant context degrades answers and costing money.
Temperature 0 for anything that must be determinate — extraction, classification, code. Higher only where variety is wanted.
Stream the output if a user is waiting. Time to first token is what feels like latency, and it is ruled by prefill.
Does the model understand what it reads? It models numeric structure in text very well. Whether that constitutes grasp is a genuine open question rather than a settled one.
Why is the same question answered differently each time? Sampling, unless temperature is 0. And even then, batching and hardware non-determinism give small changes.
Can it read a file I mention? Only if something puts the contents into the context. Naming a path does nothing.
Why does it fail at simple arithmetic? Tokenisation fragments numbers, and the model pattern-matches rather than finding. Use a tool.
What is the context window shared between? Prompt, system cues, fetched documents, conversation history and made output. One budget for all of them.
Why are output tokens more costly? Prefill is parallel; decode is sequential and bandwidth-bound.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Stages 1–3: Text Becomes Integers”?
What does this module say about “Stage 4: Embeddings Give IDs Meaning”?
What does this module say about “Stage 5: Spot Must Be Injected”?
Follow a prompt through tokenisation, embedding, positional encoding, transformer layers, softmax and sampling — one token out.