Follow a sentence through every stage a transformer puts it through — tokens, IDs, embeddings, positional encoding, self-attention and output logits. Real sinusoidal encodings and a real attention matrix, computed live.
A language model never sees text. It sees numbers — and the transformation from your sentence to those numbers, and back to a prediction, is a fixed pipeline of stages. Stepping through them is the fastest way to demystify what "the model read my prompt" actually means.
Tokenization splits the string into subword pieces, and each piece is looked up in the vocabulary to get an integer ID. That is the model's entire input: a list of integers. Nothing about the meaning of a word has entered yet — ID 4021 is no closer to ID 4022 than to ID 9. The IDs are arbitrary addresses, not measurements.
Each ID indexes into a giant lookup table, pulling out a learned vector of d numbers. These vectors are where meaning lives. Words used in similar contexts end up with similar vectors, and this table is trained along with everything else.
Notice the shape change: a list of n integers becomes an n × d matrix. Every stage from here on preserves that shape — the transformer transforms these vectors repeatedly without ever changing the grid size.
Self-attention has no inherent sense of order — it sees a set of vectors, so "dog bites man" and "man bites dog" would be identical. Position is therefore added directly into the vectors, classically with sinusoids of geometrically increasing wavelength:
PE(pos, 2i) = sin(pos / 100002i/d) PE(pos, 2i+1) = cos(pos / 100002i/d)
The values in this lab are the genuine formula. Look along a row and you will see fast oscillation in the early dimensions and near-constant values in the later ones — that mix gives every position a unique fingerprint the model can decode.
Now every token looks at every other token. Each position produces a query and a key; their dot product, scaled by √d, scores how relevant one token is to another, and a softmax over each row turns those scores into weights that sum to 1:
Attention(Q, K, V) = softmax(QKᵀ / √d) V
The heatmap in this stage is a real computation from the vectors above it. Each row shows where one token sent its attention. This is the step where a token stops being an isolated word and becomes a contextual representation.
One attention block is repeated dozens of times, each with a feed-forward network, residual connections and normalisation. Early layers capture syntax; deeper layers capture semantics and task structure.
At the very end only the last position's vector is used for the next token. It is multiplied by the vocabulary matrix to produce one logit per possible token — and we are back to a distribution over words. The chosen token is appended to the input and the entire pipeline runs again, from stage 1, for every single word generated.
Text in, integers, vectors, mixed vectors, logits, text out. The embeddings and attention weights in this lab are deterministic stand-ins for learned parameters, but the operations — sinusoidal positions, scaled dot-product attention, softmax — are the real ones. Understanding this pipeline makes context windows, tokenizer quirks and attention costs stop feeling arbitrary.
Follow a sentence through every stage a transformer puts it through — tokens, IDs, embeddings, positional encoding, self-attention and output logits. Real sinusoidal encodings and a real attention matrix, computed live.