Positional Encoding

Quick Context

Self-attention computes every position from every other position by dot products and weighted sums. Nowhere in that computation does an index appear. Permute the input and the outputs permute with it, unchanged — the mechanism is permutation-equivariant.

For a bag of words that would be a feature. For language it is fatal: "dog bites man" and "man bites dog" contain exactly the same words, and a model that cannot tell them apart cannot do anything useful. An RNN got order for free by reading left to right. Throwing out recurrence threw that away too, and it has to be put back by hand.

The obvious ideas, and why they fail

• Append the integer position. Position 512 then contributes a value five hundred times larger than position 1, swamping the embedding, and the model has never seen position 3000 during training.
• Append position / length. Now it is bounded, but the same value means different things in different sentences — 0.5 is word 5 of 10 and word 50 of 100. Nothing consistent can be learned from it.

Both are available in the Scheme control above. What is wanted is something bounded, unique per position, consistent across sentence lengths, and — ideally — carrying information about relative distance, since "the adjective three words back" is a far more useful notion than "the word at index 17".

The sinusoidal answer

The original transformer used a fixed function of position, with a different frequency in every pair of dimensions:

PE(pos, 2i) = sin( pos / 10000^2i/d )

PE(pos, 2i+1) = cos( pos / 10000^2i/d )

The wavelengths form a geometric series from about 2 up to about 10000·2π. Early dimensions flip every couple of positions; late dimensions barely move across the whole sequence. Together they behave like the digits of a binary counter — the fast bits distinguish neighbours, the slow bits distinguish regions — which is why the matrix above has that striped, fanning appearance.

Every value stays inside −1 to 1 no matter how long the sequence, and the pattern for position 500 is defined whether or not the model ever trained on a sequence that long.

Interactive Exploration Guide

1. Break word order. Set the Scheme to None and tick Swap Two Words. Input Difference reads 0.000 and the two sentences are literally identical to the model. Switch the Scheme back to Sinusoidal and the difference becomes non-zero — the encoding is what makes them different inputs.Run this
2. See the frequency fan. Look at the matrix with the Model Dimension slider at 64. The leftmost columns stripe rapidly down the page while the rightmost are almost flat. That spread of wavelengths is the whole design.Run this
3. Check it encodes distance. Set the Inspect Position slider to 12 and look at the lower panel. Similarity peaks at position 12 and falls away smoothly on both sides, so nearby positions have similar encodings. Move the position and the peak travels with it, keeping its shape — the pattern depends on the gap, not on where you are.Run this
4. Try the naive schemes. Set the Scheme to Raw integer position, then drag the Inspect Position slider up and watch Largest Value climb with it, without bound. Compare against Sinusoidal, where it never leaves 1. Then try Position / length and lengthen the sequence: every encoding shifts, because the same position now means a different fraction.Run this
5. Push past training length. Set the Sequence Length slider to its maximum with Sinusoidal selected. The values stay inside −1 to 1 and every row is still distinct. Nothing had to be learned for those positions to exist.Run this

Added, not concatenated

The encoding is added to the word embedding, not stapled onto the end of it. That surprises people — surely adding corrupts the meaning?

In practice it does not, and the usual explanation is that in a space of several hundred dimensions there is room for the model to keep the two kinds of information in largely separate subspaces, and the projections that follow can pull them apart again. Concatenating would be cleaner conceptually but would spend dimensions that are more useful elsewhere.

What is used now

• Learned absolute — a plain embedding table indexed by position, used by BERT and GPT-2. Simple and works well, but has a hard maximum length and learns nothing about positions beyond it.
• Relative — bias the attention scores by the distance between the two positions rather than encoding absolutes at all. Closer to what attention actually wants to know.
• RoPE (rotary) — rotate the query and key vectors by an angle proportional to position, so the dot product between them depends on their difference by construction. Now the default in most open-weight models.

Sinusoidal encoding is no longer the state of the art, but it is the clearest illustration of what the problem is, and RoPE is easier to understand once you have seen it.

What usually goes wrong

• Leaving it out. The model still trains and the loss still falls, so the bug is silent — it just performs like a bag-of-words model and nobody knows why.
• Exceeding the trained length. With learned encodings, positions past the table simply do not exist. Most context-extension work is about making positional information survive past where it was trained.
• Adding it at every layer. The original adds it once, at the input. Re-adding it at each layer is a common misreading of the diagram.
• Assuming the model reads it as a number. It is a pattern, not a counter. What the model learns is which patterns co-occur, not that dimension 7 means "position 12".

Key Takeaway

Self-attention has no notion of order, so position has to be supplied explicitly, and the naive options fail: raw indices grow without bound and normalised ones mean different things in different-length sentences. Sinusoidal encoding solves it with a bank of sine and cosine waves at geometrically spaced frequencies, giving every position a bounded, unique fingerprint that is defined for any length and whose similarity to other positions depends on the distance between them. It is added to the embedding rather than concatenated, and modern models mostly use learned or rotary variants — but all of them exist to answer the same question this one does.