Positional Encoding

Quick Context

Self-attention computes each spot from each other spot by dot products and weighted sums. Nowhere in that work does an index appear. Permute the input and the outputs permute with it, unchanged. The method is permutation-equivariant.

For a bag of words that would be a feature. For language it is fatal: "dog bites man" and "man bites dog" hold just the same words. And a model that cannot tell them apart cannot do anything useful. An RNN got order for free by reading left to right. Throwing out recurrence threw that away too, and it has to be put back by hand.

The first Transformer paper solved this by adding a fixed vector to each token embedding before attention sees it. The vector is different for each spot, so tokens at different spots become distinguishable even though the method itself is spot-blind. The encoding uses sine and cosine functions at different counts. Even dimensions get sin(pos / 10000^(2i/d)) and odd dimensions get the cosine of the same argument. Where pos is the spot and i is the dimension index.

This scheme has two useful properties. First, the encoding for each spot is unique, so no two spots receive the same sign. Second, relative spots can be recovered from the encodings by a linear transformation,. This is because the link between sin(a) and sin(a + k) is a fixed rotation that depends only on k, not on a. That means the model can learn to attend to "the token two spots back" no matter absolute spot.

Learned positional embeddings are the main other. Instead of computing the encoding from a formula, each spot gets a trainable vector that is learned during training, the same way word embeddings are learned. BERT and GPT use learned embeddings, and in practice they perform comparably to sinusoidal encodings for sequences within the training length. The downside is that they do not generalise beyond the longest spot seen during training.

The obvious ideas, and why they fail

• Append the integer spot. Spot 512 then adds a value five hundred times larger than spot 1, swamping the embedding. And the model has never seen spot 3000 during training.
• Append spot / length. Now it is bounded. But the same value means different things in different sentences — 0.5 is word 5 of 10 and word 50 of 100. Nothing steady can be learned from it.

Both are available in the Scheme control above. What is wanted is something bounded, unique per spot, steady across sentence lengths, and — ideally — carrying data about relative distance. Since "the adjective three words back" is a far more useful notion than "the word at index 17".

The sinusoidal answer

The first transformer used a fixed function of spot, with a different count in each pair of dimensions:

PE(pos, 2i) = sin( pos / 10000^2i/d )

PE(pos, 2i+1) = cos( pos / 10000^2i/d )

The wavelengths form a geometric series from about 2 up to about 10000·2π. Early dimensions flip each couple of spots; late dimensions barely move across the whole sequence. Together they behave like the digits of a binary counter. The fast bits distinguish neighbours, the slow bits distinguish regions. Which is why the matrix above has that striped, fanning appearance.

Each value stays inside −1 to 1 no matter how long the sequence. And the pattern for spot 500 is set whether or not the model ever trained on a sequence that long.

Attention cannot see order

Self-attention computes a weighted mixture over all spots. Shuffle the input tokens and the same set of outputs comes back, permuted. The method treats the sequence as a bag.

For language that is fatal. "The dog bit the man" and "The man bit the dog" hold the same tokens. And a model without positional data cannot tell them apart.

So order has to be injected apart. That is the entire job of positional encoding, and there are three families of answer.

Sinusoidal encodings

The first scheme adds a fixed pattern to each token's embedding, using sines and cosines at geometrically spaced counts:

PE(pos, 2i) = sin(pos / 10000^2i/d)   PE(pos, 2i+1) = cos(pos / 10000^2i/d)

Each dimension oscillates at a different rate. Low dimensions change quickly with spot. Useful for distinguishing neighbours. And high dimensions change slowly, encoding coarse location. Together they give each spot a unique signature, rather like a binary counter in smooth form.

Two properties made this attractive. It needs no parameters, and it extends to spots never seen during training, because the formula is set for any pos. There is also a neat math fact: the encoding for spot pos + k is a linear function of the encoding for pos. So relative offsets are representable.

Its weakness is that it encodes absolute spot, and what often matters in language is relative distance — "the adjective two words before this noun". Rather than "the word at index 47".

Rotary embeddings, and why they won

RoPE takes a different route. Instead of adding something to the embedding, it rotates the query and key vectors by an angle tied to to their spot, in each two-dimensional slice of the vector.

The result falls out of trigonometry: when a rotated query is dotted with a rotated key, the result depends on the gap between their spots, not on their absolute values. Relative spot is built into the attention score itself, without any extra terms.

That gives three practical gains, which is why in effect each current large model uses it:

• Relative by construction, matching what language really needs.
• Extends further beyond the training length, and can be stretched on purpose with interpolation tricks (NTK scaling, YaRN) to extend context after training.
• No parameters, and it applies inside attention rather than at the input. So it affects each layer always.

ALiBi is the other approach worth knowing: it adds a cost to attention scores tied to to distance. So tokens attend less to things far away. Simple, parameter-free, and it extrapolates well — at the cost of building in a bias towards locality.

Giving attention a sense of order

Attention is permutation-invariant -- this proves it, then builds the sinusoidal encoding that fixes it and checks the property that makes it work.

import numpy as np

rng = np.random.default_rng(0)

def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def attend(E, Wq, Wk, Wv):
    Q, K, V = E @ Wq, E @ Wk, E @ Wv
    A = softmax(Q @ K.T / np.sqrt(Q.shape[1]))
    return A, A @ V

T, D = 5, 8
E = rng.normal(0, 1.0, (T, D))
Wq, Wk, Wv = (rng.normal(0, 0.5, (D, D)) for _ in range(3))

print("FIRST, THE PROBLEM. attention has no idea what order the tokens are in.")
A1, out1 = attend(E, Wq, Wk, Wv)
perm = np.array([3, 0, 4, 1, 2])
A2, out2 = attend(E[perm], Wq, Wk, Wv)
print("   token 3's output in the original order  :", np.round(out1[3][:4], 5))
print("   the same token after shuffling the input:", np.round(out2[0][:4], 5))
print("   identical:", np.allclose(out1[perm], out2))
print()
print("   shuffle the sentence and every token gets exactly the same answer,")
print("   just moved. 'dog bites man' and 'man bites dog' are the same input")
print("   to a transformer. that is not a bug to fix in the attention -- it")
print("   is a property of a weighted sum, and the fix has to come from the")
print("   embeddings.")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

D_MODEL, MAXLEN = 64, 128
pos = np.arange(MAXLEN)[:, None]
i = np.arange(D_MODEL // 2)[None, :]
denom = 10000 ** (2 * i / D_MODEL)
PE = np.zeros((MAXLEN, D_MODEL))
PE[:, 0::2] = np.sin(pos / denom)
PE[:, 1::2] = np.cos(pos / denom)

print("THE SINUSOIDAL ENCODING. each dimension is a wave, and the")
print("wavelengths grow geometrically across the dimensions:")
print("%10s %16s %20s" % ("dim pair", "wavelength", "what it distinguishes"))
for k in (0, 8, 20, 31):
    wl = 2 * np.pi * 10000 ** (2 * k / D_MODEL)
    print("%10d %16.1f %20s"
          % (k, wl, "adjacent tokens" if wl < 20 else
             ("nearby phrases" if wl < 500 else "whole document")))
print("   fast dimensions separate neighbours; slow ones separate distant")
print("   regions. together they give every position a unique fingerprint.")
print()

print("the first few positions, first 6 dimensions:")
print("%8s %s" % ("pos", "".join("%9d" % d for d in range(6))))
for p in (0, 1, 2, 3, 10, 50):
    print("%8d %s" % (p, "".join("%9.4f" % v for v in PE[p][:6])))
print()

norms = np.linalg.norm(PE, axis=1)
print("every position has the same magnitude: %.4f to %.4f"
      % (norms.min(), norms.max()))
print("   so adding it to a token embedding shifts the token without")
print("   changing how loud it is.")
print()

The below block covers the left steps.

import numpy as np

print("similarity between position 0 and the rest -- it falls away with")
print("distance, though not perfectly smoothly:")
unit = PE / np.linalg.norm(PE, axis=1, keepdims=True)
for p in (0, 1, 2, 5, 10, 25, 50, 100):
    print("   pos 0 vs pos %3d: %+.4f  %s"
          % (p, unit[0] @ unit[p], "#" * max(0, int(30 * unit[0] @ unit[p]))))
print("   the trend is downward, with ripples -- those come from the faster")
print("   dimensions cycling back into phase. a model can still read")
print("   'nearby' off this, which is what matters.")
print()

print("THE PROPERTY IT WAS DESIGNED FOR: a fixed offset is a fixed linear")
print("map. PE(pos + k) can be computed from PE(pos) by one rotation that")
print("depends only on k, not on pos:")
for k in (1, 5, 20):
    sims = [unit[p] @ unit[p + k] for p in range(0, 80)]
    print("   offset k=%2d: similarity between pos and pos+k is %.4f +/- %.6f"
          % (k, np.mean(sims), np.std(sims)))
print("   the standard deviation is in effect zero -- the relationship")
print("   between two positions depends only on the GAP between them. that")
print("   is what lets a model generalise to sentence lengths it never saw.")
print()

The next block goes on with the left demonstrations.

print("and it does solve the original problem:")
Emb = rng.normal(0, 1.0, (T, D_MODEL))
Wq2, Wk2, Wv2 = (rng.normal(0, 0.5, (D_MODEL, D_MODEL)) for _ in range(3))
with_pos = Emb + PE[:T]
A3, out3 = attend(with_pos, Wq2, Wk2, Wv2)
A4, out4 = attend((Emb + PE[:T])[perm], Wq2, Wk2, Wv2)
print("   with positional encoding, shuffling now changes the answer:")
print("      max difference: %.4f" % np.abs(out3[perm] - out4).max())
print("   (the shuffled version keeps each token's ORIGINAL position vector,")
print("    which is why it differs -- position is now part of the input.)")
print()

print("the alternatives, briefly:")
print("   learned absolute -- a lookup table of position vectors. simple,")
print("      and cannot extrapolate past the longest length you trained on.")
print("   RoPE -- rotate Q and K by an angle proportional to position, so")
print("      the dot product depends on the gap directly. what most current")
print("      open models use.")
print("   ALiBi -- add a distance penalty straight to the attention scores.")
print("      no vectors at all, and extrapolates well.")
print()
print("all three are answers to the same question: a weighted sum does not")
print("know what order its inputs were in, so the order has to be encoded")
print("into the values being summed.")

Tests to try

1. Break word order. Set the Scheme to None and tick Swap Two Words. Input Gap reads 0.000 and the two sentences are literally the same to the model. Switch the Scheme back to Sinusoidal and the gap becomes non-zero. The encoding is what makes them different inputs.
2. See the count fan. Look at the matrix with the Model Dimension slider at 64. The leftmost columns stripe rapidly down the page while the rightmost are almost flat. That spread of wavelengths is the whole design.
3. Check it encodes distance. Set the Inspect Spot slider to 12 and look at the lower panel. Likeness peaks at spot 12 and falls away smoothly on both sides. So nearby spots have close encodings. Move the spot and the peak travels with it, keeping its shape. The pattern depends on the gap, not on where you are.
4. Try the naive schemes. Set the Scheme to Raw integer spot. Then drag the Inspect Spot slider up and watch Largest Value climb with it, without bound. Compare against Sinusoidal, where it never leaves 1. Then try Spot / length and lengthen the sequence. Each encoding shifts, because the same spot now means a different fraction.
5. Push past training length. Set the Sequence Length slider to its maximum with Sinusoidal selected. The values stay inside −1 to 1 and each row is still distinct. Nothing had to be learned for those spots to exist.

Added, not concatenated

The encoding is added to the word embedding, not stapled onto the end of it. That surprises people — surely adding corrupts the meaning?

In practice it does not, and the usual reason is that in a space of a few hundred dimensions there is room for the model to keep the two kinds of data in largely separate subspaces. And the projections that follow can pull them apart again. Concatenating would be cleaner conceptually but would spend dimensions that are more useful elsewhere.

What is used now

• Learned absolute — a plain embedding table indexed by spot, used by BERT and GPT-2. Simple and works well, but has a hard maximum length and learns nothing about spots beyond it.
• Relative — bias the attention scores by the distance between the two spots rather than encoding absolutes at all. Closer to what attention really wants to know.
• RoPE (rotary) — rotate the query and key vectors by an angle tied to to spot. So the dot product between them depends on their gap by construction. Now the default in most open-weight models.

Sinusoidal encoding is no longer the state of the art. But it is the clearest illustration of what the problem is, and RoPE is easier to understand once you have seen it.

Where this goes wrong

• Leaving it out. The model still trains and the loss still falls. So the bug is silent. It just performs like a bag-of-words model and nobody knows why.
• Exceeding the trained length. With learned encodings, spots past the table simply do not exist. Most context-extension work is about making positional data survive past where it was trained.
• Adding it at each layer. The first adds it once, at the input. Re-adding it at each layer is a common misreading of the diagram.
• Assuming the model reads it as a number. It is a pattern, not a counter. What the model learns is which patterns co-occur, not that dimension 7 means "spot 12".

Where that leaves you

Self-attention has no notion of order, so spot has to be supplied by name. And the naive options fail: raw indices grow without bound and normalised ones mean different things in different-length sentences. Sinusoidal encoding solves it with a bank of sine and cosine waves at geometrically spaced counts. This gives each spot a bounded, unique fingerprint that is set for any length and whose likeness to other spots depends on the distance between them. It is added to the embedding rather than concatenated, and modern models mostly use learned or rotary variants. But all of them exist to answer the same question this one does.

Learned absolute encodings, and their limit

The simplest option is a trainable vector per spot — in effect an embedding table indexed by spot instead of by token. BERT and GPT-2 both did this.

It works well within the trained range and has one hard limitation: spot 512 has no vector if the model was trained to 512. There is nothing to look up, so the model cannot process longer input at all without adding and training new rows.

That single constraint is a large part of why the field moved to rotary and ALiBi schemes. Extending a context window from 4,000 to 128,000 tokens is a plausible fine-tuning project with RoPE and in effect not doable with learned absolute spots.

Extending context after training

Because RoPE is a formula rather than a table, the angles can be reinterpreted. And that is how long-context versions of existing models are gave.

Spot interpolation scales spots down so that a longer sequence maps into the range the model was trained on. Cheap, and it compresses the model's sense of fine-grained distance.

NTK-aware scaling adjusts the count base rather than the spots, distorting high counts less and preserving local resolution better.

YaRN combines interpolation with attention-temperature tweak and is the current standard for large extensions.

All three often involve a short fine-tune on long sequences to let the model adapt. The reason any of it works is that RoPE's rotations are smooth in spot. So middle and stretched values remain meaningful.

Common mistakes

• Assuming attention handles order. It does not, at all. This is the most common misconception about transformers.
• Feeding input longer than the positional scheme backs, which either errors or silently degrades.
• Mixing schemes — loading weights trained with learned spots into an design using RoPE gives fluent nonsense.
• Forgetting spots shift with padding. Left-padding a batch moves each real token's index; most implementations handle it, and custom code often does not.
• Expecting extrapolation for free. Even RoPE degrades well beyond its training length without an interpolation fix.

Questions people ask

Why add positional data rather than concatenate it? Addition keeps the dimension unchanged and works empirically. Joining would spend model width on spot, and was tried and abandoned.

Does adding to the embedding not corrupt the meaning? Somewhat. And the model learns to separate the two. There is enough room in a few hundred dimensions for both signs to coexist.

Do vision transformers need positional encodings? Yes — image patches are also an unordered set without them. Learned 2-D encodings are common there.

Which should I use? RoPE, if you are choosing. In practice you use any the pretrained model uses.

Why 10,000 in the sinusoidal formula? It sets the range of counts, chosen so the wavelengths span from a couple of spots to tens of thousands. It is a tuned constant, not a derived one.

Can a model learn spot without any encoding? A causal decoder can partly infer it,. This is because masking makes spot observable through how many tokens are visible. It is a weak sign, and explicit encoding works far better.

Recap in one screen

• Self-attention is permutation-invariant, so order must be supplied apart.
• Sinusoidal encodings are fixed patterns at many counts, parameter-free and extendable.
• Learned absolute encodings work well up to the trained length and stop dead beyond it.
• RoPE rotates queries and keys by spot, so attention scores depend on relative distance. And it is what current models use.
• Long-context versions of models come from reinterpreting RoPE's angles, plus a short fine-tune.