Positional Encoding

The obvious ideas, and why they fail

• Append the integer position. Position 512 then contributes a value five hundred times larger than position 1, swamping the embedding, and the model has never seen position 3000 during training.
• Append position / length. Now it is bounded, but the same value means different things in different sentences — 0.5 is word 5 of 10 and word 50 of 100. Nothing consistent can be learned from it.

Both are available in the Scheme control above. What is wanted is something bounded, unique per position, consistent across sentence lengths, and — ideally — carrying information about relative distance, since "the adjective three words back" is a far more useful notion than "the word at index 17".

The sinusoidal answer

The original transformer used a fixed function of position, with a different frequency in every pair of dimensions:

PE(pos, 2i) = sin( pos / 10000^2i/d )

PE(pos, 2i+1) = cos( pos / 10000^2i/d )

The wavelengths form a geometric series from about 2 up to about 10000·2π. Early dimensions flip every couple of positions; late dimensions barely move across the whole sequence. Together they behave like the digits of a binary counter — the fast bits distinguish neighbours, the slow bits distinguish regions — which is why the matrix above has that striped, fanning appearance.

Every value stays inside −1 to 1 no matter how long the sequence, and the pattern for position 500 is defined whether or not the model ever trained on a sequence that long.

Attention cannot see order

Self-attention computes a weighted mixture over all positions. Shuffle the input tokens and the same set of outputs comes back, permuted — the mechanism treats the sequence as a bag.

For language that is fatal. "The dog bit the man" and "The man bit the dog" contain identical tokens, and a model without positional information cannot tell them apart.

So order has to be injected separately. That is the entire job of positional encoding, and there are three families of answer.

Sinusoidal encodings

The original scheme adds a fixed pattern to each token's embedding, using sines and cosines at geometrically spaced frequencies:

PE(pos, 2i) = sin(pos / 10000^2i/d)   PE(pos, 2i+1) = cos(pos / 10000^2i/d)

Each dimension oscillates at a different rate. Low dimensions change quickly with position — useful for distinguishing neighbours — and high dimensions change slowly, encoding coarse location. Together they give every position a unique signature, rather like a binary counter in continuous form.

Two properties made this attractive. It requires no parameters, and it extends to positions never seen during training, because the formula is defined for any pos. There is also a neat mathematical fact: the encoding for position pos + k is a linear function of the encoding for pos, so relative offsets are representable.

Its weakness is that it encodes absolute position, and what usually matters in language is relative distance — "the adjective two words before this noun" rather than "the word at index 47".

Rotary embeddings, and why they won

RoPE takes a different route. Instead of adding something to the embedding, it rotates the query and key vectors by an angle proportional to their position, in each two-dimensional slice of the vector.

The consequence falls out of trigonometry: when a rotated query is dotted with a rotated key, the result depends on the difference between their positions, not on their absolute values. Relative position is built into the attention score itself, without any extra terms.

That gives three practical advantages, which is why essentially every current large model uses it:

• Relative by construction, matching what language actually needs.
• Extends further beyond the training length, and can be stretched deliberately with interpolation tricks (NTK scaling, YaRN) to extend context after training.
• No parameters, and it applies inside attention rather than at the input, so it affects every layer consistently.

ALiBi is the other approach worth knowing: it adds a penalty to attention scores proportional to distance, so tokens attend less to things far away. Simple, parameter-free, and it extrapolates well — at the cost of building in a bias towards locality.

Giving attention a sense of order

Attention is permutation-invariant -- this proves it, then builds the sinusoidal encoding that fixes it and checks the property that makes it work.

import numpy as np

rng = np.random.default_rng(0)

def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def attend(E, Wq, Wk, Wv):
    Q, K, V = E @ Wq, E @ Wk, E @ Wv
    A = softmax(Q @ K.T / np.sqrt(Q.shape[1]))
    return A, A @ V

T, D = 5, 8
E = rng.normal(0, 1.0, (T, D))
Wq, Wk, Wv = (rng.normal(0, 0.5, (D, D)) for _ in range(3))

print("FIRST, THE PROBLEM. attention has no idea what order the tokens are in.")
A1, out1 = attend(E, Wq, Wk, Wv)
perm = np.array([3, 0, 4, 1, 2])
A2, out2 = attend(E[perm], Wq, Wk, Wv)
print("   token 3's output in the original order  :", np.round(out1[3][:4], 5))
print("   the same token after shuffling the input:", np.round(out2[0][:4], 5))
print("   identical:", np.allclose(out1[perm], out2))
print()
print("   shuffle the sentence and every token gets exactly the same answer,")
print("   just moved. 'dog bites man' and 'man bites dog' are the same input")
print("   to a transformer. that is not a bug to fix in the attention -- it")
print("   is a property of a weighted sum, and the fix has to come from the")
print("   embeddings.")
print()

D_MODEL, MAXLEN = 64, 128
pos = np.arange(MAXLEN)[:, None]
i = np.arange(D_MODEL // 2)[None, :]
denom = 10000 ** (2 * i / D_MODEL)
PE = np.zeros((MAXLEN, D_MODEL))
PE[:, 0::2] = np.sin(pos / denom)
PE[:, 1::2] = np.cos(pos / denom)

print("THE SINUSOIDAL ENCODING. each dimension is a wave, and the")
print("wavelengths grow geometrically across the dimensions:")
print("%10s %16s %20s" % ("dim pair", "wavelength", "what it distinguishes"))
for k in (0, 8, 20, 31):
    wl = 2 * np.pi * 10000 ** (2 * k / D_MODEL)
    print("%10d %16.1f %20s"
          % (k, wl, "adjacent tokens" if wl < 20 else
             ("nearby phrases" if wl < 500 else "whole document")))
print("   fast dimensions separate neighbours; slow ones separate distant")
print("   regions. together they give every position a unique fingerprint.")
print()

print("the first few positions, first 6 dimensions:")
print("%8s %s" % ("pos", "".join("%9d" % d for d in range(6))))
for p in (0, 1, 2, 3, 10, 50):
    print("%8d %s" % (p, "".join("%9.4f" % v for v in PE[p][:6])))
print()

norms = np.linalg.norm(PE, axis=1)
print("every position has the same magnitude: %.4f to %.4f"
      % (norms.min(), norms.max()))
print("   so adding it to a token embedding shifts the token without")
print("   changing how loud it is.")
print()

print("similarity between position 0 and the rest -- it falls away with")
print("distance, though not perfectly smoothly:")
unit = PE / np.linalg.norm(PE, axis=1, keepdims=True)
for p in (0, 1, 2, 5, 10, 25, 50, 100):
    print("   pos 0 vs pos %3d: %+.4f  %s"
          % (p, unit[0] @ unit[p], "#" * max(0, int(30 * unit[0] @ unit[p]))))
print("   the trend is downward, with ripples -- those come from the faster")
print("   dimensions cycling back into phase. a model can still read")
print("   'nearby' off this, which is what matters.")
print()

print("THE PROPERTY IT WAS DESIGNED FOR: a fixed offset is a fixed linear")
print("map. PE(pos + k) can be computed from PE(pos) by one rotation that")
print("depends only on k, not on pos:")
for k in (1, 5, 20):
    sims = [unit[p] @ unit[p + k] for p in range(0, 80)]
    print("   offset k=%2d: similarity between pos and pos+k is %.4f +/- %.6f"
          % (k, np.mean(sims), np.std(sims)))
print("   the standard deviation is essentially zero -- the relationship")
print("   between two positions depends only on the GAP between them. that")
print("   is what lets a model generalise to sentence lengths it never saw.")
print()

print("and it does solve the original problem:")
Emb = rng.normal(0, 1.0, (T, D_MODEL))
Wq2, Wk2, Wv2 = (rng.normal(0, 0.5, (D_MODEL, D_MODEL)) for _ in range(3))
with_pos = Emb + PE[:T]
A3, out3 = attend(with_pos, Wq2, Wk2, Wv2)
A4, out4 = attend((Emb + PE[:T])[perm], Wq2, Wk2, Wv2)
print("   with positional encoding, shuffling now changes the answer:")
print("      max difference: %.4f" % np.abs(out3[perm] - out4).max())
print("   (the shuffled version keeps each token's ORIGINAL position vector,")
print("    which is why it differs -- position is now part of the input.)")
print()

print("the alternatives, briefly:")
print("   learned absolute -- a lookup table of position vectors. simple,")
print("      and cannot extrapolate past the longest length you trained on.")
print("   RoPE -- rotate Q and K by an angle proportional to position, so")
print("      the dot product depends on the gap directly. what most current")
print("      open models use.")
print("   ALiBi -- add a distance penalty straight to the attention scores.")
print("      no vectors at all, and extrapolates well.")
print()
print("all three are answers to the same question: a weighted sum does not")
print("know what order its inputs were in, so the order has to be encoded")
print("into the values being summed.")

Experiments to try

1. Break word order. Set the Scheme to None and tick Swap Two Words. Input Difference reads 0.000 and the two sentences are literally identical to the model. Switch the Scheme back to Sinusoidal and the difference becomes non-zero — the encoding is what makes them different inputs.
2. See the frequency fan. Look at the matrix with the Model Dimension slider at 64. The leftmost columns stripe rapidly down the page while the rightmost are almost flat. That spread of wavelengths is the whole design.
3. Check it encodes distance. Set the Inspect Position slider to 12 and look at the lower panel. Similarity peaks at position 12 and falls away smoothly on both sides, so nearby positions have similar encodings. Move the position and the peak travels with it, keeping its shape — the pattern depends on the gap, not on where you are.
4. Try the naive schemes. Set the Scheme to Raw integer position, then drag the Inspect Position slider up and watch Largest Value climb with it, without bound. Compare against Sinusoidal, where it never leaves 1. Then try Position / length and lengthen the sequence: every encoding shifts, because the same position now means a different fraction.
5. Push past training length. Set the Sequence Length slider to its maximum with Sinusoidal selected. The values stay inside −1 to 1 and every row is still distinct. Nothing had to be learned for those positions to exist.

Added, not concatenated

The encoding is added to the word embedding, not stapled onto the end of it. That surprises people — surely adding corrupts the meaning?

In practice it does not, and the usual explanation is that in a space of several hundred dimensions there is room for the model to keep the two kinds of information in largely separate subspaces, and the projections that follow can pull them apart again. Concatenating would be cleaner conceptually but would spend dimensions that are more useful elsewhere.

What is used now

• Learned absolute — a plain embedding table indexed by position, used by BERT and GPT-2. Simple and works well, but has a hard maximum length and learns nothing about positions beyond it.
• Relative — bias the attention scores by the distance between the two positions rather than encoding absolutes at all. Closer to what attention actually wants to know.
• RoPE (rotary) — rotate the query and key vectors by an angle proportional to position, so the dot product between them depends on their difference by construction. Now the default in most open-weight models.

Sinusoidal encoding is no longer the state of the art, but it is the clearest illustration of what the problem is, and RoPE is easier to understand once you have seen it.

Where this goes wrong

• Leaving it out. The model still trains and the loss still falls, so the bug is silent — it just performs like a bag-of-words model and nobody knows why.
• Exceeding the trained length. With learned encodings, positions past the table simply do not exist. Most context-extension work is about making positional information survive past where it was trained.
• Adding it at every layer. The original adds it once, at the input. Re-adding it at each layer is a common misreading of the diagram.
• Assuming the model reads it as a number. It is a pattern, not a counter. What the model learns is which patterns co-occur, not that dimension 7 means "position 12".

Where that leaves you

Self-attention has no notion of order, so position has to be supplied explicitly, and the naive options fail: raw indices grow without bound and normalised ones mean different things in different-length sentences. Sinusoidal encoding solves it with a bank of sine and cosine waves at geometrically spaced frequencies, giving every position a bounded, unique fingerprint that is defined for any length and whose similarity to other positions depends on the distance between them. It is added to the embedding rather than concatenated, and modern models mostly use learned or rotary variants — but all of them exist to answer the same question this one does.

Learned absolute encodings, and their limit

The simplest option is a trainable vector per position — effectively an embedding table indexed by position instead of by token. BERT and GPT-2 both did this.

It works well within the trained range and has one hard limitation: position 512 has no vector if the model was trained to 512. There is nothing to look up, so the model cannot process longer input at all without adding and training new rows.

That single constraint is a large part of why the field moved to rotary and ALiBi schemes. Extending a context window from 4,000 to 128,000 tokens is a plausible fine-tuning project with RoPE and effectively impossible with learned absolute positions.

Extending context after training

Because RoPE is a formula rather than a table, the angles can be reinterpreted — and that is how long-context versions of existing models are produced.

Position interpolation scales positions down so that a longer sequence maps into the range the model was trained on. Cheap, and it compresses the model's sense of fine-grained distance.

NTK-aware scaling adjusts the frequency base rather than the positions, distorting high frequencies less and preserving local resolution better.

YaRN combines interpolation with attention-temperature adjustment and is the current standard for large extensions.

All three usually involve a short fine-tune on long sequences to let the model adapt. The reason any of it works is that RoPE's rotations are continuous in position, so intermediate and stretched values remain meaningful.

Common mistakes

• Assuming attention handles order. It does not, at all. This is the most common misconception about transformers.
• Feeding input longer than the positional scheme supports, which either errors or silently degrades.
• Mixing schemes — loading weights trained with learned positions into an architecture using RoPE produces fluent nonsense.
• Forgetting positions shift with padding. Left-padding a batch moves every real token's index; most implementations handle it, and custom code frequently does not.
• Expecting extrapolation for free. Even RoPE degrades well beyond its training length without an interpolation fix.

Questions people ask

Why add positional information rather than concatenate it? Addition keeps the dimension unchanged and works empirically. Concatenation would spend model width on position, and was tried and abandoned.

Does adding to the embedding not corrupt the meaning? Somewhat, and the model learns to separate the two — there is enough room in a few hundred dimensions for both signals to coexist.

Do vision transformers need positional encodings? Yes — image patches are also an unordered set without them. Learned 2-D encodings are common there.

Which should I use? RoPE, if you are choosing. In practice you use whatever the pretrained model uses.

Why 10,000 in the sinusoidal formula? It sets the range of frequencies, chosen so the wavelengths span from a couple of positions to tens of thousands. It is a tuned constant, not a derived one.

Can a model learn position without any encoding? A causal decoder can partly infer it, because masking makes position observable through how many tokens are visible. It is a weak signal, and explicit encoding works far better.

Recap in one screen

• Self-attention is permutation-invariant, so order must be supplied separately.
• Sinusoidal encodings are fixed patterns at many frequencies, parameter-free and extendable.
• Learned absolute encodings work well up to the trained length and stop dead beyond it.
• RoPE rotates queries and keys by position, so attention scores depend on relative distance — and it is what current models use.
• Long-context versions of models come from reinterpreting RoPE's angles, plus a short fine-tune.