How an image becomes a sequence of tokens, and what that costs when the patches get smaller.
Overview
Images are grids; transformers want sequences
A transformer operates on a sequence of vectors and has no built-in notion of two dims. An image is a grid of pixels. a thing has to convert one into the other.
The obvious approach — one token per pixel — is hopeless. A 224×224 image is 50,176 pixels, and focus costs grow with the square of the sequence length, so that is 2.5 billion pairwise interplays per layer.
The Vision Transformer's answer is almost aggressively simple: cut the image into square patches and treat each patch as a token.
Vision Transformer Patches
This module needs JavaScript: the
images are computed in the page rather than downloaded.
Worth knowing
A transformer takes a sequence of vectors. An image is a grid, so it has to be cut into patches and flattened.
Each patch is flattened to P²×3 numbers and projected to the model's width by a single linear layer.
The patch grid is the resolution the model sees. There is no convolution underneath it.
Attention cost grows with the square of the token count, so halving the patch size costs sixteen times the attention.
Vision Transformer Patches
The one step that lets a language architecture read a photo. The quadratic cost hiding in it.
The whole preprocessing step
Drag the patch control and watch the grid change. That is truly all that happens to the image before it enters the model:
Cut into non-shared P×P patches.
Flatten each patch to a vector of P * P * 3 numbers.
Times by one learned mat to get the model width.
Add a spot-based embedding, because or else the order is unknown.
At the default of 16 pixels on a 224-pixel image, that is a 14×14 grid, 196 tokens, each starting as 768 raw numbers. The readout gives the same figures for what patch size you choose.
Step 3 is the only learned part, and it's a linear map with no non-linearity. Equivalently, it's a convolution with kernel size P and stride P — which is how it's often built, and a nice reminder that the gap between "convolutional" and "not" is thinner than the naming suggests.
The quadratic cost
Halve the patch size and the token count quadruples. The grid gains a factor of two in each dim. Focus cost goes with the square of the token count, so it rises sixteenfold.
Patch
Tokens (224px)
Attention cost
32
49
1×
16
196
16×
8
784
256×
4
3,136
4,096×
That table is the central constraint on ViT design, and it shows a great deal of what followed. Patch size isn't a minor setting. It's the detail at which the model sees any of it, traded right against find.
What is lost inside a patch
A 16×16 patch is compressed to a single vector. Structure within it isn't attended to at all. What the map preserves is what lives.
This is the ViT's real weakness at small scale. A convolutional network has locality built in — nearby pixels are processed together by build — and this inductive bias matches how images really work. A ViT has none of it. each tie, including "these two patches are adjacent", must be learned from data through the spot-based embeddings.
Which is why the first paper's headline finding was about data. On ImageNet alone the ViT underperformed a ResNet. Pre-trained on 300 million images it overtook it. With enough data the model learns the structure that convolution assumes, and gains flexibility convolution doesn't have.
Two responses followed. DeiT showed careful boost and distillation could train a ViT on ImageNet alone. Swin Transformer reintroduced locality by restricting focus to windows and merging patches hierarchically, which gives back the pyramid a CNN has and makes dense guess hands-on.
Positional embeddings
Focus is permutation-rule: shuffle the tokens and the output is shuffled the same way, with nothing else changed. Without spot-based info a ViT truly can't tell a photo from a jigsaw of itself.
The fix is to add a learned vector per spot. It's worth appreciating how weak this is checked with convolution: the model is told "this is spot 37" and must learn from data that spot 37 is next to 36 and above 23.
Changing the input detail changes the number of spots. This Is why ViTs interpolate their spot-based embeddings when fine-tuned at a different size.
The patch size is the whole budget
Cutting the image into patches decides all downstream, and the token count makes it concrete. A 224-pixel image at patch size 16 gives a 14×14 grid = 196 tokens, each starting as 16×16×3 = 768 raw numbers, and focus costs the square of the token count:
patch
tokens
attention pairs
relative cost
32
49
2,401
1×
16
196
38,416
16×
8
784
614,656
256×
Halve the patch and the token count quadruples, so focus rises sixteenfold. That is why patch size isn't a minor setting but the detail at which the model sees any of it, traded right against find. And why one token per pixel (50,176 tokens, 2.5 billion pairs a layer) is hopeless. The cost inside a patch is subtler: a 16×16 region is projected to one vector, so structure within it's never attended to, and unlike a convolution the ViT has no built-in locality — it must *learn* that patch 37 sits beside 36, from data, through the spot-based embeddings. Which is why the first ViT needed 300 million pre-training images to beat a ResNet that had locality for free.
Focus is permutation-rule — shuffle the tokens and the output is shuffled the same way. Without spot-based embeddings a ViT truly can't tell a photo from a jigsaw of itself. The fix, a learned vector per spot, is strikingly weak next to convolution: the model is told "this is spot 37" and must learn from data that 37 sits beside 36 and above 23. That missing inductive bias is why the plain ViT needed huge pre-training, and why two responses followed — DeiT showed distillation and heavy boost could train one on ImageNet alone, and Swin reintroduced locality by restricting focus to windows and merging patches hierarchically, getting back the pyramid a CNN has for free and making dense guess hands-on.
Where it goes wrong with Vision Transformer Patches
Shrinking the patch to gain detail. The focus cost is quadratic in tokens, so this gets costly faster than expected.
Training a plain ViT on a small dataset. Without the inductive bias it needs either far more data or the DeiT training recipe.
Losing spot-based embeddings on a resized input. The count must match, and interpolation is needed.
given patches must be square and non-shared. shared patches help. A few later architectures use them.
Questions people ask about Vision Transformer Patches
How does a Vision Transformer turn an image into tokens? It cuts the image into fixed-size patches, such as 16x16, flattens each one and projects it linearly into an embedding. Each patch acts as one token.
Why 16x16 patches? It is a trade-off. Smaller patches give more tokens and finer detail but attention cost grows with the square of the token count.
Why does a ViT need position embeddings? Self-attention ignores order, so without them the model would see patches as an unordered set. Learned position embeddings tell it where each patch came from.
What is the CLS token? A learnable extra token prepended to the sequence. Its final state aggregates information from all patches and feeds the classification head.
Does a ViT need more data than a CNN? Generally yes. It lacks built-in locality, so it needs large datasets or strong augmentation and distillation to match CNNs.
How many tokens does a 224x224 image give? With 16x16 patches, 14 times 14 equals 196 tokens, plus the CLS token.
Cutting an image into tokens
A vision transformer starts by chopping the image into patches and flattening each one into a vector. That single step decides the model's cost, its detail limit. All it doesn't know about geometry -- which it then has to be told.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
IMG, P = 12, 4
img = np.arange(IMG * IMG).reshape(IMG, IMG) % 10
print("THE IMAGE, %dx%d, with a %dx%d patch grid:" % (IMG, IMG, P, P))
for i in range(IMG):
if i % P == 0:
print(" " + "+----" * (IMG // P) + "+")
row = ""
for j in range(IMG):
row += ("|" if j % P == 0 else "") + str(img[i, j])
print(" " + row + "|")
print(" " + "+----" * (IMG // P) + "+")
n = (IMG // P) ** 2
patches = []
for i in range(0, IMG, P):
for j in range(0, IMG, P):
patches.append(img[i:i+P, j:j+P].reshape(-1))
patches = np.array(patches, float)
print(" %d patches of %dx%d = %d pixels each." % (n, P, P, P*P))
print("EACH PATCH IS FLATTENED INTO A VECTOR:")
print(" patch 0: %s" % " ".join("%.0f" % v for v in patches[0]))
print(" patch 1: %s" % " ".join("%.0f" % v for v in patches[1]))
print(" shape %s -- %d tokens of %d dims, exactly what a text" % (str(patches.shape), n, P*P))
print(" transformer expects. turn the image into a sentence.")
Output
The next block shows that the patch-flatten-project step is identical to a convolution with kernel size and stride equal to the patch size.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
IMG, P, D = 12, 4, 8
img = np.arange(IMG*IMG).reshape(IMG, IMG) % 10
patches = []
for i in range(0, IMG, P):
for j in range(0, IMG, P):
patches.append(img[i:i+P, j:j+P].reshape(-1))
patches = np.array(patches, float)
E = rng.normal(0, 0.3, (P*P, D))
tokens = patches @ E
print("LINEAR PROJECTION: patches @ E, E is %s -> %s" % (str(E.shape), str(tokens.shape)))
print(" 'cut, flatten, multiply by E' IS a conv with kernel=%d, stride=%d." % (P, P))
K = E.T.reshape(D, P, P)
conv_out = np.zeros((D, IMG//P, IMG//P))
for d in range(D):
for a, i in enumerate(range(0, IMG, P)):
for b, j in enumerate(range(0, IMG, P)):
conv_out[d, a, b] = (img[i:i+P, j:j+P] * K[d]).sum()
print(" max difference from a real convolution: %.2e"
% np.abs(conv_out.reshape(D, -1).T - tokens).max())
print(" identical. every ViT uses nn.Conv2d(3, D, %d, %d) for this." % (P, P))
Output
The next block demonstrates the permutation problem: without position embeddings, shuffling the patches produces the same token set, so the model cannot tell a scrambled image from the original.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
IMG, P, D = 12, 4, 8
img = np.arange(IMG*IMG).reshape(IMG, IMG) % 10
n = (IMG // P) ** 2
patches = []
for i in range(0, IMG, P):
for j in range(0, IMG, P):
patches.append(img[i:i+P, j:j+P].reshape(-1))
patches = np.array(patches, float)
E = rng.normal(0, 0.3, (P*P, D))
tokens = patches @ E
perm = rng.permutation(n)
pos = np.zeros((n, D))
for k in range(n):
for d in range(0, D, 2):
w = 1.0 / (10000 ** (d / float(D)))
pos[k, d] = np.sin(k * w); pos[k, d+1] = np.cos(k * w)
print("POSITION EMBEDDING. without it, attention is permutation equivariant:")
def as_multiset(M): return M[np.lexsort(M.T[::-1])]
def same(A, B): return "identical" if np.allclose(as_multiset(A), as_multiset(B)) else "DIFFERENT"
print(" without position: tokens vs tokens[perm] -> %s" % same(tokens, tokens[perm]))
print(" with position: t+pos vs t[perm]+pos -> %s" % same(tokens+pos, tokens[perm]+pos))
print(" position is the only thing that breaks the symmetry.")
print(" most modern ViTs use a LEARNED position embedding -- a plain")
print(" lookup table trained with the rest. the model must learn from")
print(" data that patch 0 is next to patch 1; a CNN has that built in.")
Output
The next block tabulates the attention cost as the patch size shrinks, then shows that a patch is the smallest unit the model can reason about.
example_04.pyNumPy
import numpy as np
print("PATCH SIZE COST. attention is quadratic in the token count:")
print("%-12s %8s %14s %14s" % ("patch", "tokens", "attn ops", "relative"))
for p in (32, 16, 14, 8, 4):
t = (224 // p) ** 2; ops = t * t
print("%-12d %8d %14s %13.1fx"
% (p, t, "{:,}".format(ops), ops / float((224//32)**4)))
print(" 32->4 multiplies cost by %d. that is why ViT-B/16 exists"
% int(((224//4)**4)/((224//32)**4)))
print(" and ViT-B/2 does not.")
print()
rng = np.random.default_rng(0)
P, D = 4, 8
img = np.arange(144).reshape(12, 12) % 10
patches = np.array([img[i:i+P, j:j+P].reshape(-1)
for i in range(0,12,P) for j in range(0,12,P)], float)
E = rng.normal(0, 0.3, (P*P, D))
flat = patches[0].copy(); scrambled = flat.copy(); rng.shuffle(scrambled)
pa, pb = patches[0] @ E, scrambled @ E
print("RESOLUTION LIMIT. a %dx%d patch is ONE token." % (P, P))
print(" shuffling its pixels changes the vector by %.2f in norm."
% np.linalg.norm(pa - pb))
print(" information is preserved, but ADDRESSING is not: a patch is the")
print(" smallest thing attention can point at.")
print(" ViT-B/16 maps 768 inputs to D=768, losing nothing. the limit")
print(" is the patch grid itself, so halving patch size is the only way")
print(" to see finer detail -- and the cost table above is the wall.")
print()
print("THE DESIGN TRADE: a ViT gives up the CNN's spatial prior for the")
print("freedom to relate any patch to any other in one step. that is why")
print("ViTs need far more data to match a CNN, and why with enough data")
print("they exceed what the prior would have allowed.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does the projection step applied to each patch amount to?
It is the only learned part of the patching step and it has no non-linearity - which is why it is usually implemented as a strided convolution.
Halving the patch size multiplies attention cost by roughly how much?
Token count quadruples because the grid doubles in each dimension, and attention is quadratic in token count, so cost rises sixteenfold.
Why did the original ViT underperform a ResNet on ImageNet alone?
Pre-trained on 300 million images it overtook the ResNet. With enough data it learns the structure convolution assumes, and gains flexibility convolution lacks.
Cheat sheet
Vision Transformer Patches
Split an image into fixed-size patches, project each into a token, and follow the sequence through a transformer to see its quadratic cost.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.