Networks never see letters. Pick a word and follow it through the pipeline: word → ID → one-hot vector → dense embedding.
Overview
The Chain of Translations
A neural network is a pile of multiplications and additions — it can only consume numbers. So every word goes through a chain of translations: word → token ID → one-hot vector → dense embedding. Each stage exists to fix a shortcoming of the previous one.
Vocabulary (10 words)
Click a word to trace its representation. Click a second word to compare embeddings.
Similarity Check
Select two words to compare their dense embeddings.
Each cell is a learned feature. Similar words end up with similar numbers — that's what makes embeddings powerful.
From "cat" to a Vector: Word Representation in Neural Networks
The four stages every word passes through before a network can compute with it.
Token IDs and Their Trap
The dictionary lookup ("cat" → 1, "dog" → 3) is compact, but the raw integers smuggle in a false claim: that "dog" (3) is somehow three times "cat" (1), or that words with adjacent IDs are related. The IDs are arbitrary labels, and arithmetic on labels is meaningless — a network fed raw IDs will happily learn those fake relationships.
One-Hot: Honest but Wasteful
One-hot encoding fixes the fake-ordering problem: each word becomes a vector of zeros with a single 1 at its index. Now no word is numerically "bigger" than another. The costs:
Size: the vector is as long as the vocabulary — 50,000 dimensions for a modest one.
No similarity: every pair of one-hot vectors is exactly the same distance apart. "cat" is as far from "kitten" as from "carburetor".
Dense Embeddings: Small and Meaningful
An embedding layer maps each token ID to a short vector of learned real numbers (4 dims in the demo; 300–4096 in practice). Because these values are trained rather than assigned, words used in similar contexts drift toward similar vectors — similarity becomes measurable with a dot product. This single idea underpins everything from word2vec to the input layer of GPT.
From an index to a vector
Inside a network, a word is a row of a matrix. The path from text to that row is short:
The tokeniser maps the word (or subword) to an integer id.
That id indexes the embedding matrix, of shape (vocabulary size × embedding dimension).
The selected row is the word's vector, and it is what every subsequent layer operates on.
"cat" → id 2317 → row 2317 of a 50,000 × 768 matrix → 768 numbers
That matrix is learned. It starts as small random values and is updated by gradient descent along with everything else, so the geometry that emerges — which words end up near which — is a product of the training objective, not of anything designed.
Mathematically the lookup is equivalent to multiplying a one-hot vector by the matrix, which is why embeddings are sometimes described that way. Implementations never do it: nn.Embedding is a row lookup, because multiplying by a vector of 49,999 zeros is wasted work.
Why not one-hot all the way through
A one-hot representation has two defects that the embedding fixes.
Size. 50,000 dimensions per token, of which one is non-zero. A first layer taking that input needs 50,000 weights per unit.
No similarity. Every pair of words is exactly equidistant. "Cat" is as far from "dog" as from "bureaucracy", so nothing a model learns about one word transfers to a related one.
An embedding of 768 dimensions is 65 times smaller and places related words near each other, so evidence generalises. That combination — compact and meaningfully arranged — is what makes the representation useful.
One-hot
Embedding
Dimensions
Vocabulary size
100–1,024
Values
One 1, rest 0
All non-zero
Similar words
Equidistant
Nearby
Learned
No
Yes
Unseen word
Impossible to represent
Handled by subword pieces
Static and contextual
Static embeddings (Word2Vec, GloVe, and the embedding layer of a small model) give one vector per word, fixed after training. "Bank" gets a single vector that averages the river and the financial senses into something that is neither.
Contextual embeddings are what a transformer produces at each layer: the vector for "bank" in "river bank" has absorbed information from "river" and differs from the same token in "savings bank".
The embedding table still exists in a transformer — it supplies the input to layer one. Everything after that is contextual, refined by each block. So a transformer has both: a static lookup at the bottom, and increasingly context-dependent representations above it.
A detail worth knowing: the output projection to vocabulary size is frequently tied to the embedding matrix, using the same weights transposed. It saves a large number of parameters (50,000 × 768 is 38 million) and usually improves quality slightly, on the reasoning that the input and output spaces describe the same vocabulary.
From a string to a vector, in four steps
A word becomes an index, the index becomes a static vector, and the static vector becomes a context-dependent one. Each step is run here, ending with the same word getting two different representations in two sentences.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
vocab = ["<pad>", "<unk>", "the", "river", "bank", "money", "deposit",
"flooded", "closed", "at", "we", "sat", "by"]
idx = {w: i for i, w in enumerate(vocab)}
V, D = len(vocab), 8
print("STEP 1 -- the string becomes an INDEX. that is all a tokeniser")
print("ultimately produces:")
sent = "we sat by the river bank"
ids = [idx.get(w, idx["<unk>"]) for w in sent.split()]
print(" %r" % sent)
print(" %s" % list(zip(sent.split(), ids)))
print(" the index is arbitrary. nothing about %d means 'bank'." % idx["bank"])
print()
E = rng.normal(0, 0.5, (V, D))
print("STEP 2 -- the index becomes a VECTOR, by looking up a row:")
for w in ("bank", "river", "money"):
print(" %-8s id %2d -> %s" % (w, idx[w], np.round(E[idx[w]], 3)))
print(" %d x %d = %d numbers, and they are learned parameters like any"
% (V, D, E.size))
print(" others -- the gradient reaches them and moves them.")
print()
print(" the lookup IS a matrix multiply by a one-hot vector, skipped:")
oh = np.zeros(V); oh[idx["bank"]] = 1
print(" one-hot @ E == E[%d] : %s" % (idx["bank"], np.allclose(oh @ E, E[idx["bank"]])))
print()
print("STEP 3 -- but that vector is the SAME in every sentence. one row per")
print("word means one meaning per word:")
s1 = "we sat by the river bank".split()
s2 = "we closed the money bank".split()
for s in (s1, s2):
v = E[idx.get(s[-1], 1)]
print(" %-32s 'bank' -> %s" % (" ".join(s), np.round(v[:4], 3)))
print(" identical. word2vec and GloVe stop here, and that is their limit:")
print(" every sense of a word is averaged into one point.")
print()
def softmax(z):
e = np.exp(z - z.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
Wq, Wk, Wv = (rng.normal(0, 0.4, (D, D)) for _ in range(3))
def contextualise(tokens):
X = np.array([E[idx.get(t, 1)] for t in tokens])
A = softmax((X @ Wq) @ (X @ Wk).T / np.sqrt(D))
return A, X + A @ (X @ Wv) # a residual, as in a real block
print("STEP 4 -- mix in the neighbours, and the vector becomes")
print("CONTEXTUAL. one attention layer is enough to show it:")
for s in (s1, s2):
A, out = contextualise(s)
b = s.index("bank")
print(" %r" % " ".join(s))
print(" 'bank' attends most to '%s' (%.4f)"
% (s[int(A[b].argmax())], A[b].max()))
print(" its output vector: %s" % np.round(out[b][:4], 3))
A1, o1 = contextualise(s1)
A2, o2 = contextualise(s2)
b1, b2 = s1.index("bank"), s2.index("bank")
print()
print(" the same word, two sentences, two different vectors:")
print(" distance between them: %.4f" % np.linalg.norm(o1[b1] - o2[b2]))
print(" distance before the attention layer: %.4f"
% np.linalg.norm(E[idx["bank"]] - E[idx["bank"]]))
print(" that second number is exactly 0 by construction. the first is not.")
print(" THAT is what 'contextual embedding' means, and it is why asking")
print(" for 'the embedding of bank' stopped being a well-posed question.")
print()
print(" (the projections here are random, so WHICH neighbour each token")
print(" attends to is luck. what is not luck is that the two vectors")
print(" differ at all -- that follows from the mechanism, and a trained")
print(" model is the same mechanism with weights that mean something.)")
print()
print("THE FOUR REPRESENTATIONS, and where each one lives:")
rows = [("index", "an integer", "the tokeniser's output"),
("one-hot", "%d numbers, %d zeros" % (V, V - 1), "conceptual only"),
("static embedding", "%d numbers per word" % D, "the embedding table"),
("contextual", "%d numbers per POSITION" % D, "every layer's output")]
print("%22s %26s %s" % ("representation", "size", "where it comes from"))
for a, b, c in rows:
print("%22s %26s %s" % (a, b, c))
print()
print("and two practical notes. the embedding table is often the largest")
print("single parameter block in a model, and it is frequently TIED to the")
print("output layer -- the same matrix used to look words up is transposed")
print("to score them, halving the count.")
print()
print("the special tokens at the top of the vocabulary are not decoration:")
for t in ("<pad>", "<unk>"):
print(" %-8s id %d -- %s"
% (t, idx[t],
"fills short sequences, and must be masked out of the loss"
if t == "<pad>" else "anything outside the vocabulary"))
print(" a model that is trained to predict <pad> will learn to predict it.")
Output
Try it yourself
Click "cat". Follow all four stages — note the one-hot row has a single lit cell at index 1, and the embedding is just 4 numbers.
Click "dog" as the second word. The similarity panel shows a high cosine score — the demo embeddings encode that cats and dogs are both animals.
Compare "cat" with "on". An animal versus a preposition: the similarity collapses. One-hot vectors could never express this difference — embeddings do it natively.
Worth remembering
Word representation is a ladder: IDs are compact but lie about order, one-hot is honest but huge and similarity-blind, and dense embeddings are small, learned, and encode meaning as geometry. Modern NLP starts at the top of that ladder.
What the geometry contains
The arrangement that emerges from training is genuinely structured, in ways that can be measured.
Neighbourhoods are semantic. The nearest vectors to "January" are the other months. The nearest to "hospital" are clinic, doctor, patient.
Directions carry relations. The offset from "man" to "woman" is roughly the offset from "king" to "queen", which is what makes the famous analogy arithmetic work. It works cleanly for a small number of well-chosen relations and is frequently overstated.
Frequency affects magnitude. Rare words often have longer vectors than common ones, which is one reason cosine similarity — which ignores length — is preferred to the dot product for comparing them.
Bias is encoded too. The vectors reproduce the statistical associations of the training corpus, including its prejudices. This is a documented and repeatedly reproduced finding, and it matters whenever embeddings feed decisions about people.
Individual dimensions are not interpretable. Directions in the space often are, and probing classifiers can find them — there is a direction that separates singular from plural, another that tracks sentiment.
Practical notes
import torch.nn as nn
emb = nn.Embedding(num_embeddings=50_000, embedding_dim=768,
padding_idx=0) # keeps the pad vector at zero
x = emb(token_ids) # (batch, seq) -> (batch, seq, 768)
padding_idx matters: it keeps that row fixed at zero and excludes it from gradient updates, so padding contributes nothing.
Three further points that come up in real work:
The embedding table is often the largest single parameter block in a small model. 50,000 × 768 is 38 million weights, which may exceed all the transformer layers in a compact model.
Freezing pretrained embeddings is worth trying with small datasets — there is far less to overfit. Fine-tune them when you have enough data or a specialised vocabulary.
Dimension is a real trade-off. 100–300 for word vectors, 768–4,096 in transformers. Larger captures more and costs memory and compute at every layer.
Questions people ask
Is the embedding matrix the same as a one-hot multiplication? Mathematically yes, computationally no — it is implemented as a row lookup.
Are embeddings trained or downloaded? Both are used. For text, downloading pretrained is almost always better. For categorical features in your own model, trained jointly.
What dimension should I use? 300 for standalone word vectors; whatever the pretrained model uses otherwise. For categorical features, roughly the fourth root of the cardinality as a starting point.
How are unseen words handled? Subword tokenisation splits them into known pieces, so there is always something to look up.
Can I inspect what a dimension means? Not usefully. Directions in the space are more interpretable than axes, and probing is the tool for finding them.
Why tie the input and output embeddings? It saves tens of millions of parameters and typically helps slightly, since both describe the same vocabulary.
Recap in one screen
A word becomes an integer id, which indexes a learned embedding matrix; the selected row is its vector.
The lookup is equivalent to a one-hot multiplication and is implemented as indexing.
Embeddings are compact and place similar words nearby, so learning transfers between related words.
Transformers keep a static table at the input and produce contextual representations above it.
The geometry encodes semantics, relational directions, frequency effects and corpus bias.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Word representation is a ladder: IDs are compact but lie about order, one-hot is honest but huge and similarity-blind, and dense embeddings are small, learned, and encode meaning as geometry. Modern NLP starts at the top of that ladder.
What does this module say about “The Chain of Translations”?
A neural network is a pile of multiplications and additions — it can only consume numbers. So every word goes through a chain of translations: word → token ID → one-hot vector → dense embedding . Each stage exists to fix a shortcoming of the previous one.
What does this module say about “Token IDs and Their Trap”?
The dictionary lookup ("cat" → 1, "dog" → 3) is compact, but the raw integers smuggle in a false claim: that "dog" (3) is somehow three times "cat" (1), or that words with adjacent IDs are related. The IDs are arbitrary labels, and arithmetic on labels is meaningless — a network fed raw IDs will happily learn those fake relationships.
Cheat sheet
How Words are Represented in Neural Networks
A neural network is a pile of multiplications and additions — it can only consume numbers. So every word goes through a chain of translations: word → token ID → one-hot vector → dense embedding. Each stage exists to fix a shortcoming of the previous one.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.