Each cell is a learned feature. Similar words end up with similar numbers — that's what makes embeddings powerful.
Overview
The Chain of Translations
A neural network is a pile of products and additions — it can only consume numbers. So each word goes through a chain of translations: word → token ID → one-hot vector → dense embedding. Each stage exists to fix a shortcoming of the last one.
Token IDs assign each word an integer, which is compact but implies a fake ordering. One-hot vectors remove the ordering by giving each word its own dimension. But the vectors are enormous (one dimension per vocabulary word) and each pair of words is equally distant — "cat" is as far from "kitten" as from "parliament". Dense embeddings solve both problems: each word maps to a short vector of learned real numbers (often 128 to 4096 dimensions). Where words used in close contexts drift toward close vectors.
The embedding matrix is the bridge. It has one row per vocabulary entry and one column per embedding dimension. Looking up a word means selecting its row — a single index step, not a product. During training, backpropagation adjusts these rows so that semantically related words end up in nearby regions of the vector space. After training, you can measure likeness between words with a dot product or cosine likeness. And the results align with human intuition: "king" minus "man" plus "woman" lands near "queen".
Modern language models go one step further with context embeddings. In a static embedding (Word2Vec, GloVe), the word "bank" always has the same vector, whether it means a riverbank or a financial institution. In a context embedding (BERT, GPT), the vector for "bank" depends on the sentence around it. The embedding layer still exists — it gives the initial form — but the transformer layers above it mix data from the nearby tokens. This gives a vector that encodes both the word's meaning and its role in this set sentence.
From "cat" to a Vector: Word Representation in Neural Networks
The four stages every word passes through before a network can compute with it.
The Chain of Translations
A neural network is a pile of products and additions — it can only consume numbers. So each word goes through a chain of translations: word → token ID → one-hot vector → dense embedding. Each stage exists to fix a shortcoming of the last one.
Token IDs assign each word an integer, which is compact but implies a fake ordering. One-hot vectors remove the ordering by giving each word its own dimension. But the vectors are enormous (one dimension per vocabulary word) and each pair of words is equally distant — "cat" is as far from "kitten" as from "parliament". Dense embeddings solve both problems: each word maps to a short vector of learned real numbers (often 128 to 4096 dimensions). Where words used in close contexts drift toward close vectors.
The embedding matrix is the bridge. It has one row per vocabulary entry and one column per embedding dimension. Looking up a word means selecting its row — a single index step, not a product. During training, backpropagation adjusts these rows so that semantically related words end up in nearby regions of the vector space. After training, you can measure likeness between words with a dot product or cosine likeness. And the results align with human intuition: "king" minus "man" plus "woman" lands near "queen".
Modern language models go one step further with context embeddings. In a static embedding (Word2Vec, GloVe), the word "bank" always has the same vector, whether it means a riverbank or a financial institution. In a context embedding (BERT, GPT), the vector for "bank" depends on the sentence around it. The embedding layer still exists — it gives the initial form — but the transformer layers above it mix data from the nearby tokens. This gives a vector that encodes both the word's meaning and its role in this set sentence.
Token IDs and Their Trap
The dictionary lookup ("cat" → 1, "dog" → 3) is compact. But the raw integers smuggle in a false claim. That "dog" (3) is somehow three times "cat" (1). Or that words with adjacent IDs are related. The IDs are arbitrary labels, and arithmetic on labels is meaningless — a network fed raw IDs will happily learn those fake links.
One-Hot: Honest but Wasteful
One-hot encoding fixes the fake-ordering problem: each word becomes a vector of zeros with a single 1 at its index. Now no word is in numbers "bigger" than another. The costs:
Size: the vector is as long as the vocabulary — 50,000 dimensions for a modest one.
No likeness: each pair of one-hot vectors is just the same distance apart. "cat" is as far from "kitten" as from "carburetor".
Dense Embeddings: Small and Meaningful
An embedding layer maps each token ID to a short vector of learned real numbers (4 dims in the demo; 300–4096 in practice). Because these values are trained rather than assigned, words used in close contexts drift toward close vectors — likeness becomes measurable with a dot product. This single idea underpins all of it from word2vec to the input layer of GPT.
From an index to a vector
Inside a network, a word is a row of a matrix. The path from text to that row is short:
The tokeniser maps the word (or subword) to an integer id.
That id indexes the embedding matrix, of shape (vocabulary size × embedding dimension).
The selected row is the word's vector, and it is what each later layer operates on.
"cat" → id 2317 → row 2317 of a 50,000 × 768 matrix → 768 numbers
That matrix is learned. It starts as small random values and is updated by gradient descent along with all of it else. So the geometry that emerges. Which words end up near which. Is a product of the training objective, not of anything designed.
In math the lookup is the same to multiplying a one-hot vector by the matrix. This is why embeddings are sometimes described that way. Implementations never do it: nn.Embedding is a row lookup, because multiplying by a vector of 49,999 zeros is wasted work.
Why not one-hot all the way through
A one-hot form has two defects that the embedding fixes.
Size. 50,000 dimensions per token, of which one is non-zero. A first layer taking that input needs 50,000 weights per unit.
No likeness. Each pair of words is just equidistant. "Cat" is as far from "dog" as from "bureaucracy", so nothing a model learns about one word transfers to a related one.
An embedding of 768 dimensions is 65 times smaller and places related words near each other, so evidence extends. That combination — compact and meaningfully arranged — is what makes the form useful.
One-hot
Embedding
Dimensions
Vocabulary size
100–1,024
Values
One 1, rest 0
All non-zero
Similar words
Equidistant
Nearby
Learned
No
Yes
Unseen word
Impossible to represent
Handled by subword pieces
Static and context
Static embeddings (Word2Vec, GloVe, and the embedding layer of a small model) give one vector per word, fixed after training. "Bank" gets a single vector that averages the river and the financial senses into something that is neither.
Context embeddings are what a transformer gives at each layer. The vector for "bank" in "river bank" has absorbed data from "river" and differs from the same token in "savings bank".
The embedding table still exists in a transformer — it supplies the input to layer one. All of it after that is context, refined by each block. So a transformer has both: a static lookup at the bottom, and increasingly context-dependent forms above it.
A detail worth knowing: the output projection to vocabulary size is often tied to the embedding matrix, using the same weights transposed. It saves many parameters (50,000 × 768 is 38 million) and often improves quality slightly, on the reasoning that the input and output spaces describe the same vocabulary.
From a string to a vector, in four steps
A word becomes an index, the index becomes a static vector. And the static vector becomes a context-dependent one. Each step is run here, ending with the same word getting two different forms in two sentences.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
vocab = ["<pad>", "<unk>", "the", "river", "bank", "money", "deposit",
"flooded", "closed", "at", "we", "sat", "by"]
idx = {w: i for i, w in enumerate(vocab)}
V, D = len(vocab), 8
print("STEP 1 -- the string becomes an INDEX. that is all a tokeniser")
print("ultimately produces:")
sent = "we sat by the river bank"
ids = [idx.get(w, idx["<unk>"]) for w in sent.split()]
print(" %r" % sent)
print(" %s" % list(zip(sent.split(), ids)))
print(" the index is arbitrary. nothing about %d means 'bank'." % idx["bank"])
print()
E = rng.normal(0, 0.5, (V, D))
print("STEP 2 -- the index becomes a VECTOR, by looking up a row:")
for w in ("bank", "river", "money"):
print(" %-8s id %2d -> %s" % (w, idx[w], np.round(E[idx[w]], 3)))
print(" %d x %d = %d numbers, and they are learned parameters like any"
% (V, D, E.size))
print(" others -- the gradient reaches them and moves them.")
print()
print(" the lookup IS a matrix multiply by a one-hot vector, skipped:")
oh = np.zeros(V); oh[idx["bank"]] = 1
print(" one-hot @ E == E[%d] : %s" % (idx["bank"], np.allclose(oh @ E, E[idx["bank"]])))
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
print("STEP 3 -- but that vector is the SAME in every sentence. one row per")
print("word means one meaning per word:")
s1 = "we sat by the river bank".split()
s2 = "we closed the money bank".split()
for s in (s1, s2):
v = E[idx.get(s[-1], 1)]
print(" %-32s 'bank' -> %s" % (" ".join(s), np.round(v[:4], 3)))
print(" identical. word2vec and GloVe stop here, and that is their limit:")
print(" every sense of a word is averaged into one point.")
print()
def softmax(z):
e = np.exp(z - z.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
Wq, Wk, Wv = (rng.normal(0, 0.4, (D, D)) for _ in range(3))
def contextualise(tokens):
X = np.array([E[idx.get(t, 1)] for t in tokens])
A = softmax((X @ Wq) @ (X @ Wk).T / np.sqrt(D))
return A, X + A @ (X @ Wv) # a residual, as in a real block
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
print("STEP 4 -- mix in the neighbours, and the vector becomes")
print("CONTEXTUAL. one attention layer is enough to show it:")
for s in (s1, s2):
A, out = contextualise(s)
b = s.index("bank")
print(" %r" % " ".join(s))
print(" 'bank' attends most to '%s' (%.4f)"
% (s[int(A[b].argmax())], A[b].max()))
print(" its output vector: %s" % np.round(out[b][:4], 3))
A1, o1 = contextualise(s1)
A2, o2 = contextualise(s2)
b1, b2 = s1.index("bank"), s2.index("bank")
print()
print(" the same word, two sentences, two different vectors:")
print(" distance between them: %.4f" % np.linalg.norm(o1[b1] - o2[b2]))
print(" distance before the attention layer: %.4f"
% np.linalg.norm(E[idx["bank"]] - E[idx["bank"]]))
print(" that second number is exactly 0 by construction. the first is not.")
print(" THAT is what 'contextual embedding' means, and it is why asking")
print(" for 'the embedding of bank' stopped being a well-posed question.")
print()
print(" (the projections here are random, so WHICH neighbour each token")
print(" attends to is luck. what is not luck is that the two vectors")
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
print(" differ at all -- that follows from the mechanism, and a trained")
print(" model is the same mechanism with weights that mean something.)")
print()
print("THE FOUR REPRESENTATIONS, and where each one lives:")
rows = [("index", "an integer", "the tokeniser's output"),
("one-hot", "%d numbers, %d zeros" % (V, V - 1), "conceptual only"),
("static embedding", "%d numbers per word" % D, "the embedding table"),
("contextual", "%d numbers per POSITION" % D, "every layer's output")]
print("%22s %26s %s" % ("representation", "size", "where it comes from"))
for a, b, c in rows:
print("%22s %26s %s" % (a, b, c))
print()
print("and two practical notes. the embedding table is often the largest")
print("single parameter block in a model, and it is frequently TIED to the")
print("output layer -- the same matrix used to look words up is transposed")
print("to score them, halving the count.")
print()
print("the special tokens at the top of the vocabulary are not decoration:")
for t in ("<pad>", "<unk>"):
print(" %-8s id %d -- %s"
% (t, idx[t],
"fills short sequences, and must be masked out of the loss"
if t == "<pad>" else "anything outside the vocabulary"))
print(" a model that is trained to predict <pad> will learn to predict it.")
Output
Try it yourself
Click "cat". Follow all four stages — note the one-hot row has a single lit cell at index 1. And the embedding is just 4 numbers.
Click "dog" as the second word. The likeness panel shows a high cosine score — the demo embeddings encode that cats and dogs are both animals.
Compare "cat" with "on". An animal versus a preposition: the likeness collapses. One-hot vectors could never express this gap — embeddings do it natively.
Worth recalling
Word form is a ladder: IDs are compact but lie about order, one-hot is honest but huge and likeness-blind, and dense embeddings are small, learned, and encode meaning as geometry. Modern NLP starts at the top of that ladder.
What the geometry holds
The arrangement that emerges from training is truly structured, in ways that can be measured.
Neighbourhoods are semantic. The nearest vectors to "January" are the other months. The nearest to "hospital" are clinic, doctor, patient.
Directions carry relations. The offset from "man" to "woman" is roughly the offset from "king" to "queen". This is what makes the famous analogy arithmetic work. It works cleanly for a small number of well-chosen relations and is often overstated.
Count affects size. Rare words often have longer vectors than common ones. This is one reason cosine likeness. Which ignores length. Is preferred to the dot product for comparing them.
Bias is encoded too. The vectors reproduce the numeric associations of the training corpus, including its prejudices. This is a documented and repeatedly reproduced finding. And it matters when embeddings feed decisions about people.
Single dimensions are not interpretable. Directions in the space often are, and probing classifiers can find them. There is a direction that separates singular from plural, another that tracks sentiment.
Practical notes
import torch.nn as nn
emb = nn.Embedding(num_embeddings=50_000, embedding_dim=768,
padding_idx=0) # keeps the pad vector at zero
x = emb(token_ids) # (batch, seq) -> (batch, seq, 768)
padding_idx matters: it keeps that row fixed at zero and excludes it from gradient updates. So padding adds nothing.
Three further points that come up in real work:
The embedding table is often the largest single parameter block in a small model. 50,000 × 768 is 38 million weights, which may exceed all the transformer layers in a compact model.
Freezing pretrained embeddings is worth trying with small datasets. There is far less to overfit. Fine-tune them when you have enough data or a specialised vocabulary.
Dimension is a real trade-off. 100–300 for word vectors, 768–4,096 in transformers. Larger captures more and costs memory and compute at each layer.
Questions people ask
Is the embedding matrix the same as a one-hot product? In math yes, computationally no. It is implemented as a row lookup.
Are embeddings trained or downloaded? Both are used. For text, downloading pretrained is almost always better. For categorical features in your own model, trained jointly.
What dimension should I use? 300 for standalone word vectors; any the pretrained model uses if not. For categorical features, roughly the fourth root of the cardinality as a starting point.
How are unseen words handled? Subword tokenisation splits them into known pieces. So there is always something to look up.
Can I inspect what a dimension means? Not usefully. Directions in the space are more interpretable than axes, and probing is the tool for finding them.
Why tie the input and output embeddings? It saves tens of millions of parameters and often helps slightly. Since both describe the same vocabulary.
Recap in one screen
A word becomes an integer id, which indexes a learned embedding matrix; the selected row is its vector.
The lookup is the same to a one-hot product and is implemented as indexing.
Embeddings are compact and place close words nearby, so learning transfers between related words.
Transformers keep a static table at the input and give context forms above it.
The geometry encodes semantics, relational directions, count effects and corpus bias.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Chain of Translations”?
What does this module say about “Token IDs and Their Trap”?
What does this module say about “One-Hot: Honest but Wasteful”?
Cheat sheet
How Words are Represented in Neural Networks
Trace the four-stage chain from a raw word to a token ID to a one-hot vector to the dense embedding a network can multiply.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.