Each cell is a learned feature. Similar words end up with similar numbers — that's what makes embeddings powerful.
From "cat" to a Vector: Word Representation in Neural Networks
The four stages every word passes through before a network can compute with it.
The Chain of Translations
A neural network is a pile of multiplications and additions — it can only consume numbers. So every word goes through a chain of translations: word → token ID → one-hot vector → dense embedding. Each stage exists to fix a shortcoming of the previous one.
Token IDs and Their Trap
The dictionary lookup ("cat" → 1, "dog" → 3) is compact, but the raw integers smuggle in a false claim: that "dog" (3) is somehow three times "cat" (1), or that words with adjacent IDs are related. The IDs are arbitrary labels, and arithmetic on labels is meaningless — a network fed raw IDs will happily learn those fake relationships.
One-Hot: Honest but Wasteful
One-hot encoding fixes the fake-ordering problem: each word becomes a vector of zeros with a single 1 at its index. Now no word is numerically "bigger" than another. The costs:
Size: the vector is as long as the vocabulary — 50,000 dimensions for a modest one.
No similarity: every pair of one-hot vectors is exactly the same distance apart. "cat" is as far from "kitten" as from "carburetor".
Dense Embeddings: Small and Meaningful
An embedding layer maps each token ID to a short vector of learned real numbers (4 dims in the demo; 300–4096 in practice). Because these values are trained rather than assigned, words used in similar contexts drift toward similar vectors — similarity becomes measurable with a dot product. This single idea underpins everything from word2vec to the input layer of GPT.
Interactive Exploration Guide
Click "cat". Follow all four stages — note the one-hot row has a single lit cell at index 1, and the embedding is just 4 numbers.
Click "dog" as the second word. The similarity panel shows a high cosine score — the demo embeddings encode that cats and dogs are both animals.
Compare "cat" with "on". An animal versus a preposition: the similarity collapses. One-hot vectors could never express this difference — embeddings do it natively.
Key Takeaway
Word representation is a ladder: IDs are compact but lie about order, one-hot is honest but huge and similarity-blind, and dense embeddings are small, learned, and encode meaning as geometry. Modern NLP starts at the top of that ladder.
Cheat sheet
How Words are Represented in Neural Networks
A neural network is a pile of multiplications and additions — it can only consume numbers. So every word goes through a chain of translations: word → token ID → one-hot vector → dense embedding. Each stage exists to fix a shortcoming of the previous one.