Embeddings start as random noise. Training pulls co-occurring words together — press Train and watch clusters form in real time.
Training Corpus
Words that appear in the same sentence are treated as context pairs — the engine of the distributional hypothesis.
Training
Training State
Epoch
0
Avg pair distance
-
Embedding Space During Training
RANDOM INIT
Each epoch: words that co-occur in a sentence attract; words that never co-occur repel slightly. No labels, no dictionary — structure emerges from raw co-occurrence alone.
How Embeddings are Learned: From Noise to Meaning
The distributional hypothesis, context prediction, and why random vectors organize themselves.
The Distributional Hypothesis
"You shall know a word by the company it keeps" (J.R. Firth, 1957). Words that appear in similar contexts tend to mean similar things — "cat" and "dog" both follow "the", precede "chased", appear near "pet". Embedding training is simply this idea turned into an optimization: make words with shared contexts have similar vectors.
The Training Loop
Methods like word2vec's skip-gram never see a definition of any word. The loop is:
Initialize every word's vector to random noise.
Slide a window through the corpus, collecting (center word, context word) pairs.
Pull together: nudge the vectors of each observed pair slightly closer (in the real algorithm: increase the probability the model assigns to the pair).
Push apart: nudge random non-co-occurring pairs slightly further (negative sampling).
Repeat for millions of pairs — clusters, directions, and analogies emerge as a side effect.
Interactive Exploration Guide
Look at the initial map. Pure noise — "king" might sit next to "banana". This is genuinely how every embedding matrix begins life.
Click "Train 1 Epoch" a few times. Each click applies one attract/repel pass. Watch the average pair distance fall as related words drift together.
Run 20 epochs animated. The animals gather, royalty huddles, fruit separates — and the badge flips to CLUSTERS FORMED. Then hit "Re-randomize" and watch a different random start converge to the same neighbourhood structure.
Key Takeaway
Nobody writes down that "cat" is like "dog" — the geometry is learned from co-occurrence statistics alone. Random vectors plus a pull-together/push-apart rule over enough text yields a space where distance encodes meaning. Scale the same principle up and you get the embedding layers inside every modern language model.
Cheat sheet
How are Embeddings Generated?
"You shall know a word by the company it keeps" (J.R. Firth, 1957). Words that appear in similar contexts tend to mean similar things — "cat" and "dog" both follow "the", precede "chased", appear near "pet". Embedding training is simply this idea turned into an optimization: make words with shared contexts have similar vectors.