Embeddings start as random noise. Training pulls co-occurring words together — press Train and watch clusters form in real time.
Training Corpus
Words that appear in the same sentence are treated as context pairs — the engine of the distributional hypothesis.
Training
Training State
Epoch
0
Avg pair distance
-
Embedding Space During Training
RANDOM INIT
Each epoch: words that co-occur in a sentence attract; words that never co-occur repel slightly. No labels, no dictionary — structure emerges from raw co-occurrence alone.
Overview
The distributional hypothesis
The whole field rests on one claim: a word is characterised by the company it keeps. Words appearing in close contexts tend to mean close things.
It is easy to check. Cat and dog both appear near pet, feed, vet and fur. Bulldozer does not. So if you build vectors that predict a word’s context well, words with close contexts must end up with close vectors. The semantic structure is a by-product of the prediction task, never an objective in itself. Word2Vec offers two designs: Skip-gram (predict context words from a centre word) and CBOW (predict the centre word from its context). Skip-gram works better on small datasets and rare words; CBOW is faster and works better on frequent words. Both give a weight matrix whose rows are the word embeddings. GloVe takes a different approach: it factorises the global word co-instance matrix. This captures the same distributional data but in a single matrix factorisation rather than iterative gradient updates. FastText extends Word2Vec by embedding character n-grams, so it can give vectors for words never seen in training by combining their subword pieces. All three give static embeddings (one vector per word no matter context). Context embeddings from transformers supersede them for most downstream tasks but are much more costly to compute. This page trains a small Skip-gram model on a sample corpus, shows the embedding vectors converging during training, and shows the vector-arithmetic property.
Nobody hand-labels an embedding space. It falls out of a simple prediction task: guess a word from its neighbours, and the vectors that make you good at guessing turn out to encode meaning.
The distributional hypothesis
The whole field rests on one claim: a word is characterised by the company it keeps. Words appearing in close contexts tend to mean close things.
It is easy to check. Cat and dog both appear near pet, feed, vet and fur. Bulldozer does not. So if you build vectors that predict a word’s context well, words with close contexts must end up with close vectors. The semantic structure is a by-product of the prediction task, never an objective in itself. Word2Vec offers two designs: Skip-gram (predict context words from a centre word) and CBOW (predict the centre word from its context). Skip-gram works better on small datasets and rare words; CBOW is faster and works better on frequent words. Both give a weight matrix whose rows are the word embeddings. GloVe takes a different approach: it factorises the global word co-instance matrix. This captures the same distributional data but in a single matrix factorisation rather than iterative gradient updates. FastText extends Word2Vec by embedding character n-grams, so it can give vectors for words never seen in training by combining their subword pieces. All three give static embeddings (one vector per word no matter context). Context embeddings from transformers supersede them for most downstream tasks but are much more costly to compute. This page trains a small Skip-gram model on a sample corpus, shows the embedding vectors converging during training, and shows the vector-arithmetic property.
Skip-gram and CBOW
Word2Vec offers two ways to set up the prediction:
Skip-gram — given the centre word, predict the nearby words. From “the cat sat on the mat”, given sat, predict the, cat, on, the. Slower, and better on rare words because each instance makes a few training cases.
CBOW — given the nearby words, predict the centre. Faster, and better on frequent words.
The design is on purpose trivial: an input embedding layer, no hidden layer. And an output projection. The embedding matrix is the only thing anybody wants; the output layer is discarded after training. The vectors are a side effect of a task nobody cares about the answers to.
Negative sampling
The naive setup predicts a chance distribution over the whole vocabulary. This means a softmax over 50,000 words for each training case. That is prohibitively costly.
Negative sampling replaces it with a much cheaper question. Instead of “which of 50,000 words is the context?”, ask “is this pair a real (word, context) pair or a fabricated one?” — a binary classification. For each true pair, draw perhaps 5 to 20 random words as negatives and train the model to score the real pair high and the fakes low.
Cost drops from 50,000 output steps to about 20, and quality is comparable. This is what made training on billions of words practical.
Learned from context, not assigned
Nobody writes embedding vectors by hand. They are learned, and each method rests on the same observation: words that appear in close contexts tend to mean close things. Fill in "I poured myself a cup of ___" and the plausible answers. Tea, coffee, water. Are related just because they share contexts.
Turn that into a training objective and the vectors fall out.
Method
Objective
Output
Word2Vec skip-gram
Predict surrounding words from a centre word
One vector per word
Word2Vec CBOW
Predict the centre word from its context
One vector per word
GloVe
Factorise a co-occurrence count matrix
One vector per word
FastText
Skip-gram over character n-grams
Handles unseen words
BERT-style
Predict masked tokens in context
A different vector per occurrence
Sentence encoders
Bring related sentences together
One vector per sentence
Word2Vec, concretely
Skip-gram takes a centre word and tries to predict the words around it within a window. Training on "the quick brown fox jumps" with a window of 2 gives pairs: (brown, the), (brown, quick), (brown, fox), (brown, jumps).
The model is a shallow network: an embedding lookup. Then a projection to vocabulary size. After training, the projection layer is discarded and the embedding table is the product. The prediction task was only ever a means to shape those vectors.
The practical problem is the output layer. A softmax over 100,000 words for each training pair is prohibitively costly. So Word2Vec uses negative sampling: instead of scoring all words, score the true context word and a handful (5–20) of random ones, and train the model to tell them apart. That turns a 100,000-way softmax into a few binary classifications and is the trick that made the method practical.
CBOW is the mirror image — predict the centre word from the average of its context. It trains faster and does slightly worse on rare words, since averaging washes them out.
GloVe takes a different route fully: build a matrix of how often each word co-occurs with each other word. Then factorise it so that the dot product of two vectors approximates the logarithm of their co-instance count. Global stats rather than local windows, with comparable results.
What the window size decides
This is the parameter that changes what "close" means. And it is worth knowing because it is easy to get wrong.
A small window (2–5) captures syntactic likeness — words that are alike in a sentence. "Run" ends up near "walk" and "jog", and near "runs" and "ran".
A large window (10–15) captures topical likeness — words that appear in the same subject matter. "Run" ends up near "race", "marathon" and "athlete".
Neither is correct broadly. For a part-of-speech task, small. For document retrieval or topic work, large. Setting it without thinking about which kind of likeness the task needs is how embeddings end up subtly unfit for purpose.
Other parameters that matter: dimension (100–300 for word vectors), minimum count (drop words appearing fewer than five times), and subsampling of very frequent words. This stops "the" ruling each context.
Trained by prediction, not by design
Nobody writes an embedding table. It falls out of training a model on a prediction task. And this trains one with skip-gram negative sampling -- the word2vec objective -- so you can watch the geometry appear.
example_01.pyNumPy
import numpy as np
from collections import Counter
rng = np.random.default_rng(0)
corpus = (
"the cat drinks milk the dog drinks water the cat eats fish "
"the dog eats meat the cat chases the mouse the dog chases the cat "
"the king rules the land the queen rules the land the king wears a crown "
"the queen wears a crown the cat sleeps the dog sleeps a cat drinks water "
"a dog eats fish the king sits the queen sits the man rules the land "
"the woman rules the land the man sits the woman sits the man wears a crown"
).split()
vocab = sorted(set(corpus))
idx = {w: i for i, w in enumerate(vocab)}
V, D, WINDOW = len(vocab), 12, 2
print("%d tokens, %d distinct words, %d embedding dimensions."
% (len(corpus), V, D))
print()
pairs = []
for i, centre in enumerate(corpus):
for j in range(max(0, i - WINDOW), min(len(corpus), i + WINDOW + 1)):
if i != j:
pairs.append((idx[centre], idx[corpus[j]]))
pairs = np.array(pairs)
print("STEP 1 -- build the training task. for every word, its neighbours")
print("within %d positions become POSITIVE examples:" % WINDOW)
for centre, ctx in pairs[:5]:
print(" centre %-8s -> context %-8s (label 1)" % (vocab[centre], vocab[ctx]))
print(" ... %d pairs in total." % len(pairs))
print()
print(" the task is: given a centre word, is this other word a real")
print(" neighbour of it? that is a binary classification the corpus")
print(" answers for free -- no labels, no annotation.")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
from collections import Counter
freq = Counter(corpus)
noise = np.array([freq[w] for w in vocab], float) ** 0.75
noise /= noise.sum()
print("STEP 2 -- NEGATIVE SAMPLING. real neighbours alone are not enough:")
print(" a model could score everything 1 and be right every time.")
print(" so draw fake pairs from the word distribution, raised to 0.75:")
top = np.argsort(-noise)[:5]
print("%12s %12s %14s" % ("word", "raw count", "sampling prob"))
for i in top:
print("%12s %12d %14.4f" % (vocab[i], freq[vocab[i]], noise[i]))
print(" the 0.75 exponent flattens the distribution, so common words are")
print(" still drawn most often but do not swamp everything else.")
print()
W_in = rng.normal(0, 0.3, (V, D))
W_out = rng.normal(0, 0.3, (V, D))
K, LR = 5, 0.08
def sigmoid(z):
return 1 / (1 + np.exp(-np.clip(z, -30, 30)))
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
from collections import Counter
print("STEP 3 -- train. for each pair, push the centre and its real")
print("neighbour together, and push it away from %d random words:" % K)
order = rng.permutation(len(pairs))
losses = []
for epoch in range(30):
total = 0.0
for p in order:
c, ctx = pairs[p]
negs = rng.choice(V, K, p=noise)
targets = np.concatenate([[ctx], negs])
labels = np.concatenate([[1.0], np.zeros(K)])
v = W_in[c]
u = W_out[targets]
pred = sigmoid(u @ v)
err = pred - labels
total += -np.log(np.clip(np.where(labels > 0, pred, 1 - pred), 1e-9, 1)).sum()
W_out[targets] -= LR * np.outer(err, v)
W_in[c] -= LR * (err @ u)
losses.append(total / len(pairs))
if epoch in (0, 1, 4, 9, 19, 29):
print(" epoch %2d: average loss %.4f" % (epoch, losses[-1]))
print(" it falls steeply and then flattens, wobbling a little because the")
print(" negative samples are redrawn every epoch -- a later epoch can")
print(" simply draw a harder set, which is why the last number is not")
print(" the smallest. what matters is the drop from %.2f to about %.2f:"
% (losses[0], min(losses)))
print(" the model has learned to tell a real neighbour from a random word.")
print()
E = W_in / (np.linalg.norm(W_in, axis=1, keepdims=True) + 1e-9)
def sim(a, b):
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
return float(E[idx[a]] @ E[idx[b]])
print("STEP 4 -- and the embedding is a SIDE EFFECT. we wanted a neighbour")
print("classifier; what we keep is the input matrix:")
for a, b in (("cat", "dog"), ("king", "queen"), ("man", "woman"),
("cat", "king"), ("milk", "water"), ("cat", "rules")):
print(" %-6s vs %-6s %+.4f %s"
% (a, b, sim(a, b), "#" * max(0, int(18 * sim(a, b)))))
print()
same = np.mean([sim(a, b) for a, b in
(("cat", "dog"), ("king", "queen"), ("man", "woman"),
("milk", "water"))])
diff = np.mean([sim(a, b) for a, b in
(("cat", "king"), ("cat", "rules"), ("milk", "sits"),
("dog", "crown"))])
print(" average similarity, words used alike : %+.4f" % same)
print(" average similarity, words used apart : %+.4f" % diff)
print()
print("nothing in the loss mentioned meaning, categories or similarity. the")
print("model was only ever asked 'is this a real neighbour?', and words used")
print("in the same contexts ended up with the same answer -- so they ended")
print("up with the same vector.")
print()
print("that is how every embedding is generated, including the ones inside a")
print("transformer. the task differs -- next token, masked token, neighbour")
print("prediction -- but the shape is identical: train on a task built from")
print("unlabelled text, then keep an intermediate layer and throw the rest")
print("of the model away.")
print()
print("(this corpus is only %d words, and deliberately regular -- which is"
% len(corpus))
print("why the separation above is so clean. real text is far messier and")
print("needs billions of words to reach the same clarity. the mechanism is")
print("identical; only the amount of evidence differs.)")
Output
Guided tests
Start from noise. Press Re-randomize and look at the layout. Vectors are random, so the arrangement is meaningless — no structure exists before training.
Train one epoch. Press Train 1 Epoch and watch points shift slightly. Each step nudges words that co-occur closer together and pushes unrelated ones apart.
Let clusters form. Press Train 20 Epochs. Related words gather into groups. Nothing told the model these words were related. It only ever tried to predict context.
Re-randomize and train again. The clusters re-form, but in different spots and orientations. The absolute coordinates are arbitrary; only the relative geometry carries meaning. This is why you cannot compare vectors from two apart trained models.
Common mistakes
Comparing vectors across models. Two training runs give different, incompatible spaces. Vectors are only meaningful next to others from the same run.
Too little data. These methods need tens of millions of tokens. On a small corpus the vectors are noise, and a pretrained set is almost always better.
Leaving frequent words unsubsampled.The and of co-occur with all of it and carry almost no data. Word2Vec subsamples them aggressively, which improves both speed and quality.
Assuming a bigger window is better. Small windows (2–5) capture syntactic likeness; large windows (10+) capture topical relatedness. They are different notions of “close” and the right one depends on the task.
What to recall
Embeddings are learned by training a on purpose simple model on a proxy task. Predict a word from its context. Or the context from the word. And keeping the embedding matrix while discarding all of it else. Semantic structure emerges because words in close contexts need close vectors to make the prediction work. Negative sampling replaces the vocabulary-wide softmax with a cheap binary decision. This is what made the whole approach scale.
Context embeddings, and why they superseded fixed ones
Word2Vec gives "bank" one vector, averaging the river and the financial senses into something that is neither.
A transformer gives a different vector for each instance, computed from the nearby words. In "the river bank" the vector has absorbed "river". In "the savings bank" it has absorbed "savings". Polysemy is resolved by construction.
The training objective is what makes them:
Masked language modelling (BERT) hides 15% of tokens and predicts them from both directions.
Next-token prediction (GPT) predicts each token from those before it.
Either way, the hidden states of the trained model are the context embeddings. And no separate embedding-training step is needed.
For sentence-level vectors, taking BERT's [CLS] output right works poorly. It was not trained to make close sentences nearby. Sentence-BERT fixes this by fine-tuning with a contrastive objective on pairs of related and unrelated sentences. This is why purpose-built embedding models much beat pooled hidden states from a general model.
Training your own
Two situations justify it, and neither is the common case.
An embedding layer inside your model. This is the everyday one. A high-cardinality categorical feature (product id, user id) replaced by a learned dense vector, trained jointly with the rest of the network. No separate pretraining, no corpus needed.
emb = nn.Embedding(num_embeddings=50_000, embedding_dim=64)
vecs = emb(item_ids) # learned along with everything else
Domain-set text embeddings. When your vocabulary truly differs from web text. Clinical notes, legal filings, internal product codes. Fine-tuning an existing sentence encoder on your own pairs beats a general model. Contrastive fine-tuning needs pairs of related and unrelated text, which can often be mined from existing structure (a question and its accepted answer, a ticket and its resolution).
Training word vectors from scratch on a small corpus is almost always worse than using a pretrained model. Embeddings need a great deal of text.
Questions people ask
Do I need to train embeddings myself? Rarely for text. Use a pretrained sentence encoder. Yes for categorical features in your own model.
Skip-gram or CBOW? Skip-gram for rare words and smaller corpora; CBOW when speed matters and the corpus is large.
What is negative sampling? Replacing a full softmax over the vocabulary with a few binary checks against random words. It is what made Word2Vec tractable.
Why do I get different vectors on each run? Random initialisation and sampling. The geometry is stable even though the coordinates are not. So similarities are reproducible and single dimensions are not.
Can embeddings handle words not in the vocabulary? Word2Vec cannot. FastText can, by composing character n-grams. Subword tokenisers sidestep the problem fully.
How much text do I need? Millions of words for usable word vectors from scratch. Almost none if you fine-tune a pretrained model.
Recap in one screen
Embeddings are learned from the principle that words in close contexts have close meanings.
Skip-gram predicts context from a word; CBOW does the reverse; GloVe factorises co-instance counts.
Negative sampling replaces an unaffordable softmax with a few binary checks.
Window size chooses between syntactic and topical likeness. Pick it for the task.
Context embeddings from transformers give a different vector per instance, which resolves polysemy.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The distributional hypothesis”?
What does this module say about “Negative sampling”?
What does this module say about “Learned from context, not assigned”?
Cheat sheet
How are Embeddings Generated?
Predict a word from its neighbours with Skip-gram or CBOW and watch the weight matrix converge into vectors where meaning is geometry.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.