Embeddings start as random noise. Training pulls co-occurring words together — press Train and watch clusters form in real time.
Overview
The distributional hypothesis
The whole field rests on one claim: a word is characterised by the company it keeps. Words appearing in similar contexts tend to mean similar things.
It is easy to check. Cat and dog both appear near pet, feed, vet and fur. Bulldozer does not. So if you build vectors that predict a word’s context well, words with similar contexts must end up with similar vectors — the semantic structure is a by-product of the prediction task, never an objective in itself.
Training Corpus
Words that appear in the same sentence are treated as context pairs — the engine of the distributional hypothesis.
Training
Training State
Epoch
0
Avg pair distance
-
Embedding Space During Training
RANDOM INIT
Each epoch: words that co-occur in a sentence attract; words that never co-occur repel slightly. No labels, no dictionary — structure emerges from raw co-occurrence alone.
How are Embeddings Generated?: A Practical Guide
Nobody hand-labels an embedding space. It falls out of a simple prediction task: guess a word from its neighbours, and the vectors that make you good at guessing turn out to encode meaning.
Skip-gram and CBOW
Word2Vec offers two ways to set up the prediction:
Skip-gram — given the centre word, predict the surrounding words. From “the cat sat on the mat”, given sat, predict the, cat, on, the. Slower, and better on rare words because each occurrence generates several training examples.
CBOW — given the surrounding words, predict the centre. Faster, and better on frequent words.
The architecture is deliberately trivial: an input embedding layer, no hidden layer, and an output projection. The embedding matrix is the only thing anybody wants; the output layer is discarded after training. The vectors are a side effect of a task nobody cares about the answers to.
Negative sampling
The naive setup predicts a probability distribution over the whole vocabulary, which means a softmax over 50,000 words for every training example. That is prohibitively expensive.
Negative sampling replaces it with a much cheaper question. Instead of “which of 50,000 words is the context?”, ask “is this pair a real (word, context) pair or a fabricated one?” — a binary classification. For each true pair, draw perhaps 5 to 20 random words as negatives and train the model to score the real pair high and the fakes low.
Cost drops from 50,000 output computations to about 20, and quality is comparable. This is what made training on billions of words practical.
Learned from context, not assigned
Nobody writes embedding vectors by hand. They are learned, and every method rests on the same observation: words that appear in similar contexts tend to mean similar things. Fill in "I poured myself a cup of ___" and the plausible answers — tea, coffee, water — are related precisely because they share contexts.
Turn that into a training objective and the vectors fall out.
Method
Objective
Output
Word2Vec skip-gram
Predict surrounding words from a centre word
One vector per word
Word2Vec CBOW
Predict the centre word from its context
One vector per word
GloVe
Factorise a co-occurrence count matrix
One vector per word
FastText
Skip-gram over character n-grams
Handles unseen words
BERT-style
Predict masked tokens in context
A different vector per occurrence
Sentence encoders
Bring related sentences together
One vector per sentence
Word2Vec, concretely
Skip-gram takes a centre word and tries to predict the words around it within a window. Training on "the quick brown fox jumps" with a window of 2 produces pairs: (brown, the), (brown, quick), (brown, fox), (brown, jumps).
The model is a shallow network: an embedding lookup, then a projection to vocabulary size. After training, the projection layer is discarded and the embedding table is the product. The prediction task was only ever a means to shape those vectors.
The practical problem is the output layer. A softmax over 100,000 words for every training pair is prohibitively expensive, so Word2Vec uses negative sampling: instead of scoring all words, score the true context word and a handful (5–20) of random ones, and train the model to tell them apart. That turns a 100,000-way softmax into a few binary classifications and is the trick that made the method practical.
CBOW is the mirror image — predict the centre word from the average of its context. It trains faster and does slightly worse on rare words, since averaging washes them out.
GloVe takes a different route entirely: build a matrix of how often each word co-occurs with each other word, then factorise it so that the dot product of two vectors approximates the logarithm of their co-occurrence count. Global statistics rather than local windows, with comparable results.
What the window size decides
This is the parameter that changes what "similar" means, and it is worth knowing because it is easy to get wrong.
A small window (2–5) captures syntactic similarity — words that are interchangeable in a sentence. "Run" ends up near "walk" and "jog", and near "runs" and "ran".
A large window (10–15) captures topical similarity — words that appear in the same subject matter. "Run" ends up near "race", "marathon" and "athlete".
Neither is correct in general. For a part-of-speech task, small. For document retrieval or topic work, large. Setting it without thinking about which kind of similarity the task needs is how embeddings end up subtly unfit for purpose.
Other parameters that matter: dimension (100–300 for word vectors), minimum count (drop words appearing fewer than five times), and subsampling of very frequent words, which stops "the" dominating every context.
Trained by prediction, not by design
Nobody writes an embedding table. It falls out of training a model on a prediction task, and this trains one with skip-gram negative sampling -- the word2vec objective -- so you can watch the geometry appear.
example_01.pyNumPy
import numpy as np
from collections import Counter
rng = np.random.default_rng(0)
corpus = (
"the cat drinks milk the dog drinks water the cat eats fish "
"the dog eats meat the cat chases the mouse the dog chases the cat "
"the king rules the land the queen rules the land the king wears a crown "
"the queen wears a crown the cat sleeps the dog sleeps a cat drinks water "
"a dog eats fish the king sits the queen sits the man rules the land "
"the woman rules the land the man sits the woman sits the man wears a crown"
).split()
vocab = sorted(set(corpus))
idx = {w: i for i, w in enumerate(vocab)}
V, D, WINDOW = len(vocab), 12, 2
print("%d tokens, %d distinct words, %d embedding dimensions."
% (len(corpus), V, D))
print()
pairs = []
for i, centre in enumerate(corpus):
for j in range(max(0, i - WINDOW), min(len(corpus), i + WINDOW + 1)):
if i != j:
pairs.append((idx[centre], idx[corpus[j]]))
pairs = np.array(pairs)
print("STEP 1 -- build the training task. for every word, its neighbours")
print("within %d positions become POSITIVE examples:" % WINDOW)
for centre, ctx in pairs[:5]:
print(" centre %-8s -> context %-8s (label 1)" % (vocab[centre], vocab[ctx]))
print(" ... %d pairs in total." % len(pairs))
print()
print(" the task is: given a centre word, is this other word a real")
print(" neighbour of it? that is a binary classification the corpus")
print(" answers for free -- no labels, no annotation.")
print()
freq = Counter(corpus)
noise = np.array([freq[w] for w in vocab], float) ** 0.75
noise /= noise.sum()
print("STEP 2 -- NEGATIVE SAMPLING. real neighbours alone are not enough:")
print(" a model could score everything 1 and be right every time.")
print(" so draw fake pairs from the word distribution, raised to 0.75:")
top = np.argsort(-noise)[:5]
print("%12s %12s %14s" % ("word", "raw count", "sampling prob"))
for i in top:
print("%12s %12d %14.4f" % (vocab[i], freq[vocab[i]], noise[i]))
print(" the 0.75 exponent flattens the distribution, so common words are")
print(" still drawn most often but do not swamp everything else.")
print()
W_in = rng.normal(0, 0.3, (V, D))
W_out = rng.normal(0, 0.3, (V, D))
K, LR = 5, 0.08
def sigmoid(z):
return 1 / (1 + np.exp(-np.clip(z, -30, 30)))
print("STEP 3 -- train. for each pair, push the centre and its real")
print("neighbour together, and push it away from %d random words:" % K)
order = rng.permutation(len(pairs))
losses = []
for epoch in range(30):
total = 0.0
for p in order:
c, ctx = pairs[p]
negs = rng.choice(V, K, p=noise)
targets = np.concatenate([[ctx], negs])
labels = np.concatenate([[1.0], np.zeros(K)])
v = W_in[c]
u = W_out[targets]
pred = sigmoid(u @ v)
err = pred - labels
total += -np.log(np.clip(np.where(labels > 0, pred, 1 - pred), 1e-9, 1)).sum()
W_out[targets] -= LR * np.outer(err, v)
W_in[c] -= LR * (err @ u)
losses.append(total / len(pairs))
if epoch in (0, 1, 4, 9, 19, 29):
print(" epoch %2d: average loss %.4f" % (epoch, losses[-1]))
print(" it falls steeply and then flattens, wobbling a little because the")
print(" negative samples are redrawn every epoch -- a later epoch can")
print(" simply draw a harder set, which is why the last number is not")
print(" the smallest. what matters is the drop from %.2f to about %.2f:"
% (losses[0], min(losses)))
print(" the model has learned to tell a real neighbour from a random word.")
print()
E = W_in / (np.linalg.norm(W_in, axis=1, keepdims=True) + 1e-9)
def sim(a, b):
return float(E[idx[a]] @ E[idx[b]])
print("STEP 4 -- and the embedding is a SIDE EFFECT. we wanted a neighbour")
print("classifier; what we keep is the input matrix:")
for a, b in (("cat", "dog"), ("king", "queen"), ("man", "woman"),
("cat", "king"), ("milk", "water"), ("cat", "rules")):
print(" %-6s vs %-6s %+.4f %s"
% (a, b, sim(a, b), "#" * max(0, int(18 * sim(a, b)))))
print()
same = np.mean([sim(a, b) for a, b in
(("cat", "dog"), ("king", "queen"), ("man", "woman"),
("milk", "water"))])
diff = np.mean([sim(a, b) for a, b in
(("cat", "king"), ("cat", "rules"), ("milk", "sits"),
("dog", "crown"))])
print(" average similarity, words used alike : %+.4f" % same)
print(" average similarity, words used apart : %+.4f" % diff)
print()
print("nothing in the loss mentioned meaning, categories or similarity. the")
print("model was only ever asked 'is this a real neighbour?', and words used")
print("in the same contexts ended up with the same answer -- so they ended")
print("up with the same vector.")
print()
print("that is how every embedding is generated, including the ones inside a")
print("transformer. the task differs -- next token, masked token, neighbour")
print("prediction -- but the shape is identical: train on a task built from")
print("unlabelled text, then keep an intermediate layer and throw the rest")
print("of the model away.")
print()
print("(this corpus is only %d words, and deliberately regular -- which is"
% len(corpus))
print("why the separation above is so clean. real text is far messier and")
print("needs billions of words to reach the same clarity. the mechanism is")
print("identical; only the amount of evidence differs.)")
Output
Guided experiments
Start from noise. Press Re-randomize and look at the layout. Vectors are random, so the arrangement is meaningless — no structure exists before training.
Train one epoch. Press Train 1 Epoch and watch points shift slightly. Each step nudges words that co-occur closer together and pushes unrelated ones apart.
Let clusters form. Press Train 20 Epochs. Related words gather into groups. Nothing told the model these words were related — it only ever tried to predict context.
Re-randomize and train again. The clusters re-form, but in different positions and orientations. The absolute coordinates are arbitrary; only the relative geometry carries meaning, which is why you cannot compare vectors from two separately trained models.
Common mistakes
Comparing vectors across models. Two training runs produce different, incompatible spaces. Vectors are only meaningful relative to others from the same run.
Too little data. These methods need tens of millions of tokens. On a small corpus the vectors are noise, and a pretrained set is almost always better.
Leaving frequent words unsubsampled.The and of co-occur with everything and carry almost no information. Word2Vec subsamples them aggressively, which improves both speed and quality.
Assuming a bigger window is better. Small windows (2–5) capture syntactic similarity; large windows (10+) capture topical relatedness. They are different notions of “similar” and the right one depends on the task.
What to remember
Embeddings are learned by training a deliberately simple model on a proxy task — predict a word from its context, or the context from the word — and keeping the embedding matrix while discarding everything else. Semantic structure emerges because words in similar contexts need similar vectors to make the prediction work. Negative sampling replaces the vocabulary-wide softmax with a cheap binary decision, which is what made the whole approach scale.
Contextual embeddings, and why they superseded fixed ones
Word2Vec gives "bank" one vector, averaging the river and the financial senses into something that is neither.
A transformer produces a different vector for each occurrence, computed from the surrounding words. In "the river bank" the vector has absorbed "river"; in "the savings bank" it has absorbed "savings". Polysemy is resolved by construction.
The training objective is what generates them:
Masked language modelling (BERT) hides 15% of tokens and predicts them from both directions.
Next-token prediction (GPT) predicts each token from those before it.
Either way, the hidden states of the trained model are the contextual embeddings, and no separate embedding-training step is needed.
For sentence-level vectors, taking BERT's [CLS] output directly works poorly — it was not trained to make similar sentences nearby. Sentence-BERT fixes this by fine-tuning with a contrastive objective on pairs of related and unrelated sentences, which is why purpose-built embedding models substantially outperform pooled hidden states from a general model.
Training your own
Two situations justify it, and neither is the common case.
An embedding layer inside your model. This is the everyday one: a high-cardinality categorical feature (product id, user id) replaced by a learned dense vector, trained jointly with the rest of the network. No separate pretraining, no corpus needed.
emb = nn.Embedding(num_embeddings=50_000, embedding_dim=64)
vecs = emb(item_ids) # learned along with everything else
Domain-specific text embeddings. When your vocabulary genuinely differs from web text — clinical notes, legal filings, internal product codes — fine-tuning an existing sentence encoder on your own pairs beats a general model. Contrastive fine-tuning needs pairs of related and unrelated text, which can often be mined from existing structure (a question and its accepted answer, a ticket and its resolution).
Training word vectors from scratch on a small corpus is almost always worse than using a pretrained model. Embeddings need a great deal of text.
Questions people ask
Do I need to train embeddings myself? Rarely for text — use a pretrained sentence encoder. Yes for categorical features in your own model.
Skip-gram or CBOW? Skip-gram for rare words and smaller corpora; CBOW when speed matters and the corpus is large.
What is negative sampling? Replacing a full softmax over the vocabulary with a few binary comparisons against random words. It is what made Word2Vec tractable.
Why do I get different vectors on each run? Random initialisation and sampling. The geometry is stable even though the coordinates are not, so similarities are reproducible and individual dimensions are not.
Can embeddings handle words not in the vocabulary? Word2Vec cannot. FastText can, by composing character n-grams. Subword tokenisers sidestep the problem entirely.
How much text do I need? Millions of words for usable word vectors from scratch. Almost none if you fine-tune a pretrained model.
Recap in one screen
Embeddings are learned from the principle that words in similar contexts have similar meanings.
Skip-gram predicts context from a word; CBOW does the reverse; GloVe factorises co-occurrence counts.
Negative sampling replaces an unaffordable softmax with a few binary comparisons.
Window size chooses between syntactic and topical similarity — pick it for the task.
Contextual embeddings from transformers give a different vector per occurrence, which resolves polysemy.
Recall check
0 of 2
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Masked language modelling” here?
(BERT) hides 15% of tokens and predicts them from both directions.
What is meant by “Next-token prediction” here?
(GPT) predicts each token from those before it.
Cheat sheet
How are Embeddings Generated?
The whole field rests on one claim: a word is characterised by the company it keeps. Words appearing in similar contexts tend to mean similar things.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.