Embeddings place words in a space where distance means meaning. Click around the map — similar words live in the same neighbourhood.
Overview
What one-hot encoding cannot express
The obvious way to feed a word to a network is a one-hot vector: one dimension per vocabulary word, all zeros except a single 1. With a 50,000-word vocabulary each word is a 50,000-dimensional vector.
Two problems. It is enormous and almost entirely zeros. And — the real issue — every pair of words is exactly equidistant. The vectors for cat and kitten are precisely as far apart as cat and bulldozer, because any two distinct one-hot vectors differ in exactly two positions. The representation contains no information about meaning whatsoever, so nothing learned about one word transfers to a related one.
How to Explore
01Click any word on the map to select it.
02Click a second word — a line appears with their cosine similarity.
03Run the analogy to see vector arithmetic in action.
Vector Arithmetic
king − man + woman ≈ ?
Similarity Readout
Select two words on the map.
The Embedding Space (2D projection)
20 WORDS
Real embeddings have hundreds of dimensions; this map is a 2D projection. The clusters — royalty, animals, fruit, vehicles, emotions — emerge purely from how words are used in text.
What are Embeddings?: A Practical Guide
An embedding maps each word to a dense vector positioned so that similar words sit close together. That geometry is what lets a model generalise from words it has seen to words it has not.
A dense vector with learned geometry
An embedding replaces that with a short dense vector — typically 100 to 300 dimensions — learned during training. Now cat and kitten can be near each other while bulldozer is far away, and “near” is measured by cosine similarity between the vectors.
The consequence is generalisation. If the model has learned something about sentences containing cat, and kitten sits nearby in the space, that knowledge partially transfers without kitten ever appearing in the same context. One-hot vectors make this structurally impossible.
The famous analogy, and what it actually shows
Trained embeddings encode relationships as consistent directions, which is why vector arithmetic works:
king − man + woman ≈ queen
Subtracting man from king isolates a direction that roughly means royalty-without-gender; adding woman moves along the gender axis, landing near queen. The same holds for Paris − France + Italy ≈ Rome and for grammatical relations such as singular to plural.
Worth being precise: the result is not exactly queen, it is a point whose nearest neighbour is queen, and the effect is weaker on rare words. But it demonstrates the real claim — that these dimensions carry semantic structure learned entirely from co-occurrence, with no one ever labelling a “gender axis”.
Numbers that carry meaning
A one-hot vector says only "this is word 4,712". Every word is equally distant from every other, so the representation contains no information about which words are related.
An embedding replaces that with a dense vector of a few hundred numbers, learned from data, positioned so that similar words end up near each other.
Representation
"king"
Similar to "queen"?
One-hot
[0, 0, …, 1, …, 0] — 50,000 long
No — equidistant from everything
Embedding
[0.21, −0.44, 0.83, …] — 300 long
Yes — nearby in the space
The famous demonstration is vector arithmetic: king − man + woman lands close to queen. It works because the training objective forces words that appear in similar contexts into similar positions, and the difference between "king" and "man" ends up encoding something like royalty as a direction in the space.
The demonstration is often overstated — it works cleanly for a small number of well-chosen analogies — but the underlying property is real and it is what makes embeddings useful.
Where they come from
Word2Vec and GloVe learn one fixed vector per word from co-occurrence statistics. Cheap, useful, and limited: "bank" gets a single vector averaging the river and the financial senses.
Contextual embeddings from BERT-style models give a different vector for each occurrence, computed from the surrounding words. "River bank" and "savings bank" now differ, which resolves the ambiguity that fixed embeddings cannot.
Sentence embeddings (Sentence-BERT, E5, OpenAI's embedding models) represent whole passages as one vector, tuned so that similar meanings land near each other. These are what retrieval systems use.
Learned task embeddings are the everyday case in applied deep learning: an embedding layer trained jointly with your model, replacing a high-cardinality categorical column with a compact dense vector.
Comparing them: cosine similarity
The standard measure is the angle between two vectors, ignoring their lengths:
similarity = (a · b) / (‖a‖ ‖b‖)
Length in an embedding often reflects word frequency or confidence rather than meaning, so ignoring it is the point. On vectors normalised to length 1, cosine similarity and the dot product are the same number — which is why libraries normalise on the way in and then use fast dot products.
Calibrate the threshold rather than assuming. Many sentence-embedding models score unrelated text at 0.2–0.4, so a threshold of 0.5 admits a great deal of noise. Score a few hundred pairs you know to be related and a few hundred you know are not, and put the threshold where the two distributions separate.
From an index to a vector, and what that buys
A one-hot vector says every word is equally unrelated to every other. An embedding does not -- and this trains one from co-occurrence counts so you can watch the relationships appear.
example_01.pyNumPy
import numpy as np
from collections import Counter
rng = np.random.default_rng(0)
corpus = (
"the cat drinks milk the dog drinks water the cat eats fish "
"the dog eats meat the king rules the land the queen rules the land "
"the cat sleeps the dog sleeps the king sits the queen sits "
"a cat drinks water a dog eats fish the king eats meat the queen drinks milk "
"the man rules the land the woman rules the land the man sits the woman sits"
).split()
vocab = sorted(set(corpus))
idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
print("%d tokens, %d distinct words." % (len(corpus), V))
print()
print("ONE-HOT first, so the problem is concrete:")
for w in ("cat", "dog", "milk"):
v = np.zeros(V, int); v[idx[w]] = 1
print(" %-6s -> %s" % (w, v))
print()
print("every pair of one-hot vectors has dot product exactly 0:")
for a, b in (("cat", "dog"), ("cat", "milk"), ("king", "queen")):
va, vb = np.zeros(V), np.zeros(V)
va[idx[a]] = vb[idx[b]] = 1
print(" %-6s . %-6s = %.1f" % (a, b, va @ vb))
print(" cat and dog are exactly as unrelated as cat and milk. the")
print(" representation contains no information at all beyond identity.")
print(" and it is %d numbers per word to say that." % V)
print()
WINDOW = 2
C = np.zeros((V, V))
for i, w in enumerate(corpus):
for j in range(max(0, i - WINDOW), min(len(corpus), i + WINDOW + 1)):
if i != j:
C[idx[w], idx[corpus[j]]] += 1
print("CO-OCCURRENCE. count what appears within %d words of what:" % WINDOW)
show = ["cat", "dog", "king", "queen", "milk"]
print("%8s %s" % ("", "".join("%8s" % w for w in show)))
for a in show:
print("%8s %s" % (a, "".join("%8.0f" % C[idx[a], idx[b]] for b in show)))
print(" 'cat' and 'dog' never appear next to each other -- but they appear")
print(" next to the SAME words, and that is the signal.")
print()
logC = np.log1p(C)
U, S, Vt = np.linalg.svd(logC - logC.mean(0), full_matrices=False)
DIM = 5
Emb = U[:, :DIM] * S[:DIM]
Emb = Emb / (np.linalg.norm(Emb, axis=1, keepdims=True) + 1e-9)
print("factorise that matrix and keep %d dimensions. that is all an" % DIM)
print("embedding is -- a compressed co-occurrence table:")
print(" %d x %d counts -> %d x %d vectors (%.1fx smaller)"
% (V, V, V, DIM, V / DIM))
print()
def sim(a, b):
return Emb[idx[a]] @ Emb[idx[b]]
print("now ask which words ended up near which:")
pairs = [("cat", "dog"), ("king", "queen"), ("man", "woman"),
("cat", "king"), ("milk", "water"), ("cat", "rules")]
for a, b in pairs:
print(" %-6s vs %-6s : %+.4f %s"
% (a, b, sim(a, b), "#" * max(0, int(20 * sim(a, b)))))
print()
print("nothing told it that cats and dogs are both animals. it counted")
print("neighbours, and words with similar neighbours came out close.")
print("that is the distributional hypothesis doing all the work: a word is")
print("characterised by the company it keeps.")
print()
print("nearest neighbours, which is the query embeddings are really for:")
for w in ("cat", "king", "drinks"):
sims = [(sim(w, o), o) for o in vocab if o != w]
top = sorted(sims, reverse=True)[:3]
print(" %-8s -> %s" % (w, ", ".join("%s (%+.3f)" % (o, s) for s, o in top)))
print()
print("and the property that made these famous -- arithmetic on meaning:")
def nearest(vec, exclude):
best = None
for w in vocab:
if w in exclude:
continue
s = Emb[idx[w]] @ vec / (np.linalg.norm(vec) + 1e-9)
if best is None or s > best[0]:
best = (s, w)
return best
target = Emb[idx["king"]] - Emb[idx["man"]] + Emb[idx["woman"]]
s, w = nearest(target, {"king", "man", "woman"})
print(" king - man + woman = %s (%+.4f)" % (w, s))
print(" on a corpus this small that is as much luck as linguistics --")
print(" the real result needs billions of words. the mechanism is the")
print(" point: directions in the space carry meaning, so you can do")
print(" arithmetic with them.")
print()
print("the cost side, briefly:")
for v, d in ((50_000, 300), (50_000, 768), (200_000, 4096)):
print(" vocab %7d x %4d dims = %11s parameters (%5.1f MB at float32)"
% (v, d, "{:,}".format(v * d), v * d * 4 / 1024 / 1024))
print(" in a large language model this table is often the single biggest")
print(" layer, and it is frequently tied to the output layer to halve it.")
Output
Try it yourself
Run the analogy. Press Run the Famous Analogy and follow the vector arithmetic. The result is a point in space, and the answer is whichever word happens to lie nearest to it.
Look at what is near what. Note which words cluster. Related terms group together not because anyone grouped them, but because they appeared in similar contexts during training.
Watch the direction, not the position. The vector from man to woman points much the same way as the vector from king to queen. It is that parallelism, repeated across many pairs, that makes the arithmetic work at all.
Static and contextual
Word2Vec, GloVe and FastText produce static embeddings: one fixed vector per word, forever. That breaks on polysemy — bank gets a single vector that averages the riverside and the financial senses, serving neither.
BERT and every modern language model produce contextual embeddings instead: the vector for a word is computed from the sentence it appears in, so bank in “river bank” and “bank account” gets genuinely different representations. This is the single biggest improvement in word representation since embeddings were introduced, and it is why static embeddings are now mostly of historical and pedagogical interest.
Common mistakes
Using cosine similarity on unnormalised vectors and calling it distance. Cosine measures angle and ignores magnitude, which is usually what you want — but it is not Euclidean distance and the two rank neighbours differently.
Expecting a static embedding to handle ambiguity. One vector per word cannot represent two senses.
Ignoring inherited bias. Embeddings learn the statistical associations of their training corpus, including the prejudiced ones, and those propagate into anything built on top.
Over-large embedding dimensions on small vocabularies. 300 dimensions for 500 words is mostly free parameters to overfit with.
The short version
An embedding replaces a sparse, meaningless one-hot vector with a short dense one whose position encodes meaning, so similar words end up nearby and knowledge transfers between them. Relationships appear as consistent directions, which is what makes vector analogies work. Static embeddings give each word one vector and cannot handle ambiguity; contextual embeddings compute the vector from the surrounding sentence, and that is what modern models use.
What they are used for
Semantic search and RAG. Embed the question and every stored passage; retrieve the passages with the highest cosine similarity. Every vector database is a machine for doing this quickly.
Recommendations. Embed users and items in the same space and recommend the nearest items.
Classification with little data. Embed the text and train a simple classifier on the vectors. Often beats fine-tuning when you have a few hundred labels.
Clustering and topic discovery. Embed, then run k-means or HDBSCAN in the embedding space.
Deduplication. Support tickets or product listings above about 0.9 similarity are usually the same thing said twice.
High-cardinality categorical features. An embedding layer instead of 10,000 one-hot columns.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
vecs = model.encode(["a cat on a mat", "a feline on a rug"],
normalize_embeddings=True)
float(vecs[0] @ vecs[1]) # ~0.7 - close in meaning, no words shared
What they get wrong
They inherit the training data's biases. Word2Vec's doctor − man + woman famously lands near nurse. The vectors encode statistical regularities of the corpus, including its prejudices, and using them in hiring or lending pipelines propagates that.
Similarity is not truth. "The drug is effective" and "The drug is not effective" are close in most embedding spaces, because they share nearly every word. Negation, numbers and named entities are all weak spots. Retrieval systems built on embeddings alone will confidently return the opposite of what was asked, which is why hybrid retrieval — embeddings plus keyword search — is standard in production.
Domain mismatch. A model trained on web text has thin coverage of medical or legal vocabulary. Fine-tuning on domain text, or choosing a domain-specific model, matters more than the general benchmark ranking.
They are model-specific. Vectors from two different models are not comparable, so changing model means re-embedding your whole corpus.
Questions people ask
What dimension should I use? 384 to 1,536 for text. Larger costs more memory and search time for diminishing gains; for categorical features, a rough starting point is the fourth root of the cardinality.
Word or sentence embeddings? Sentence embeddings for search, similarity and classification. Word embeddings for token-level tasks or as a cheap baseline.
Do I need a vector database? Below about 100,000 vectors, a NumPy matrix and a dot product is fine. Above that, an approximate index (FAISS, HNSW) pays for itself.
Can I fine-tune embeddings? Yes — contrastive training on your own pairs of related and unrelated text, which typically beats a general-purpose model on a specific domain.
Why do all my similarities look high? Some models pack everything into a narrow cone of the space. Relative ranking still works; absolute thresholds need calibrating.
Are embeddings interpretable? Individual dimensions, no. Directions in the space often are — and probing classifiers can find them.
Recap in one screen
An embedding is a dense learned vector positioned so that similar items sit near each other.
Fixed embeddings give one vector per word; contextual ones give a different vector per occurrence.
Compare with cosine similarity, and calibrate the threshold against known pairs.
They power search, RAG, recommendations, clustering and high-cardinality features.
They inherit corpus bias, handle negation and numbers poorly, and are not comparable across models.
Check yourself
0 of 3
Answer without scrolling back up.
What is an embedding?
Embeddings place words in a space where distance means something. Nothing about the meaning is written down - it is inferred entirely from which contexts a word turns up in.
Why are embeddings better than one-hot vectors for words?
Under one-hot, 'cat' is exactly as far from 'dog' as from 'parliament'. Embeddings can put related words near each other, which is the whole point.
Embeddings trained on ordinary web text reliably reproduce:
The vectors encode how words are actually used, including every stereotype in the corpus. This is measurable, well documented, and a real problem in deployed systems.
Cheat sheet
What are Embeddings?
The obvious way to feed a word to a network is a one-hot vector: one dimension per vocabulary word, all zeros except a single 1. With a 50,000-word vocabulary each word is a 50,000-dimensional vector.
GloVe: Global Vectors for Word RepresentationPennington, Socher & Manning, EMNLP 2014
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.