Embeddings place words in a space where distance means meaning. Click around the map — similar words live in the same neighbourhood.
How to Explore
01Click any word on the map to select it.
02Click a second word — a line appears with their cosine similarity.
03Run the analogy to see vector arithmetic in action.
Vector Arithmetic
king − man + woman ≈ ?
Similarity Readout
Select two words on the map.
The Embedding Space (2D projection)
20 WORDS
Real embeddings have hundreds of dimensions; this map is a 2D projection. The clusters — royalty, animals, fruit, vehicles, emotions — emerge purely from how words are used in text.
Overview
What one-hot encoding cannot express
The obvious way to feed a word to a network is a one-hot vector. One dimension per vocabulary word, all zeros except a single 1. With a 50,000-word vocabulary each word is a 50,000-dimensional vector.
Two problems. It is enormous and almost fully zeros. And — the real issue — each pair of words is just equidistant. The vectors for cat and kitten are just as far apart as cat and bulldozer,. This is since any two distinct one-hot vectors differ in just two spots. The form holds no data about meaning whatsoever, so nothing learned about one word transfers to a tied one. An embedding fixes both problems by replacing the sparse one-hot vector with a short, dense vector. Much 100 to 300 dimensions. Whose vals are learned in training. In the coming space, words with close meanings sit close together: cat and kitten are close while bulldozer is far away. The geometry encodes semantic links as steady ways, which is why vector arithmetic like king - man + woman ≈ queen works without being by name engineered. Word2Vec, GloVe, and FastText give static embeddings (one vector per word no matter context). Transformer models like BERT give context embeddings that give the same word else vectors in else sentences. Cosine likeness between two embedding vectors measures how tied their meanings are. This is the foundation of search, recommendation, and retrieval. This page shows a small embedding space, shows the arithmetic property, and covers the gap between static and context embeddings.
Embeddings place words in a space where distance means meaning.
An embedding maps each word to a dense vector positioned so that similar words sit close together. That geometry is what lets a model generalise from words it has seen to words it has not.
What one-hot encoding cannot express
The obvious way to feed a word to a network is a one-hot vector. One dimension per vocabulary word, all zeros except a single 1. With a 50,000-word vocabulary each word is a 50,000-dimensional vector.
Two problems. It is enormous and almost fully zeros. And — the real issue — each pair of words is just equidistant. The vectors for cat and kitten are just as far apart as cat and bulldozer,. This is since any two distinct one-hot vectors differ in just two spots. The form holds no data about meaning whatsoever, so nothing learned about one word transfers to a tied one. An embedding fixes both problems by replacing the sparse one-hot vector with a short, dense vector. Much 100 to 300 dimensions. Whose vals are learned in training. In the coming space, words with close meanings sit close together: cat and kitten are close while bulldozer is far away. The geometry encodes semantic links as steady ways, which is why vector arithmetic like king - man + woman ≈ queen works without being by name engineered. Word2Vec, GloVe, and FastText give static embeddings (one vector per word no matter context). Transformer models like BERT give context embeddings that give the same word else vectors in else sentences. Cosine likeness between two embedding vectors measures how tied their meanings are. This is the foundation of search, recommendation, and retrieval. This page shows a small embedding space, shows the arithmetic property, and covers the gap between static and context embeddings.
A dense vector with learned geometry
An embedding replaces that with a short dense vector — much 100 to 300 dimensions — learned in training. Now cat and kitten can be near each else while bulldozer is far away, and “near” is measured by cosine likeness between the vectors.
The result is extension. If the model has learned a thing about sentences holding cat, and kitten sits close in the space, that knowledge partially transfers without kitten ever appearing in the same context. One-hot vectors make this structurally not doable. The geometry of the space encodes meaning: vector arithmetic like king - man + woman ≈ queen falls out of the training without being engineered,. This is since the ways in the space align with semantic links. Cosine likeness between two embedding vectors measures how tied their meanings are. This is why embeddings are the foundation of search, recommendation, and retrieval systems. Word2Vec, GloVe, and FastText give static embeddings (one vector per word no matter context). Transformer models like BERT give context embeddings (a else vector for the same word in else sentences). This is why "bank" near "river" and "bank" near "money" get else forms. This page shows a small embedding space, shows the arithmetic property, and covers the gap between static and context embeddings.
The famous analogy, and what it really shows
Trained embeddings encode links as steady ways, which is why vector arithmetic works:
king − man + woman ≈ queen
Subtracting man from king isolates a way that roughly means royalty-without-gender; adding woman moves with the gender axis, landing near queen. The same holds for Paris − France + Italy ≈ Rome and for grammatical relations such as singular to plural.
Worth being precise: the result is not just queen, it is a point whose nearest neighbour is queen. And the effect is weaker on rare words. But it shows the real claim — that these dimensions carry semantic structure learned fully from co-instance, with no one ever labelling a “gender axis”.
Counts that carry meaning
A one-hot vector says just "this is word 4,712". Each word is equally distant from each else, so the form holds no data about which words are tied.
An embedding replaces that with a dense vector of a few hundred counts, learned from data, positioned. That way, close words end up near each else.
Representation
"king"
Similar to "queen"?
One-hot
[0, 0, …, 1, …, 0] — 50,000 long
No — equidistant from everything
Embedding
[0.21, −0.44, 0.83, …] — 300 long
Yes — nearby in the space
The famous case is vector arithmetic: king − man + woman lands close to queen. It works since the training objective forces words that appear in close contexts into close spots. And the gap between "king" and "man" ends up encoding a thing like royalty as a way in the space.
The case is much overstated. It works cleanly for a small count of well-chosen analogies. But the underlying property is real and it is what makes embeddings useful.
Where they come from
Word2Vec and GloVe learn one fixed vector per word from co-instance stats. Cheap, useful, and limited: "bank" gets a single vector averaging the river and the financial senses.
Context embeddings from BERT-style models give a else vector for each instance, computed from the close words. "River bank" and "savings bank" now differ, which resolves the ambiguity that fixed embeddings cannot.
Sentence embeddings (Sentence-BERT, E5, OpenAI's embedding models) represent whole passages as one vector, tuned. That way, close meanings land near each else. These are what retrieval systems use.
Learned task embeddings are the everyday case in applied deep learning. An embedding layer trained jointly with your model. This replaces a high-cardinality categorical column with a compact dense vector.
Comparing them: cosine likeness
The standard measure is the angle between two vectors, ignoring their lengths:
likeness = (a · b) / (‖a‖ ‖b‖)
Length in an embedding much reflects word count or confidence rather than meaning. So ignoring it is the point. On vectors normalised to length 1, cosine likeness and the dot product are the same count —. This is why libraries normalise on the way in. Then use fast dot products.
Calibrate the threshold rather than assuming. Many sentence-embedding models score unrelated text at 0.2–0.4, so a threshold of 0.5 admits a great deal of noise. Score a few hundred pairs you know to be tied and a few hundred you know are not, and put the threshold where the two distributions separate.
From an index to a vector, and what that buys
A one-hot vector says each word is equally unrelated to each else. An embedding does not -- and this trains one from co-instance counts so you can watch the links appear.
example_01.pyNumPy
import numpy as np
from collections import Counter
rng = np.random.default_rng(0)
corpus = (
"the cat drinks milk the dog drinks water the cat eats fish "
"the dog eats meat the king rules the land the queen rules the land "
"the cat sleeps the dog sleeps the king sits the queen sits "
"a cat drinks water a dog eats fish the king eats meat the queen drinks milk "
"the man rules the land the woman rules the land the man sits the woman sits"
).split()
vocab = sorted(set(corpus))
idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
print("%d tokens, %d distinct words." % (len(corpus), V))
print()
print("ONE-HOT first, so the problem is concrete:")
for w in ("cat", "dog", "milk"):
v = np.zeros(V, int); v[idx[w]] = 1
print(" %-6s -> %s" % (w, v))
print()
print("every pair of one-hot vectors has dot product exactly 0:")
for a, b in (("cat", "dog"), ("cat", "milk"), ("king", "queen")):
va, vb = np.zeros(V), np.zeros(V)
va[idx[a]] = vb[idx[b]] = 1
print(" %-6s . %-6s = %.1f" % (a, b, va @ vb))
print(" cat and dog are exactly as unrelated as cat and milk. the")
print(" representation contains no information at all beyond identity.")
print(" and it is %d numbers per word to say that." % V)
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
from collections import Counter
WINDOW = 2
C = np.zeros((V, V))
for i, w in enumerate(corpus):
for j in range(max(0, i - WINDOW), min(len(corpus), i + WINDOW + 1)):
if i != j:
C[idx[w], idx[corpus[j]]] += 1
print("CO-OCCURRENCE. count what appears within %d words of what:" % WINDOW)
show = ["cat", "dog", "king", "queen", "milk"]
print("%8s %s" % ("", "".join("%8s" % w for w in show)))
for a in show:
print("%8s %s" % (a, "".join("%8.0f" % C[idx[a], idx[b]] for b in show)))
print(" 'cat' and 'dog' never appear next to each other -- but they appear")
print(" next to the SAME words, and that is the signal.")
print()
logC = np.log1p(C)
U, S, Vt = np.linalg.svd(logC - logC.mean(0), full_matrices=False)
DIM = 5
Emb = U[:, :DIM] * S[:DIM]
Emb = Emb / (np.linalg.norm(Emb, axis=1, keepdims=True) + 1e-9)
print("factorise that matrix and keep %d dimensions. that is all an" % DIM)
print("embedding is -- a compressed co-occurrence table:")
print(" %d x %d counts -> %d x %d vectors (%.1fx smaller)"
% (V, V, V, DIM, V / DIM))
print()
def sim(a, b):
return Emb[idx[a]] @ Emb[idx[b]]
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
from collections import Counter
print("now ask which words ended up near which:")
pairs = [("cat", "dog"), ("king", "queen"), ("man", "woman"),
("cat", "king"), ("milk", "water"), ("cat", "rules")]
for a, b in pairs:
print(" %-6s vs %-6s : %+.4f %s"
% (a, b, sim(a, b), "#" * max(0, int(20 * sim(a, b)))))
print()
print("nothing told it that cats and dogs are both animals. it counted")
print("neighbours, and words with similar neighbours came out close.")
print("that is the distributional hypothesis doing all the work: a word is")
print("characterised by the company it keeps.")
print()
print("nearest neighbours, which is the query embeddings are really for:")
for w in ("cat", "king", "drinks"):
sims = [(sim(w, o), o) for o in vocab if o != w]
top = sorted(sims, reverse=True)[:3]
print(" %-8s -> %s" % (w, ", ".join("%s (%+.3f)" % (o, s) for s, o in top)))
print()
print("and the property that made these famous -- arithmetic on meaning:")
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
def nearest(vec, exclude):
best = None
for w in vocab:
if w in exclude:
continue
s = Emb[idx[w]] @ vec / (np.linalg.norm(vec) + 1e-9)
if best is None or s > best[0]:
best = (s, w)
return best
target = Emb[idx["king"]] - Emb[idx["man"]] + Emb[idx["woman"]]
s, w = nearest(target, {"king", "man", "woman"})
print(" king - man + woman = %s (%+.4f)" % (w, s))
print(" on a corpus this small that is as much luck as linguistics --")
print(" the real result needs billions of words. the mechanism is the")
print(" point: directions in the space carry meaning, so you can do")
print(" arithmetic with them.")
print()
print("the cost side, briefly:")
for v, d in ((50_000, 300), (50_000, 768), (200_000, 4096)):
print(" vocab %7d x %4d dims = %11s parameters (%5.1f MB at float32)"
% (v, d, "{:,}".format(v * d), v * d * 4 / 1024 / 1024))
print(" in a large language model this table is often the single biggest")
print(" layer, and it is frequently tied to the output layer to halve it.")
Output
Try it yourself
Run the analogy. Press Run the Famous Analogy and follow the vector arithmetic. The result is a point in space. And the answer is whichever word happens to lie nearest to it.
Look at what is near what. Note which words cluster. Tied terms group together not since anyone grouped them, but since they appeared in close contexts in training.
Watch the way, not the spot. The vector from man to woman points much the same way as the vector from king to queen. It is that parallelism, repeated across many pairs, that makes the arithmetic work at all.
Static and context
Word2Vec, GloVe and FastText give static embeddings: one fixed vector per word, for good. That breaks on polysemy — bank gets a single vector that averages the riverside and the financial senses, serving neither.
BERT and each modern language model give context embeddings instead. The vector for a word is computed from the sentence it appears in. So bank in “river bank” and “bank account” gets truly else forms. This is the single biggest improvement in word form since embeddings were added. And it is why static embeddings are now mostly of historical and teaching interest.
Common mistakes
With cosine likeness on unnormalised vectors and calling it distance. Cosine measures angle and ignores size. This is much what you want. But it is not Euclidean distance and the two rank neighbours else.
Expecting a static embedding to handle ambiguity. One vector per word cannot represent two senses.
Ignoring inherited bias. Embeddings learn the numeric links of their training corpus, with the prejudiced ones. And those propagate into a thing built on top.
Over-large embedding dimensions on small vocabularies. 300 dimensions for 500 words is mostly free parameters to overfit with.
The short version
An embedding replaces a sparse, meaningless one-hot vector with a short dense one whose spot encodes meaning. So close words end up close and knowledge transfers between them. Links appear as steady ways, which is what makes vector analogies work. Static embeddings give each word one vector and cannot handle ambiguity; context embeddings compute the vector from the close sentence. And that is what modern models use.
What they are used for
Semantic search and RAG. Embed the question and each stored passage; retrieve the passages with the highest cosine likeness. Each vector store is a machine for doing this quickly.
Recommendations. Embed users and things in the same space and recommend the nearest things.
Classification with little data. Embed the text and train a simple classifier on the vectors. Much beats fine-tuning when you have a few hundred labels.
Clustering and topic find. Embed, then run k-means or HDBSCAN in the embedding space.
Deduplication. Support tickets or product listings above about 0.9 likeness are much the same thing said twice.
High-cardinality categorical features. An embedding layer instead of 10,000 one-hot columns.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
vecs = model.encode(["a cat on a mat", "a feline on a rug"],
normalize_embeddings=True)
float(vecs[0] @ vecs[1]) # ~0.7 - close in meaning, no words shared
What they get wrong
They inherit the training data's biases. Word2Vec's doctor − man + woman famously lands near nurse. The vectors encode numeric regularities of the corpus, with its prejudices, and with them in hiring or lending pipelines propagates that.
Likeness is not truth. "The drug is effective" and "The drug is not effective" are close in most embedding spaces,. This is since they share nearly each word. Negation, counts and named entities are all weak spots. Retrieval systems built on embeddings alone will confidently return the opposite of what was asked. This is why hybrid retrieval — embeddings plus keyword search — is standard in live.
Domain mismatch. A model trained on web text has thin coverage of medical or legal vocabulary. Fine-tuning on domain text, or choosing a domain-set model, matters more than the general benchmark ranking.
They are model-set. Vectors from two else models are not comparable. So changing model means re-embedding your whole corpus.
Questions people ask
What dimension should I use? 384 to 1,536 for text. More costs more memory and search time for shrinking gains; for categorical features, a rough starting point is the fourth root of the cardinality.
Word or sentence embeddings? Sentence embeddings for search, likeness and classification. Word embeddings for token-level tasks or as a cheap baseline.
Do I need a vector store? Below about 100,000 vectors, a NumPy matrix and a dot product is fine. Above that, an approximate index (FAISS, HNSW) pays for it.
Can I fine-tune embeddings? Yes — contrastive training on your own pairs of tied and unrelated text. This much beats a general-purpose model on a set domain.
Why do all my similarities look high? Some models pack all of it into a narrow cone of the space. Relative ranking still works; absolute thresholds need calibrating.
Are embeddings interpretable? Single dimensions, no. Ways in the space much are — and probing classifiers can find them.
Recap in one screen
An embedding is a dense learned vector positioned so that close things sit near each else.
Fixed embeddings give one vector per word; context ones give a else vector per instance.
Compare with cosine likeness, and calibrate the threshold against known pairs.
They power search, RAG, recommendations, clustering and high-cardinality features.
They inherit corpus bias, handle negation and counts poorly, and are not comparable across models.
Check yourself
0 of 3
Answer without scrolling back up.
What is an embedding?
Embeddings place words in a space where distance means something. Nothing about the meaning is written down - it is inferred entirely from which contexts a word turns up in.
Why are embeddings better than one-hot vectors for words?
Under one-hot, 'cat' is exactly as far from 'dog' as from 'parliament'. Embeddings can put related words near each other, which is the whole point.
Embeddings trained on ordinary web text reliably reproduce:
The vectors encode how words are actually used, including every stereotype in the corpus. This is measurable, well documented, and a real problem in deployed systems.
Cheat sheet
What are Embeddings?
Replace sparse one-hot vectors with short dense vectors where similar words sit close together and vector arithmetic encodes meaning.
GloVe: Global Vectors for Word RepresentationPennington, Socher & Manning, EMNLP 2014
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.