A neuron computes w·x + b. Try handing it the word “cat” and see exactly where — and why — the math breaks.
Overview
The One-Sentence Argument
Every neuron in every network — from a 1958 perceptron to GPT — computes some flavour of w · x + b. Multiplication is only defined for numbers. Therefore, before any text can enter any network, it must be converted into numbers. That conversion is text encoding, and it isn't an optimization — it's a precondition.
Input Text
The Neuron
y = w · x + b
w = 0.5 b = 0.1
Multiplication and addition. That's all a neuron can do — and you cannot multiply a word.
Processing Pipeline
IDLE
Text
—
Encoder
—
Neuron w·x+b
—
Output
—
Computation Log
0 OPS
Token
x (input)
w · x + b
Result
Feed some input to the neuron to see the arithmetic.
Why Text Must Become Numbers
The most fundamental fact in NLP: neural networks are arithmetic machines, and words are not numbers.
What "Feeding Raw Text" Actually Does
Try to compute 0.5 × "cat" + 0.1 in any language and you get an error or NaN — the arithmetic is simply undefined. In practice a framework like PyTorch refuses at the door: tensors hold floats, not strings. The demo above makes this failure visible instead of hiding it in a stack trace.
What the Encoder Buys You
Insert one stage — a tokenizer plus a lookup table assigning each word an ID — and the identical neuron suddenly works: 0.5 × 2 + 0.1 = 1.1. Every downstream marvel (embeddings, attention, generation) rests on this humble dictionary lookup. The quality of the numbers matters enormously too, which is why the next modules explore encoding techniques and embeddings.
Models do arithmetic, not language
Every model you can train is arithmetic underneath. A linear model multiplies inputs by weights. A neural network does matrix multiplications. A decision tree compares numbers against thresholds.
None of that applies to the string "hello". There is no defined way to multiply a word by a weight, so before any modelling can happen, text has to become numbers — and how you do that determines what the model can possibly learn.
The requirement is not merely "any numbers". Three properties matter:
Consistency. The same word must map to the same representation every time, or the model cannot accumulate evidence about it.
Coverage. Words absent from training must produce something usable rather than a crash or a meaningless <UNK>.
Meaningful geometry. Ideally, related words should end up near each other, so that what the model learns about one transfers to the other.
That third property is the one the early methods lack and embeddings provide, and it is the reason the field moved.
Why the naive approach fails
The obvious idea — assign each word an integer — breaks immediately.
Suppose cat = 1, dog = 2, aardvark = 3. A model doing arithmetic on those numbers now believes dog is twice cat, and that aardvark sits beyond dog on some scale. Nothing about the words justifies that ordering, which came from the alphabet.
For a model that compares magnitudes — linear regression, a neural network, anything distance-based — that invented ordering is a real distortion. It will learn patterns from the numbering itself.
One-hot encoding fixes the ordering problem by giving each word its own dimension, so no word is numerically greater than another. It replaces it with two new problems: a 50,000-dimensional vector per word, and every pair of words exactly equidistant — so "cat" and "dog" are as unrelated as "cat" and "bureaucracy".
Everything since has been an attempt to keep one-hot's neutrality while adding useful structure.
What each level of encoding buys
Encoding
Fixes
Still missing
Integer ids
Nothing — introduces false ordering
Everything
One-hot
False ordering
Similarity, compactness
Bag of words / TF-IDF
Document representation, weighting by rarity
Word order, similarity
Word embeddings
Similarity, compactness
Word order, context
Contextual embeddings
Context, order, polysemy
Nothing structural
Reading down that table is a summary of the history of NLP. Each row is a real improvement, and each was the state of the art for years.
A network multiplies numbers, and a word is not one
Every layer is a matrix multiply, so text has to become numbers before anything else can happen. The obvious way of doing that fails in a specific and measurable way, and that failure is why embeddings exist.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
print("this is the entire reason. a layer computes:")
print(" output = input @ W + b")
print("and there is no definition of 'cat' @ W.")
try:
"cat" @ np.zeros((3, 3))
except TypeError as e:
print(" trying it: TypeError -- %s" % str(e)[:56])
print()
vocab = ["cat", "car", "dog", "truck", "happy", "sad"]
idx = {w: i for i, w in enumerate(vocab)}
print("THE OBVIOUS FIX -- number them:")
for w in vocab:
print(" %-8s -> %d" % (w, idx[w]))
print()
print("and it fails immediately, because arithmetic on those numbers means")
print("something and the numbers were arbitrary:")
print(" cat(%d) + dog(%d) = %d, which is '%s'"
% (idx["cat"], idx["dog"], idx["cat"] + idx["dog"], vocab[idx["cat"] + idx["dog"]]))
print(" (car(%d) + truck(%d)) / 2 = %.1f, which rounds to '%s'"
% (idx["car"], idx["truck"], (idx["car"] + idx["truck"]) / 2,
vocab[int((idx["car"] + idx["truck"]) / 2)]))
print(" is 'happy' further from 'cat' than 'dog' is? the numbers say yes:")
for w in ("dog", "car", "happy", "sad"):
print(" |cat - %-6s| = %d" % (w, abs(idx["cat"] - idx[w])))
print(" that ordering came from the order I typed the list in. a model")
print(" fitting these numbers is fitting my typing.")
print()
print("HOW BADLY. train a linear model on integer codes to predict a")
print("property that has nothing to do with the ordering:")
N = 800
animal = {"cat": 1, "dog": 1, "car": 0, "truck": 0, "happy": 0, "sad": 0}
words = [vocab[i] for i in rng.integers(0, len(vocab), N)]
y = np.array([animal[w] for w in words], float)
X_int = np.array([[idx[w]] for w in words], float)
X_int = np.column_stack([X_int, np.ones(N)])
w_int = np.linalg.lstsq(X_int, y, rcond=None)[0]
acc_int = (((X_int @ w_int) > 0.5) == (y > 0.5)).mean()
V = len(vocab)
X_hot = np.zeros((N, V))
for i, wd in enumerate(words):
X_hot[i, idx[wd]] = 1
X_hot = np.column_stack([X_hot, np.ones(N)])
w_hot = np.linalg.lstsq(X_hot, y, rcond=None)[0]
acc_hot = (((X_hot @ w_hot) > 0.5) == (y > 0.5)).mean()
baseline = max(y.mean(), 1 - y.mean())
print(" always guess the commoner class : accuracy %.4f" % baseline)
print(" integer codes : accuracy %.4f" % acc_int)
print(" one-hot : accuracy %.4f" % acc_hot)
print(" the animals sit at indices %s and the rest at %s, so no single"
% ([idx[w] for w in vocab if animal[w]],
[idx[w] for w in vocab if not animal[w]]))
print(" threshold on the index separates them. the integer model has ONE")
print(" weight for all six words and can only express 'higher index means")
print(" more animal', which is not true here. it beats the do-nothing")
print(" baseline by %.3f, which is to say by almost nothing."
% (acc_int - baseline))
print(" one-hot gives each word its own weight and fits exactly.")
print()
print(" and note how fragile that is: list the two animals FIRST and the")
print(" integer model would score 1.0000, having learned nothing except")
print(" the order I typed the vocabulary in.")
print()
print("SO ONE-HOT IS CORRECT AND ALSO UNUSABLE at real vocabulary sizes:")
for v, d in ((6, 8), (50_000, 300), (200_000, 1024)):
print(" vocab %7d: one-hot input is %7d wide, so the first layer is"
% (v, v))
print(" %7d x %d = %13s weights" % (v, d, "{:,}".format(v * d)))
print(" and %.4f%% of every input vector is zero." % (100 * (1 - 1 / 50_000)))
print()
print(" but look at what that first layer DOES with a one-hot input:")
E = rng.normal(0, 0.4, (V, 4))
onehot = np.zeros(V); onehot[idx["dog"]] = 1
print(" one-hot('dog') @ E = %s" % np.round(onehot @ E, 4))
print(" E[%d] = %s" % (idx["dog"], np.round(E[idx["dog"]], 4)))
print(" identical: %s" % np.allclose(onehot @ E, E[idx["dog"]]))
print(" multiplying by a one-hot vector SELECTS A ROW. so you skip the")
print(" multiplication and index instead -- and that is an embedding")
print(" layer. it is not a different idea, it is the same operation done")
print(" without the wasted arithmetic.")
print()
print("THE THREE PROPERTIES a usable encoding needs, and how each scheme")
print("scores:")
rows = [("no false ordering", "no", "yes", "yes"),
("compact", "yes", "no", "yes"),
("similar words nearby", "no", "no", "yes")]
print("%26s %14s %12s %14s" % ("", "integer codes", "one-hot", "embedding"))
for r in rows:
print("%26s %14s %12s %14s" % r)
print()
print("the third row is the one that matters most and the one neither of the")
print("first two schemes can offer at all. 'cat' and 'dog' are equidistant")
print("from everything under one-hot, so a model that learns something about")
print("cats learns nothing about dogs. an embedding places them near each")
print("other, and that sharing is most of why the approach works.")
Output
Guided experiments
Click "Feed RAW text". Watch the pipeline: the encoder stage is skipped, the neuron receives strings, and every row of the log ends in NaN. The output stage turns red — a dead end.
Click "Encode first". Same text, same neuron — but now the encoder assigns each word an ID, and the arithmetic sails through with real numbers at every step.
Type your own sentence and run both paths. Whatever the words, the pattern is identical: raw text breaks, encoded text computes.
In one line
Text encoding isn't a preprocessing nicety — it is the bridge without which no NLP is possible. A neural network can no more process the raw word "cat" than a calculator can. First we turn language into numbers; everything else in NLP is about turning it into good numbers.
The unknown word problem
A word-level vocabulary is built from training text. Anything else becomes <UNK>, and its meaning is gone.
That happens constantly with real text: names, product codes, typos, new slang, technical terms, other languages. A vocabulary of 50,000 English words covers ordinary prose reasonably and fails on anything specialised.
Three responses, in historical order:
Character-level encoding never meets an unknown symbol, because the alphabet is closed. The cost is very long sequences and a model that must learn spelling before meaning.
Subword tokenisation is the modern answer. Common words stay whole; rare ones split into fragments that were seen in training; and the vocabulary includes individual characters as a fallback, so nothing is ever unknown. "Unhappiness" becomes "un" + "happi" + "ness" even if the whole word never appeared.
FastText represents a word as the sum of its character n-gram vectors, so an unseen word still gets a sensible vector from its parts.
Subword tokenisation is why modern models have no <UNK> token in normal operation, and it is one of the quieter reasons they generalise better than their predecessors.
The practical pipeline
For a transformer, the whole path from text to model input:
Normalise — Unicode form, whitespace, markup. Not case or punctuation.
Tokenise into subwords with the model's own tokeniser.
Map to ids — each token becomes an integer index.
Embed — each id looks up a learned vector.
Add positional information, since attention is order-blind.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
batch = tok(["encoding is not optional"], padding=True, truncation=True,
return_tensors="pt")
batch["input_ids"] # integers, ready for the embedding layer
The critical constraint: the tokeniser must be the model's own. Token id 1,547 means one thing to one vocabulary and something else to another, and a mismatch produces fluent nonsense rather than an error.
Questions people ask
Why not just use integer ids directly? They imply an ordering that does not exist, and models that compare magnitudes will learn from it.
Is one-hot encoding ever right for text? Only for tiny vocabularies or as a teaching example. It is wide, sparse and carries no similarity.
Do embeddings need training? They are learned — either as part of your model, or downloaded pretrained, which is almost always the better option for text.
What happens to punctuation and emoji? Modern tokenisers include them, and they carry real signal for sentiment and intent. Do not strip them.
How are numbers encoded? Inconsistently, by subword tokenisers — "1234" may be one token or four. It is a known weakness behind poor arithmetic in language models.
Can the same encoding serve several languages? Yes — multilingual subword vocabularies cover many scripts, at the cost of more tokens per word for languages far from the training distribution.
Recap in one screen
Models do arithmetic, so text must become numbers before anything else happens.
Plain integer ids invent an ordering that does not exist; one-hot removes it but carries no similarity.
TF-IDF adds weighting by rarity; embeddings add similarity; contextual embeddings add context and order.
Subword tokenisation removes the unknown-word problem, which is why <UNK> has largely disappeared.
Always use the model's own tokeniser — a mismatch produces confident nonsense.
Recall check
0 of 2
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Normalise” here?
— Unicode form, whitespace, markup. Not case or punctuation.
What is meant by “Tokenise” here?
into subwords with the model's own tokeniser.
Cheat sheet
Why Text Encoding is Needed in NLP
Every neuron in every network — from a 1958 perceptron to GPT — computes some flavour of w · x + b. Multiplication is only defined for numbers. Therefore, before any text can enter any network, it must be converted into numbers. That conversion is text encoding, and it isn't an optimization — it's a precondition.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.