vizlearn

Home / Natural Language Processing

Why Text Encoding is Needed in NLP

Natural Language ProcessingBy Ashish JangraUpdated 19 October 2026

A neuron computes w·x + b. Try handing it the word “cat” and see exactly where — and why — the math breaks.

Input Text

The Neuron

y = w · x + b

w = 0.5 b = 0.1

Multiplication and addition. That's all a neuron can do — and you cannot multiply a word.

Processing Pipeline

IDLE

Text

—

Encoder

—

Neuron w·x+b

—

Output

—

Computation Log

0 OPS

Token	x (input)	w · x + b	Result

Feed some input to the neuron to see the arithmetic.

Overview

A neuron computes w·x + b.

In this guide

The One-Sentence Input
What "Feeding Raw Text" Really Does
What the Encoder Buys You
Models do arithmetic, not language
Why the naive approach fails
What each level of encoding buys
A network multiplies numbers, and a word is not one
Guided tests
In one line
The unknown word problem
The real pipeline
Questions people ask
Recap in one screen

Why Text Must Become Numbers

The most fundamental fact in NLP: neural networks are arithmetic machines, and words are not numbers.

The One-Sentence Input

Each neuron in each network — from a 1958 perceptron to GPT — computes some flavour of w · x + b. Product is only set for numbers. So, before any text can enter any network, it must be converted into numbers. That conversion is text encoding, and it isn't an optimization — it's a precondition. The simplest encoding is one-hot: assign each word in the vocabulary an integer index and represent it as a vector of zeros with a single one at that index. The vector's length equals the vocabulary size, so a 50,000-word vocabulary gives 50,000-dimensional vectors that are almost fully zeros. One-hot vectors carry no notion of likeness: "king" and "queen" are as far apart as "king" and "bicycle." Word embeddings solve this by mapping each word to a dense vector of 100-300 dimensions, learned. That way, words with close meanings land near each other. Word2Vec, GloVe, and FastText are pretrained embeddings; transformer models like BERT learn task-set embeddings during fine-tuning. At the subword level, BPE tokenisation splits words into pieces. This handles out-of-vocabulary words and morphological change without a fixed dictionary. This page converts a sentence into one-hot vectors, shows the sparsity, and compares them to dense embeddings side by side.

What "Feeding Raw Text" Really Does

Try to compute 0.5 × "cat" + 0.1 in any language and you get an error or NaN — the arithmetic is simply undefined. In practice a framework like PyTorch refuses at the door: tensors hold floats, not strings. The demo above makes this failure visible instead of hiding it in a stack trace.

What the Encoder Buys You

Insert one stage — a tokenizer plus a lookup table assigning each word an ID — and the the same neuron suddenly works: 0.5 × 2 + 0.1 = 1.1. Each downstream marvel (embeddings, attention, generation) rests on this humble dictionary lookup. The grade of the numbers matters enormously too, which is why the next modules explore encoding methods and embeddings.

Models do arithmetic, not language

Each model you can train is arithmetic below. A linear model multiplies inputs by weights. A neural network does matrix products. A decision tree compares numbers against thresholds.

None of that applies to the string "hello". There is no set way to times a word by a weight. So before any modelling can happen, text has to turn numbers. And how you do that decides what the model can possibly learn.

The requirement is not merely "any numbers". Three properties matter:

Consistency. The same word must map to the same form each time. Or the model cannot build up evidence about it.

Coverage. Words absent from training must give a thing usable rather than a crash or a meaningless <UNK>.

Meaningful geometry. Ideally, tied words should end up near each other. So that what the model learns about one transfers to the other.

That third property is the one the early methods lack and embeddings provide. And it is the reason the field moved.

Why the naive approach fails

The obvious idea — assign each word an integer — breaks at once.

Suppose cat = 1, dog = 2, aardvark = 3. A model doing arithmetic on those numbers now believes dog is twice cat. And that aardvark sits beyond dog on some scale. Nothing about the words justifies that sorting, which came from the alphabet.

For a model that compares sizes. Linear regression, a neural network, a thing distance-based. That invented sorting is a real distortion. It will learn patterns from the numbering itself.

One-hot encoding fixes the sorting problem by giving each word its own dimension. So no word is in numbers greater than one more. It replaces it with two new problems: a 50,000-dimensional vector per word. And each pair of words just equidistant. So "cat" and "dog" are as unrelated as "cat" and "bureaucracy".

All of it since has been an try to keep one-hot's neutrality while adding useful structure.

What each level of encoding buys

Encoding	Fixes	Still missing
Integer ids	Nothing — introduces false ordering	Everything
One-hot	False ordering	Similarity, compactness
Bag of words / TF-IDF	Document representation, weighting by rarity	Word order, similarity
Word embeddings	Similarity, compactness	Word order, context
Contextual embeddings	Context, order, polysemy	Nothing structural

Reading down that table is a summary of the history of NLP. Each row is a real improvement, and each was the state of the art for years.

A network multiplies numbers, and a word is not one

Each layer is a matrix times, so text has to turn numbers before a thing else can happen. The obvious way of doing that fails in a set and measurable way. And that failure is why embeddings exist.

example_01.pyNumPy

import numpy as np

rng = np.random.default_rng(0)

print("this is the entire reason. a layer computes:")
print("    output = input @ W + b")
print("and there is no definition of 'cat' @ W.")
try:
    "cat" @ np.zeros((3, 3))
except TypeError as e:
    print("    trying it: TypeError -- %s" % str(e)[:56])
print()

vocab = ["cat", "car", "dog", "truck", "happy", "sad"]
idx = {w: i for i, w in enumerate(vocab)}
print("THE OBVIOUS FIX -- number them:")
for w in vocab:
    print("   %-8s -> %d" % (w, idx[w]))
print()
print("and it fails immediately, because arithmetic on those numbers means")
print("something and the numbers were arbitrary:")
print("   cat(%d) + dog(%d) = %d, which is '%s'"
      % (idx["cat"], idx["dog"], idx["cat"] + idx["dog"], vocab[idx["cat"] + idx["dog"]]))
print("   (car(%d) + truck(%d)) / 2 = %.1f, which rounds to '%s'"
      % (idx["car"], idx["truck"], (idx["car"] + idx["truck"]) / 2,
         vocab[int((idx["car"] + idx["truck"]) / 2)]))
print("   is 'happy' further from 'cat' than 'dog' is? the numbers say yes:")
for w in ("dog", "car", "happy", "sad"):
    print("      |cat - %-6s| = %d" % (w, abs(idx["cat"] - idx[w])))
print("   that ordering came from the order I typed the list in. a model")
print("   fitting these numbers is fitting my typing.")
print()

Output

Building on the same setup, the next block shows the left cases.

example_02.pyNumPy

import numpy as np

print("HOW BADLY. train a linear model on integer codes to predict a")
print("property that has nothing to do with the ordering:")
N = 800
animal = {"cat": 1, "dog": 1, "car": 0, "truck": 0, "happy": 0, "sad": 0}
words = [vocab[i] for i in rng.integers(0, len(vocab), N)]
y = np.array([animal[w] for w in words], float)
X_int = np.array([[idx[w]] for w in words], float)
X_int = np.column_stack([X_int, np.ones(N)])
w_int = np.linalg.lstsq(X_int, y, rcond=None)[0]
acc_int = (((X_int @ w_int) > 0.5) == (y > 0.5)).mean()

V = len(vocab)
X_hot = np.zeros((N, V))
for i, wd in enumerate(words):
    X_hot[i, idx[wd]] = 1
X_hot = np.column_stack([X_hot, np.ones(N)])
w_hot = np.linalg.lstsq(X_hot, y, rcond=None)[0]
acc_hot = (((X_hot @ w_hot) > 0.5) == (y > 0.5)).mean()

Output

The below block covers the left steps.

example_03.pyNumPy

import numpy as np

baseline = max(y.mean(), 1 - y.mean())
print("   always guess the commoner class : accuracy %.4f" % baseline)
print("   integer codes                   : accuracy %.4f" % acc_int)
print("   one-hot                         : accuracy %.4f" % acc_hot)
print("   the animals sit at indices %s and the rest at %s, so no single"
      % ([idx[w] for w in vocab if animal[w]],
         [idx[w] for w in vocab if not animal[w]]))
print("   threshold on the index separates them. the integer model has ONE")
print("   weight for all six words and can only express 'higher index means")
print("   more animal', which is not true here. it beats the do-nothing")
print("   baseline by %.3f, which is to say by almost nothing."
      % (acc_int - baseline))
print("   one-hot gives each word its own weight and fits exactly.")
print()
print("   and note how fragile that is: list the two animals FIRST and the")
print("   integer model would score 1.0000, having learned nothing except")
print("   the order I typed the vocabulary in.")
print()

Output

The next block goes on with the left demonstrations.

example_04.pyNumPy

import numpy as np

print("SO ONE-HOT IS CORRECT AND ALSO UNUSABLE at real vocabulary sizes:")
for v, d in ((6, 8), (50_000, 300), (200_000, 1024)):
    print("   vocab %7d: one-hot input is %7d wide, so the first layer is"
          % (v, v))
    print("      %7d x %d = %13s weights" % (v, d, "{:,}".format(v * d)))
print("   and %.4f%% of every input vector is zero." % (100 * (1 - 1 / 50_000)))
print()
print("   but look at what that first layer DOES with a one-hot input:")
E = rng.normal(0, 0.4, (V, 4))
onehot = np.zeros(V); onehot[idx["dog"]] = 1
print("      one-hot('dog') @ E = %s" % np.round(onehot @ E, 4))
print("      E[%d]              = %s" % (idx["dog"], np.round(E[idx["dog"]], 4)))
print("      identical: %s" % np.allclose(onehot @ E, E[idx["dog"]]))
print("   multiplying by a one-hot vector SELECTS A ROW. so you skip the")
print("   multiplication and index instead -- and that is an embedding")
print("   layer. it is not a different idea, it is the same operation done")
print("   without the wasted arithmetic.")
print()

print("THE THREE PROPERTIES a usable encoding needs, and how each scheme")
print("scores:")
rows = [("no false ordering", "no", "yes", "yes"),
        ("compact", "yes", "no", "yes"),
        ("similar words nearby", "no", "no", "yes")]
print("%26s %14s %12s %14s" % ("", "integer codes", "one-hot", "embedding"))
for r in rows:
    print("%26s %14s %12s %14s" % r)
print()
print("the third row is the one that matters most and the one neither of the")
print("first two schemes can offer at all. 'cat' and 'dog' are equidistant")
print("from everything under one-hot, so a model that learns something about")
print("cats learns nothing about dogs. an embedding places them near each")
print("other, and that sharing is most of why the approach works.")

Output

Guided tests

Click "Feed RAW text". Watch the pipeline: the encoder stage is skipped, the neuron receives strings. And each row of the log ends in NaN. The output stage turns red — a dead end.
Click "Encode first". Same text, same neuron — but now the encoder assigns each word an ID. And the arithmetic sails through with real numbers at each step.
Type your own sentence and run both paths. Any the words, the pattern is the same: raw text breaks, encoded text computes.

In one line

Text encoding isn't a preprocessing nicety — it is the bridge without which no NLP is doable. A neural network can no more process the raw word "cat" than a calculator can. First we turn language into numbers; all of it else in NLP is about turning it into good numbers.

The unknown word problem

A word-level vocabulary is built from training text. A thing else turns <UNK>, and its meaning is gone.

That happens constantly with real text: names, product codes, typos, new slang, technical terms, other languages. A vocabulary of 50,000 English words covers ordinary prose fairly and fails on a thing specialised.

Three responses, in historical order:

Character-level encoding never meets an unknown symbol, because the alphabet is closed. The cost is very long sequences and a model that must learn spelling before meaning.

Subword tokenisation is the modern answer. Common words stay whole; rare ones split into fragments that were seen in training; and the vocabulary includes single characters as a fallback. So nothing is ever unknown. "Unhappiness" turns "un" + "happi" + "ness" even if the whole word never appeared.

FastText represents a word as the sum of its character n-gram vectors. So an unseen word still gets a sensible vector from its parts.

Subword tokenisation is why modern models have no <UNK> token in normal step. And it is one of the quieter reasons they generalise better than their parents.

The real pipeline

For a transformer, the whole path from text to model input:

Normalise — Unicode form, whitespace, markup. Not case or punctuation.
Tokenise into subwords with the model's own tokeniser.
Map to ids — each token turns an integer index.
Embed — each id looks up a learned vector.
Add positional data, since attention is order-blind.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")

batch = tok(["encoding is not optional"], padding=True, truncation=True,
            return_tensors="pt")
batch["input_ids"]        # integers, ready for the embedding layer

The critical constraint: the tokeniser must be the model's own. Token id 1,547 means one thing to one vocabulary and a thing else to one more. And a mismatch gives fluent nonsense rather than an error.

Questions people ask

Why not just use integer ids right? They imply an sorting that does not exist, and models that compare sizes will learn from it.

Is one-hot encoding ever right for text? Only for tiny vocabularies or as a teaching case. It is wide, sparse and carries no likeness.

Do embeddings need training? They are learned — either as part of your model, or downloaded pretrained. This is almost always the better option for text.

What happens to punctuation and emoji? Modern tokenisers include them. And they carry real sign for sentiment and intent. Do not strip them.

How are numbers encoded? Inconsistently, by subword tokenisers — "1234" may be one token or four. It is a known weakness behind poor arithmetic in language models.

Can the same encoding serve a few languages? Yes — multilingual subword vocabularies cover many scripts, at the cost of more tokens per word for languages far from the training distribution.

Recap in one screen

Models do arithmetic, so text must turn numbers before a thing else happens.
Plain integer ids invent an sorting that does not exist; one-hot removes it but carries no likeness.
TF-IDF adds weighting by rarity; embeddings add likeness; context embeddings add context and order.
Subword tokenisation removes the unknown-word problem, which is why <UNK> has largely disappeared.
Always use the model's own tokeniser — a mismatch gives confident nonsense.

Recall check

0 of 2

Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.

What is meant by “Normalise” here?
What is meant by “Tokenise” here?

Cheat sheet