Text Encoding Techniques in NLP
Four classic ways to turn a corpus into numbers — computed live from a corpus you can edit.
Four classic ways to turn a corpus into numbers — computed live from a corpus you can edit.
Once you accept that text must become numbers, the question is which numbers. The four classic answers form a ladder — each rung keeps more data or removes more distortion than the one below it.
Label encoding assigns each word an integer — compact. But the integers imply a fake ordering that the model will treat as real. One-hot encoding removes that lie by giving each word its own dimension, at the cost of vectors as wide as the vocabulary and no notion of likeness. Bag of words counts words per document, so documents can be compared, but common filler words dominate. TF-IDF reweights those counts by rarity, so words distinctive to a single document score high while words that appear all over score near zero.
Each method gives a fixed form: the same word always gets the same vector, no matter context. That limitation is what word embeddings (Word2Vec, GloVe) and context embeddings (BERT, GPT) later solved — learned vectors that encode meaning. And that change with the sentence around them. But the four classic methods remain the right choice when the input is tabular categories. Rather than natural language, or when the dataset is too small to train embeddings.
Grasp this ladder also clarifies what you lose at each rung. Label encoding loses honesty about distance. One-hot loses compactness and likeness. Bag of words loses word order. TF-IDF loses context. Knowing which loss matters for your task tells you which rung to stop at. And when to climb higher.
Four classic ways to turn a corpus into numbers — computed live from a corpus you can edit.
Label encoding, one-hot, bag-of-words, and TF-IDF — each fixes a weakness of the one before it.
Once you accept that text must become numbers, the question is which numbers. The four classic answers form a ladder — each rung keeps more data or removes more distortion than the one below it.
Label encoding assigns each word an integer — compact. But the integers imply a fake ordering that the model will treat as real. One-hot encoding removes that lie by giving each word its own dimension, at the cost of vectors as wide as the vocabulary and no notion of likeness. Bag of words counts words per document, so documents can be compared, but common filler words dominate. TF-IDF reweights those counts by rarity, so words distinctive to a single document score high while words that appear all over score near zero.
Each method gives a fixed form: the same word always gets the same vector, no matter context. That limitation is what word embeddings (Word2Vec, GloVe) and context embeddings (BERT, GPT) later solved — learned vectors that encode meaning. And that change with the sentence around them. But the four classic methods remain the right choice when the input is tabular categories. Rather than natural language, or when the dataset is too small to train embeddings.
Grasp this ladder also clarifies what you lose at each rung. Label encoding loses honesty about distance. One-hot loses compactness and likeness. Bag of words loses word order. TF-IDF loses context. Knowing which loss matters for your task tells you which rung to stop at. And when to climb higher.
Label encoding assigns each vocabulary word an integer. It's maximally compact, but the integers imply a fake ordering — the model may conclude that word #7 is "more" than word #3. One-hot encoding removes that lie by giving each word its own dimension. A vector of zeros with a single 1. Honest, but the vectors are as wide as the vocabulary, almost fully zeros. And each pair of words is just equidistant — no notion of likeness survives.
Bag of words moves from single words to whole documents: each document becomes a vector of word counts. Suddenly documents can be compared — but frequent filler words like "the" dominate the counts while carrying no meaning. TF-IDF fixes that with a reweighting: term count (how often a word appears in this document) times by inverse document count (how rare it is across documents). Words that appear all over score near zero; words distinctive to one document light up.
tf-idf(w, d) = tf(w, d) × log(N / df(w))
Each model needs numbers. The methods form a clear progression, each fixing a limitation of the last.
| Technique | Represents | Captures order? | Captures meaning? |
|---|---|---|---|
| One-hot | One position per word | No | No |
| Bag of words | Word counts per document | No | No |
| TF-IDF | Weighted counts | No | Weakly — via rarity |
| Word embeddings | Learned dense vectors | No | Yes |
| Contextual embeddings | Vectors computed in context | Yes | Yes |
One-hot gives each word a vector of zeros with a single 1. With a 50,000-word vocabulary that is a 50,000-dimensional vector per word. And each pair of words is equally distant — "cat" is no closer to "dog" than to "bureaucracy".
Bag of words counts instances per document, giving one vector per document rather than per word. Simple, surprisingly effective for classification, and it discards order fully: "the dog bit the man" and "the man bit the dog" are the same.
Raw counts overweight common words. "The" appears in each document and tells you nothing about which document you are looking at.
TF-IDF multiplies two terms:
TF-IDF = (count in this document) × log(total documents / documents holding the word)
The second factor is the inverse document count. A word appearing in each document has an IDF near zero and is in effect discarded. A word appearing in a handful gets a large weight, because its presence is informative.
Worked through: with 1,000 documents, a word in all 1,000 has IDF = log(1) = 0. A word in 10 has IDF = log(100) = 4.6. So the rare word's share is amplified and the ubiquitous one's is erased. Automatic stop-word cutting, without a list.
That is why TF-IDF plus logistic regression remains a strong, fast baseline for text classification and should be the first thing you try before reaching for a transformer.
The gap between TF-IDF and embeddings is extension across words.
To TF-IDF, "excellent" and "superb" are two unrelated columns. A model trained on documents holding "excellent" learns nothing about documents holding "superb". Embeddings place them near each other, so evidence transfers.
Static embeddings (Word2Vec, GloVe, FastText) give one vector per word. Good, and blind to context — "bank" has one vector.
Context embeddings (BERT, GPT and successors) compute a vector per instance from the nearby words, resolving ambiguity and capturing order.
To represent a whole document with static embeddings, the usual approach is to average the word vectors —. This works better than it should and still loses order. Sentence encoders trained with a likeness objective do the job properly.
One-hot, bag-of-words, TF-IDF, co-instance and learned embeddings, all built on the same four documents. So the progression from "counting" to "meaning" is visible rather than asserted.
Building on the same setup, the next block shows the left cases.
The below block covers the left steps.
The next block goes on with the left demonstrations.
Building on the same setup, the next block shows the left cases.
Sparse encodings are a progression of fixes: one-hot removes label encoding's fake ordering, bag-of-words adds document structure, TF-IDF suppresses noise words. What none of them can do is say that "cat" and "kitten" are related — for that you need embeddings, the subject of the next module.
| Situation | Reach for |
|---|---|
| Fast baseline, any classification task | TF-IDF + logistic regression |
| Small labelled dataset, need accuracy | Sentence embeddings + a simple classifier |
| Semantic search, RAG, deduplication | Sentence embeddings |
| Token-level tasks (NER, tagging) | Contextual embeddings from a transformer |
| Interpretability required | TF-IDF — coefficients map to words |
| Noisy text, misspellings, many languages | Character n-grams, or FastText |
| Generation of any kind | A decoder-only language model |
Two observations from practice. TF-IDF is not a toy — on topic classification with plenty of labels it often matches a fine-tuned transformer at a thousandth of the cost. And embedding a document and training a small classifier on the vectors often beats fine-tuning when labels are scarce,. This is because there is far less to overfit.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
model = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=3, sublinear_tf=True),
LogisticRegression(max_iter=1000),
)
model.fit(train_texts, train_labels) # a strong baseline in three linesSparse forms (one-hot, bag of words, TF-IDF) are wide and mostly zero. Keep them in sparse matrices. Never call .toarray() on a large one. And use models that handle sparsity natively. Linear models, Naive Bayes, LightGBM.
Dense forms (embeddings) are compact and each dimension is used. They need less memory per document, support fast likeness search, and are not interpretable dimension by dimension.
A useful hybrid exists and is standard in production retrieval. Combine a sparse keyword score (BM25, a refined TF-IDF) with a dense embedding score. Keyword search catches exact terms, names and numbers that embeddings blur; embeddings catch paraphrase that keywords miss. Together they beat either alone, which is why "hybrid search" is the default recommendation for RAG systems.
Is TF-IDF outdated? No. It is fast, interpretable, needs no GPU, and remains strong on topic classification with adequate labels.
What is BM25? A refinement of TF-IDF with term-count saturation and document-length normalisation. The standard keyword retrieval function, and better than plain TF-IDF for search.
Should I average word embeddings for a sentence? It works as a baseline and loses word order. A sentence encoder is materially better.
How do I pick the vocabulary size? Via min_df. Drop terms appearing in fewer than three to five documents. That often removes most of the vocabulary and almost none of the sign.
Do I still need stop-word cutting with TF-IDF? Rarely — IDF already suppresses ubiquitous words.
Which encoding for a language model? Subword tokenisation into learned embeddings, contextualised by the model itself. The earlier methods do not apply.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “A Ladder of Encodings”?
What does this module say about “Label Encoding & One-Hot”?
What does this module say about “Bag of Words & TF-IDF”?
Climb from label encoding to one-hot to bag-of-words to TF-IDF, and see which information each rung preserves and which it drops.