Text Encoding Techniques in NLP
Four classic ways to turn a corpus into numbers — computed live from a corpus you can edit.
Four classic ways to turn a corpus into numbers — computed live from a corpus you can edit.
Label encoding, one-hot, bag-of-words, and TF-IDF — each fixes a weakness of the one before it.
Once you accept that text must become numbers, the question is which numbers. The four classic answers form a ladder — each rung keeps more information or removes more distortion than the one below it.
Label encoding assigns each vocabulary word an integer. It's maximally compact, but the integers imply a fake ordering — the model may conclude that word #7 is "more" than word #3. One-hot encoding removes that lie by giving each word its own dimension: a vector of zeros with a single 1. Honest, but the vectors are as wide as the vocabulary, almost entirely zeros, and every pair of words is exactly equidistant — no notion of similarity survives.
Bag of words moves from single words to whole documents: each document becomes a vector of word counts. Suddenly documents can be compared — but frequent filler words like "the" dominate the counts while carrying no meaning. TF-IDF fixes that with a reweighting: term frequency (how often a word appears in this document) multiplied by inverse document frequency (how rare it is across documents). Words that appear everywhere score near zero; words distinctive to one document light up.
tf-idf(w, d) = tf(w, d) × log(N / df(w))
Every model needs numbers. The methods form a clear progression, each fixing a limitation of the last.
| Technique | Represents | Captures order? | Captures meaning? |
|---|---|---|---|
| One-hot | One position per word | No | No |
| Bag of words | Word counts per document | No | No |
| TF-IDF | Weighted counts | No | Weakly — via rarity |
| Word embeddings | Learned dense vectors | No | Yes |
| Contextual embeddings | Vectors computed in context | Yes | Yes |
One-hot gives each word a vector of zeros with a single 1. With a 50,000-word vocabulary that is a 50,000-dimensional vector per word, and every pair of words is equally distant — "cat" is no closer to "dog" than to "bureaucracy".
Bag of words counts occurrences per document, producing one vector per document rather than per word. Simple, surprisingly effective for classification, and it discards order entirely: "the dog bit the man" and "the man bit the dog" are identical.
Raw counts overweight common words. "The" appears in every document and tells you nothing about which document you are looking at.
TF-IDF multiplies two terms:
TF-IDF = (count in this document) × log(total documents / documents containing the word)
The second factor is the inverse document frequency. A word appearing in every document has an IDF near zero and is effectively discarded. A word appearing in a handful gets a large weight, because its presence is informative.
Worked through: with 1,000 documents, a word in all 1,000 has IDF = log(1) = 0. A word in 10 has IDF = log(100) = 4.6. So the rare word's contribution is amplified and the ubiquitous one's is erased — automatic stop-word removal, without a list.
That is why TF-IDF plus logistic regression remains a strong, fast baseline for text classification and should be the first thing you try before reaching for a transformer.
The gap between TF-IDF and embeddings is generalisation across words.
To TF-IDF, "excellent" and "superb" are two unrelated columns. A model trained on documents containing "excellent" learns nothing about documents containing "superb". Embeddings place them near each other, so evidence transfers.
Static embeddings (Word2Vec, GloVe, FastText) give one vector per word. Good, and blind to context — "bank" has one vector.
Contextual embeddings (BERT, GPT and successors) compute a vector per occurrence from the surrounding words, resolving ambiguity and capturing order.
To represent a whole document with static embeddings, the usual approach is to average the word vectors — which works better than it should and still loses order. Sentence encoders trained with a similarity objective do the job properly.
One-hot, bag-of-words, TF-IDF, co-occurrence and learned embeddings, all built on the same four documents -- so the progression from "counting" to "meaning" is visible rather than asserted.
Sparse encodings are a progression of fixes: one-hot removes label encoding's fake ordering, bag-of-words adds document structure, TF-IDF suppresses noise words. What none of them can do is say that "cat" and "kitten" are related — for that you need embeddings, the subject of the next module.
| Situation | Reach for |
|---|---|
| Fast baseline, any classification task | TF-IDF + logistic regression |
| Small labelled dataset, need accuracy | Sentence embeddings + a simple classifier |
| Semantic search, RAG, deduplication | Sentence embeddings |
| Token-level tasks (NER, tagging) | Contextual embeddings from a transformer |
| Interpretability required | TF-IDF — coefficients map to words |
| Noisy text, misspellings, many languages | Character n-grams, or FastText |
| Generation of any kind | A decoder-only language model |
Two observations from practice. TF-IDF is not a toy — on topic classification with plenty of labels it frequently matches a fine-tuned transformer at a thousandth of the cost. And embedding a document and training a small classifier on the vectors often beats fine-tuning when labels are scarce, because there is far less to overfit.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
model = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=3, sublinear_tf=True),
LogisticRegression(max_iter=1000),
)
model.fit(train_texts, train_labels) # a strong baseline in three linesSparse representations (one-hot, bag of words, TF-IDF) are wide and mostly zero. Keep them in sparse matrices — never call .toarray() on a large one — and use models that handle sparsity natively: linear models, Naive Bayes, LightGBM.
Dense representations (embeddings) are compact and every dimension is used. They need less memory per document, support fast similarity search, and are not interpretable dimension by dimension.
A useful hybrid exists and is standard in production retrieval: combine a sparse keyword score (BM25, a refined TF-IDF) with a dense embedding score. Keyword search catches exact terms, names and numbers that embeddings blur; embeddings catch paraphrase that keywords miss. Together they beat either alone, which is why "hybrid search" is the default recommendation for RAG systems.
Is TF-IDF outdated? No. It is fast, interpretable, needs no GPU, and remains competitive on topic classification with adequate labels.
What is BM25? A refinement of TF-IDF with term-frequency saturation and document-length normalisation. The standard keyword retrieval function, and better than plain TF-IDF for search.
Should I average word embeddings for a sentence? It works as a baseline and loses word order. A sentence encoder is materially better.
How do I pick the vocabulary size? Via min_df — drop terms appearing in fewer than three to five documents. That usually removes most of the vocabulary and almost none of the signal.
Do I still need stop-word removal with TF-IDF? Rarely — IDF already suppresses ubiquitous words.
Which encoding for a language model? Subword tokenisation into learned embeddings, contextualised by the model itself. The earlier techniques do not apply.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Sparse encodings are a progression of fixes: one-hot removes label encoding's fake ordering, bag-of-words adds document structure, TF-IDF suppresses noise words. What none of them can do is say that "cat" and "kitten" are related — for that you need embeddings , the subject of the next module.
What does this module say about “A Ladder of Encodings”?
Once you accept that text must become numbers, the question is which numbers. The four classic answers form a ladder — each rung keeps more information or removes more distortion than the one below it.
What does this module say about “Label Encoding & One-Hot”?
Label encoding assigns each vocabulary word an integer. It's maximally compact, but the integers imply a fake ordering — the model may conclude that word #7 is "more" than word #3. One-hot encoding removes that lie by giving each word its own dimension: a vector of zeros with a single 1.
Once you accept that text must become numbers, the question is which numbers. The four classic answers form a ladder — each rung keeps more information or removes more distortion than the one below it.