Text Encoding Techniques in NLP
Four classic ways to turn a corpus into numbers — computed live from a corpus you can edit.
Four classic ways to turn a corpus into numbers — computed live from a corpus you can edit.
Label encoding, one-hot, bag-of-words, and TF-IDF — each fixes a weakness of the one before it.
Once you accept that text must become numbers, the question is which numbers. The four classic answers form a ladder — each rung keeps more information or removes more distortion than the one below it.
Label encoding assigns each vocabulary word an integer. It's maximally compact, but the integers imply a fake ordering — the model may conclude that word #7 is "more" than word #3. One-hot encoding removes that lie by giving each word its own dimension: a vector of zeros with a single 1. Honest, but the vectors are as wide as the vocabulary, almost entirely zeros, and every pair of words is exactly equidistant — no notion of similarity survives.
Bag of words moves from single words to whole documents: each document becomes a vector of word counts. Suddenly documents can be compared — but frequent filler words like "the" dominate the counts while carrying no meaning. TF-IDF fixes that with a reweighting: term frequency (how often a word appears in this document) multiplied by inverse document frequency (how rare it is across documents). Words that appear everywhere score near zero; words distinctive to one document light up.
tf-idf(w, d) = tf(w, d) × log(N / df(w))
Sparse encodings are a progression of fixes: one-hot removes label encoding's fake ordering, bag-of-words adds document structure, TF-IDF suppresses noise words. What none of them can do is say that "cat" and "kitten" are related — for that you need embeddings, the subject of the next module.
Once you accept that text must become numbers, the question is which numbers. The four classic answers form a ladder — each rung keeps more information or removes more distortion than the one below it.