TF-IDF

Two numbers multiplied. Term frequency rewards a word for appearing often in this document; inverse document frequency punishes it for appearing in every document. The product is high only for words that are frequent here and rare elsewhere — which is a workable definition of "what this document is about".

Overview

Two numbers multiplied.

In this guide
  1. The two halves
  2. Where it falls short
  3. Why it still matters in a RAG system
  4. Working one score out by hand
  5. The variants you will really meet
  6. Where it sits in a modern stack
  7. Weighting words by how informative they are
  8. Why the logarithm
  9. Building a working baseline
  10. Counting words, then discounting the common ones
  11. Things to try with TF-IDF
  12. What to recall
  13. What it cannot do
  14. Where it is still the right tool
  15. BM25, briefly
  16. Questions people ask about TF-IDF
  17. TF-IDF: recap in one screen

Parameters

Visualisation

—

Readout

What to watch

  • A term in every document scores zero — the log sees to that.
  • One occurrence of a rare word beats two of a common one.
  • No stopword list is needed; the maths removes them.

TF-IDF: A Practical Guide

How does TF-IDF decide which words in a document actually matter?

The two halves

Term count is how often a term occurs in a document, often damped. A word appearing twenty times is not twenty times as relevant. So implementations take a logarithm or normalise by document length.

Inverse document count is log(N / df), where df is the number of documents holding the term. When a term is in every document that ratio is 1 and the log is 0. So the term adds nothing at all.

That zero is the elegant part: stopwords are removed by the arithmetic rather than by a list you have to maintain per language.

The product of TF and IDF gives each term a score that is high when the term appears often in the current document and rarely across the corpus. A term that appears in every document gets an IDF of zero and adds nothing, no matter its count. A term that appears in only one document gets a high IDF and rules that document's representation. This is why TF-IDF works well for document retrieval. The terms that distinguish one document from the rest are exactly the terms that carry the most weight.

In scikit-learn, TfidfVectorizer computes TF-IDF in one step: it tokenises the text, counts term counts, applies IDF weights, and normalises each document vector to unit length. The normalisation matters because without it, longer documents would have higher scores simply by holding more words. The result is a sparse matrix where each row is a document and each column is a term from the vocabulary.

TF-IDF is the baseline for text classification, document similarity, keyword extraction and simple search engines. It needs no training data, no GPU. And no neural network — just a corpus and a few seconds of work. For many practical tasks it performs well enough that a more complex model is not worth the cost. Where it fails is synonymy ("car" and "automobile" are treated as unrelated terms) and polysemy ("bank" the institution and "bank" the riverbank share a vector dimension). Embedding-based methods solve both problems by mapping words to dense vectors rather than sparse term counts. But they trade interpretability and ease for that improvement.

Where it falls short

No length normalisation by default. A long document holds more of all of it. So raw TF favours it. Dividing by length overcorrects and favours very short ones. BM25's b parameter exists to tune between the two.

Unbounded term count. TF keeps growing with repetition. BM25 saturates it, so the tenth instance adds far less than the second — which matches how relevance really behaves.

No understanding of meaning. "car" and "automobile" are unrelated terms. That is what dense embeddings fix, and why modern retrieval runs both and fuses the results. See hybrid search.

Why it still matters in a RAG system

Exact terms still win on identifiers, error codes, product names and rare jargon. Exactly the queries where an embedding model has seen too little to place the token meaningfully. A vector-only pipeline reliably fails on "error TS2345".

TF-IDF is also the honest baseline. If a dense retriever cannot beat it on your evaluation set, the problem is the embedding model or the chunking, not the ranking.

Working one score out by hand

Take a four-document corpus where cat appears in two documents and the in all four. For the query cat:

idf(cat) = log(4 / 2) = 0.69
idf(the) = log(4 / 4) = 0.00

A document holding cat twice scores (1 + log 2) × 0.69 = 1.17. A document holding it once scores 0.69. And a document holding the ten times still scores zero for that term, because anything times by zero is zero.

That last line is the one worth internalising. The stopword is not filtered, thresholded or special-cased anywhere. It is removed because the logarithm of one is zero. This is a much more meeting reason than a hand-maintained list. And it adapts by itself to a corpus where patient or invoice is in effect a stopword.

The variants you will really meet

Sublinear tf. 1 + log(count) rather than the raw count, which is what the visualisation's damping toggle switches. Almost always on.

Smoothed idf. log(1 + N/df) or log((N+1)/(df+1)) + 1, which scikit-learn uses by default. It avoids a zero weight and a division by zero for unseen terms. And it means scikit-learn's numbers will not match a textbook's.

L2 normalisation. Each document vector scaled to unit length, so cosine similarity between documents is a dot product and long documents stop winning by having more of all of it.

When someone says two TF-IDF implementations disagree, it is nearly always one of these three, not a bug.

Where it sits in a modern stack

TF-IDF is rarely the ranker any more. BM25 supersedes it by saturating term count and normalising length properly. But it is still the thing to reach for in three situations.

As a baseline. If a dense retriever cannot beat TF-IDF on your evaluation set, the problem is the embedding model, the chunking or the evaluation set itself. It costs minutes to run and it has saved a great many people from tuning the wrong thing.

As a feature. TF-IDF vectors feed classical classifiers. Spam filtering, topic labelling, near-copy detection. Where a sparse interpretable representation beats a dense one and trains in seconds.

As the reason. Every score decomposes into per-term contributions, so you can say exactly why a document ranked where it did. No dense retriever can do that, and in regulated settings it is sometimes the deciding factor.

Weighting words by how informative they are

Counting words treats "the" and "encephalopathy" as equally notable. TF-IDF corrects that by multiplying two terms:

TF-IDF(t, d) = TF(t, d) × log(N / DF(t))

Term count — how often the term appears in this document. More mentions, more relevance.

Inverse document count — the logarithm of the total number of documents divided by how many hold the term. Rare terms score high; ubiquitous terms score near zero.

Worked through with 1,000 documents:

TermDocuments containing itIDF
"the"1,000log(1) = 0
"policy"200log(5) = 1.61
"parental"30log(33) = 3.50
"encephalopathy"2log(500) = 6.21

A term in every document adds exactly nothing, any its count. That is automatic stop-word cutting, derived from the corpus rather than from a list. And it is the single most useful property of the method.

Why the logarithm

Without it, a term in 1 document out of 1,000 would get a weight of 1,000 and a term in 10 would get 100. A tenfold difference that vastly overstates how much more informative the first is.

The logarithm compresses that: 6.9 against 4.6. Rare terms still win, and not by an absurd margin.

The same reasoning motivates sublinear term count: using 1 + log(TF) instead of the raw count,. This is because a term appearing 50 times is not 50 times more relevant than one appearing once. Scikit-learn exposes this as sublinear_tf=True, and it often helps.

Two more refinements that matter in practice:

L2 normalisation of each document's vector, so long documents do not dominate simply by holding more words. Applied by default in most implementations.

Smoothing the IDF denominator (log(1 + N/DF)) so a term appearing in no documents does not divide by zero.

Building a working baseline

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(
    TfidfVectorizer(
        ngram_range=(1, 2),      # unigrams and bigrams
        min_df=3,                # drop terms in fewer than 3 documents
        max_features=50_000,
        sublinear_tf=True,
        stop_words=None,         # IDF already suppresses common words
    ),
    LogisticRegression(max_iter=1000),
)
model.fit(train_texts, train_labels)

That is a strong text classifier in eight lines. It trains in seconds on tens of thousands of documents, runs on a CPU, and is fully interpretable. The model's coefficients map right to words, so you can read what it learned.

min_df is doing the heavy lifting. Dropping terms that appear in fewer than three documents often removes most of the vocabulary and almost none of the signal,. This is because the long tail is mostly typos and one-off tokens.

ngram_range=(1, 2) adds bigrams, which is what lets the model represent "not good". The single most valuable addition for sentiment tasks.

Counting words, then discounting the common ones

TF-IDF is two ideas times together, and both of them are visible in a handful of documents. This builds the matrix by hand, shows what each half adds, and finds the case where the whole thing collapses.

example_01.pyNumPy
Output

The next block goes on with the left demonstrations.

example_02.pyNumPy
Output

The next block goes on with the left demonstrations.

example_03.pyNumPy
Output

The next block goes on with the left demonstrations.

example_04.pyNumPy
Output

The next block goes on with the left demonstrations.

example_05.pyNumPy
Output

Things to try with TF-IDF

  1. Turn idf off and query the. Every document scores, and the ranking is driven by a word that distinguishes nothing.
  2. Query the cat with idf on. Only cat adds — the stopword cancels itself without a stopword list.
  3. Turn tf damping off and watch the document that repeats a term run away with the score. That runaway is what BM25's saturation fixes.

What to recall

TF-IDF weights a term by how often it appears here times how rare it is everywhere. A term in every document has an idf of zero and drops out by arithmetic rather than by a list. It knows nothing about meaning, which is why a synonym scores zero and why dense retrieval runs alongside it.

What it cannot do

No sense of similar words. "Excellent" and "superb" are unrelated columns. A model trained on documents holding one learns nothing about documents holding the other. This is the basic limitation, and it is what embeddings fix.

No word order beyond n-grams. Bigrams capture adjacent pairs; anything longer-range is invisible.

No handling of unseen vocabulary. A term absent from the training corpus has no column. So it adds nothing at prediction time.

Very high dimensionality. Fifty thousand features, almost all zero per document. Manageable with sparse matrices and sparse-aware models, and it rules out anything that needs dense input.

 TF-IDFEmbeddings
Similar words relatedNoYes
DimensionalityTens of thousands, sparseHundreds, dense
InterpretableYes — coefficients are wordsNo
Needs a model to computeNoYes
Exact term matchingExcellentWeak
Handles paraphraseNoYes

That table is the argument for using both, which is exactly what hybrid retrieval does.

Where it is still the right tool

As the first baseline for any text classification task. It is fast enough to try in five minutes. And it often comes within a few points of a fine-tuned transformer on topic classification with adequate labels. Not measuring it means not knowing whether the transformer was worth it.

Keyword retrieval, often in its refined form BM25, which adds term-count saturation and document-length normalisation. Every search engine is built on this.

Keyness review — comparing two corpora to find the terms that distinguish them. Far more informative than a word cloud.

Interpretable models in regulated settings, where "the model weighted these words" is a needed reason.

Low-resource environments — no GPU, no model download, no inference latency.

BM25, briefly

BM25 is TF-IDF with two corrections, and it is what you should use for retrieval rather than plain TF-IDF:

Term-count saturation. The contribution of repeated instances flattens, controlled by k₁ (around 1.2). A document mentioning a term twenty times is not twenty times more relevant.

Document-length normalisation relative to the corpus average, controlled by b (around 0.75). A long document will hold most query terms by chance.

Both corrections address ways raw TF-IDF over-rewards documents. For classification features TF-IDF is fine; for ranking search results, BM25 is meaningfully better.

Questions people ask about TF-IDF

Do I need stop-word cutting? Not really — IDF drives ubiquitous terms to zero. Removing them shrinks the vocabulary, which is the only gain.

Should I stem or lemmatise? For TF-IDF, yes — it pools evidence across word forms and shrinks the vocabulary. Never for transformers.

What min_df should I use? 2–5 for most corpora. It removes the long tail of typos and one-offs.

Is TF-IDF outdated? No. It is a strong baseline, the basis of keyword retrieval, and interpretable in ways embeddings are not.

Should I use it with embeddings? Yes — hybrid retrieval combining BM25 with dense search beats either alone on most real corpora.

Why is my matrix enormous? Bigrams without min_df or max_features. Cap both.

TF-IDF: recap in one screen

  • TF-IDF multiplies how often a term appears here by how rare it is across the corpus.
  • A term in every document gets an IDF of zero. Automatic, corpus-derived stop-word cutting.
  • The logarithm stops rare terms getting absurd weights; sublinear TF does the same for repetition.
  • TF-IDF plus logistic regression is a fast, interpretable baseline worth measuring before anything heavier.
  • It cannot relate similar words, which is what embeddings add. And BM25 is the version to use for retrieval.

Recall check

0 of 3

Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.

  1. What does this module say about “The two halves”?

  2. What does this module say about “Where it falls short”?

  3. What does this module say about “Why it still matters in a RAG system”?

Cheat sheet

TF-IDF

Compute TF-IDF scores by hand, watch common words shrink and rare words rise, and see how a simple formula still powers keyword-based search.

GEN AI · vizlearn.in/gen_ai/tf_idf.html

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.

Teaching a course with this? Individual teachers can use it freely; for a whole department or programme, see institutional licences.