TF-IDF

Where it falls short

No length normalisation by default. A long document contains more of everything, so raw TF favours it. Dividing by length overcorrects and favours very short ones. BM25's b parameter exists to tune between the two.

Unbounded term frequency. TF keeps growing with repetition. BM25 saturates it, so the tenth occurrence adds far less than the second — which matches how relevance actually behaves.

No understanding of meaning. "car" and "automobile" are unrelated terms. That is what dense embeddings fix, and why modern retrieval runs both and fuses the results — see hybrid search.

Why it still matters in a RAG system

Exact terms still win on identifiers, error codes, product names and rare jargon — precisely the queries where an embedding model has seen too little to place the token meaningfully. A vector-only pipeline reliably fails on "error TS2345".

TF-IDF is also the honest baseline. If a dense retriever cannot beat it on your evaluation set, the problem is the embedding model or the chunking, not the ranking.

Working one score out by hand

Take a four-document corpus where cat appears in two documents and the in all four. For the query cat:

idf(cat) = log(4 / 2) = 0.69
idf(the) = log(4 / 4) = 0.00

A document containing cat twice scores (1 + log 2) × 0.69 = 1.17. A document containing it once scores 0.69. And a document containing the ten times still scores zero for that term, because anything multiplied by zero is zero.

That last line is the one worth internalising. The stopword is not filtered, thresholded or special-cased anywhere. It is removed because the logarithm of one is zero, which is a much more satisfying reason than a hand-maintained list, and it adapts automatically to a corpus where patient or invoice is effectively a stopword.

The variants you will actually meet

Sublinear tf. 1 + log(count) rather than the raw count, which is what the visualisation's damping toggle switches. Almost always on.

Smoothed idf. log(1 + N/df) or log((N+1)/(df+1)) + 1, which scikit-learn uses by default. It avoids a zero weight and a division by zero for unseen terms, and it means scikit-learn's numbers will not match a textbook's.

L2 normalisation. Each document vector scaled to unit length, so cosine similarity between documents is a dot product and long documents stop winning by having more of everything.

When someone says two TF-IDF implementations disagree, it is nearly always one of these three, not a bug.

Where it sits in a modern stack

TF-IDF is rarely the ranker any more — BM25 supersedes it by saturating term frequency and normalising length properly — but it is still the thing to reach for in three situations.

As a baseline. If a dense retriever cannot beat TF-IDF on your evaluation set, the problem is the embedding model, the chunking or the evaluation set itself. It costs minutes to run and it has saved a great many people from tuning the wrong thing.

As a feature. TF-IDF vectors feed classical classifiers — spam filtering, topic labelling, near-duplicate detection — where a sparse interpretable representation beats a dense one and trains in seconds.

As the explanation. Every score decomposes into per-term contributions, so you can say exactly why a document ranked where it did. No dense retriever can do that, and in regulated settings it is sometimes the deciding factor.

Weighting words by how informative they are

Counting words treats "the" and "encephalopathy" as equally interesting. TF-IDF corrects that by multiplying two terms:

TF-IDF(t, d) = TF(t, d) × log(N / DF(t))

Term frequency — how often the term appears in this document. More mentions, more relevance.

Inverse document frequency — the logarithm of the total number of documents divided by how many contain the term. Rare terms score high; ubiquitous terms score near zero.

Worked through with 1,000 documents:

A term in every document contributes exactly nothing, whatever its frequency. That is automatic stop-word removal, derived from the corpus rather than from a list — and it is the single most useful property of the method.

Why the logarithm

Without it, a term in 1 document out of 1,000 would get a weight of 1,000 and a term in 10 would get 100 — a tenfold difference that vastly overstates how much more informative the first is.

The logarithm compresses that: 6.9 against 4.6. Rare terms still win, and not by an absurd margin.

The same reasoning motivates sublinear term frequency: using 1 + log(TF) instead of the raw count, because a term appearing 50 times is not 50 times more relevant than one appearing once. Scikit-learn exposes this as sublinear_tf=True, and it usually helps.

Two more refinements that matter in practice:

L2 normalisation of each document's vector, so long documents do not dominate simply by containing more words. Applied by default in most implementations.

Smoothing the IDF denominator (log(1 + N/DF)) so a term appearing in no documents does not divide by zero.

Building a working baseline

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(
    TfidfVectorizer(
        ngram_range=(1, 2),      # unigrams and bigrams
        min_df=3,                # drop terms in fewer than 3 documents
        max_features=50_000,
        sublinear_tf=True,
        stop_words=None,         # IDF already suppresses common words
    ),
    LogisticRegression(max_iter=1000),
)
model.fit(train_texts, train_labels)

That is a strong text classifier in eight lines. It trains in seconds on tens of thousands of documents, runs on a CPU, and is fully interpretable — the model's coefficients map directly to words, so you can read what it learned.

min_df is doing the heavy lifting. Dropping terms that appear in fewer than three documents typically removes most of the vocabulary and almost none of the signal, because the long tail is mostly typos and one-off tokens.

ngram_range=(1, 2) adds bigrams, which is what lets the model represent "not good" — the single most valuable addition for sentiment tasks.

Counting words, then discounting the common ones

TF-IDF is two ideas multiplied together, and both of them are visible in a handful of documents. This builds the matrix by hand, shows what each half contributes, and finds the case where the whole thing collapses.

import numpy as np

DOCS = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "the cat and the dog were friends",
    "the quantum entanglement links two particles",
    "the quantum cat is both alive and dead",
]
tok = [d.split() for d in DOCS]
vocab = sorted({w for d in tok for w in d})
V, N = len(vocab), len(DOCS)
vi = {w: i for i, w in enumerate(vocab)}

print("%d documents, %d distinct words." % (N, V))
for i, d in enumerate(DOCS):
    print("   doc %d: %s" % (i, d))
print()

tf = np.zeros((N, V))
for i, d in enumerate(tok):
    for w in d:
        tf[i, vi[w]] += 1

print("TERM FREQUENCY -- how often each word appears in each document.")
show = ["the", "cat", "dog", "quantum", "sat", "friends"]
print("%-14s %s" % ("word", "  ".join("doc %d" % i for i in range(N))))
for w in show:
    print("%-14s %s" % (w, "  ".join("%5.0f" % tf[i, vi[w]] for i in range(N))))
print("   'the' is the most common word in the collection and tells you")
print("   nothing about any document. that is the problem TF alone has,")
print("   and it is why raw counts are a bad retrieval score.")
print()

df = (tf > 0).sum(axis=0)
idf = np.log(N / df)                          # the textbook form
idf_s = np.log((1 + N) / (1 + df)) + 1.0      # the form scikit-learn uses
print("INVERSE DOCUMENT FREQUENCY -- how many documents contain the word,")
print("inverted, so rare words score high:")
print("%-14s %10s %14s %14s %14s"
      % ("word", "in N docs", "N/df", "log(N/df)", "sklearn idf"))
for w in show:
    d = int(df[vi[w]])
    print("%-14s %10d %14.4f %14.4f %14.4f"
          % (w, d, N / float(d), idf[vi[w]], idf_s[vi[w]]))
print("   'the' appears in all %d documents, so log(N/df) = log(1) = 0" % N)
print("   EXACTLY. it contributes nothing to any score, and no stopword")
print("   list was needed to arrive at that.")
print("   'friends' appears in %d, so it scores highest." % int(df[vi["friends"]]))
print()
print("   NOTE THE LAST COLUMN, because it will surprise you in practice.")
print("   scikit-learn smooths: log((1+N)/(1+df)) + 1. that +1 is a FLOOR")
print("   -- it makes the smallest possible idf %.1f, not 0 -- so a word" % 1.0)
print("   in every document still contributes %.4f x its count."
      % idf_s[vi["the"]])
print("   in a large corpus that hardly matters, because rare words score")
print("   log(N) + 1 and dwarf it. in a SMALL corpus it matters a lot: a")
print("   word appearing twice in a short document can outrank a genuinely")
print("   rare one, purely because of the floor. this is the single most")
print("   common surprise when TF-IDF is tried on a few dozen documents,")
print("   and it is a property of the smoothing, not of TF-IDF.")
print("   the log matters too: without it, a word in 1 document of a")
print("   million would outweigh a word in 10 by a factor of 100,000. the")
print("   log turns that into a factor of about %.1f, a difference you can"
      % (np.log(1000000.0 / 1) / np.log(1000000.0 / 10)))
print("   still act on without it swamping everything else.")
print()

tfidf = tf * idf
norms = np.linalg.norm(tfidf, axis=1, keepdims=True)
tfidf_n = tfidf / np.where(norms == 0, 1, norms)

print("TF-IDF = TF x IDF, then normalise each row to unit length.")
print("the top-scoring words in each document:")
for i in range(N):
    top = np.argsort(-tfidf_n[i])[:3]
    print("   doc %d: %s" % (i, "   ".join("%s %.3f" % (vocab[j], tfidf_n[i, j])
                                           for j in top)))
print("   'the' does not appear in any list, and no stopword file was")
print("   involved. that is the point: TF-IDF DISCOVERS which words carry")
print("   no information, from the collection itself, so the same code")
print("   adapts to a legal corpus or a medical one without being told")
print("   which words to ignore.")
print("   for comparison, the same documents scored with scikit-learn's")
print("   smoothed idf:")
tfidf_s = tf * idf_s
ns = np.linalg.norm(tfidf_s, axis=1, keepdims=True)
tfidf_sn = tfidf_s / np.where(ns == 0, 1, ns)
for i in range(3):
    top = np.argsort(-tfidf_sn[i])[:3]
    print("      doc %d: %s"
          % (i, "   ".join("%s %.3f" % (vocab[j], tfidf_sn[i, j]) for j in top)))
ranks = [int(np.where(np.argsort(-tfidf_sn[i]) == vi["the"])[0][0]) + 1
         for i in range(3)]
print("   'the' is back, ranking %s in those three documents -- and it"
      % ", ".join(str(r) for r in ranks))
print("   ranks NOWHERE under log(N/df), where it scores an exact 0.")
print("   the floor is doing precisely what it was designed to do, which")
print("   on five short documents is the wrong thing. on a million it")
print("   would be invisible. worth knowing which regime you are in.")
print()

print("WHY NORMALISE. a long document has bigger counts, so without it")
print("length alone would decide the ranking:")
print("%-10s %10s %16s %16s" % ("doc", "words", "raw tf-idf norm", "after norm"))
for i in range(N):
    print("%-10d %10d %16.4f %16.4f"
          % (i, len(tok[i]), np.linalg.norm(tfidf[i]),
             np.linalg.norm(tfidf_n[i])))
print("   every row is now length 1, so a dot product between two rows is")
print("   exactly their cosine similarity.")
print()

def query(q):
    v = np.zeros(V)
    for w in q.split():
        if w in vi:
            v[vi[w]] += 1
    v = v * idf
    n = np.linalg.norm(v)
    return v / n if n else v

print("RETRIEVAL. score a query against every document:")
for q in ("quantum cat", "the the the", "dog friends"):
    qv = query(q)
    scores = tfidf_n @ qv
    order = np.argsort(-scores)
    print("   query: %r" % q)
    for r in order[:3]:
        print("      %.4f  doc %d: %s" % (scores[r], r, DOCS[r]))
print("   the middle query is the interesting one. 'the the the' is three")
print("   copies of a word that appears in all %d documents, so its idf is" % N)
print("   0, so the query vector is all zeros and every score is exactly")
print("   %.4f. the query carries no information and the arithmetic says"
      % float((tfidf_n @ query("the the the")).max()))
print("   so, without a rule anywhere that mentions 'the'.")
print()

print("NOW WHERE IT FAILS, which is the part worth remembering:")
print("   four single-word queries, in synonym pairs:")
print("%-14s %12s   %s" % ("query", "best score", "best document"))
for q in ("cat", "feline", "dog", "canine"):
    qv = query(q)
    scores = tfidf_n @ qv
    bi = int(np.argmax(scores))
    print("%-14s %12.4f   %s"
          % (q, scores[bi],
             DOCS[bi] if scores[bi] > 0 else "(nothing matches at all)"))
print("   'feline' means the same thing as 'cat' and shares no characters")
print("   with it, so it scores exactly %.1f against every document in the"
      % 0.0)
print("   collection, including the %d that actually mention a cat."
      % int((tf[:, vi["cat"]] > 0).sum()))
print("   every word is an independent dimension with no relationship to")
print("   any other -- the model has no idea that cat and feline, or dog")
print("   and canine, are connected.")
print("   that single limitation is the entire reason dense embeddings")
print("   exist. they map cat and feline to nearby VECTORS, so similarity")
print("   survives a change of wording.")
print()
print("   and the reason TF-IDF and BM25 are still in production anyway:")
print("%-30s %-22s %s" % ("", "TF-IDF / BM25", "dense embeddings"))
for row in (("exact term match", "perfect", "approximate"),
            ("synonyms", "no", "yes"),
            ("rare words, product codes", "excellent", "often poor"),
            ("needs training", "no", "yes"),
            ("cost to index", "cheap", "a GPU pass per document"),
            ("explains its score", "yes, per term", "not really")):
    print("%-30s %-22s %s" % row)
print("   the two fail in different places, which is why hybrid search --")
print("   run both, fuse the rankings -- beats either one alone and is")
print("   what most serious retrieval systems actually ship.")

Things to try

1. Turn idf off and query the. Every document scores, and the ranking is driven by a word that distinguishes nothing.
2. Query the cat with idf on. Only cat contributes — the stopword cancels itself without a stopword list.
3. Turn tf damping off and watch the document that repeats a term run away with the score. That runaway is what BM25's saturation fixes.

What to remember

TF-IDF weights a term by how often it appears here times how rare it is everywhere. A term in every document has an idf of zero and drops out by arithmetic rather than by a list. It knows nothing about meaning, which is why a synonym scores zero and why dense retrieval runs alongside it.

What it cannot do

No sense of similar words. "Excellent" and "superb" are unrelated columns. A model trained on documents containing one learns nothing about documents containing the other. This is the fundamental limitation, and it is what embeddings fix.

No word order beyond n-grams. Bigrams capture adjacent pairs; anything longer-range is invisible.

No handling of unseen vocabulary. A term absent from the training corpus has no column, so it contributes nothing at prediction time.

Very high dimensionality. Fifty thousand features, almost all zero per document. Manageable with sparse matrices and sparse-aware models, and it rules out anything that requires dense input.

That table is the argument for using both, which is exactly what hybrid retrieval does.

Where it is still the right tool

As the first baseline for any text classification task. It is fast enough to try in five minutes, and it frequently comes within a few points of a fine-tuned transformer on topic classification with adequate labels. Not measuring it means not knowing whether the transformer was worth it.

Keyword retrieval, usually in its refined form BM25, which adds term-frequency saturation and document-length normalisation. Every search engine is built on this.

Keyness analysis — comparing two corpora to find the terms that distinguish them. Far more informative than a word cloud.

Interpretable models in regulated settings, where "the model weighted these words" is a required explanation.

Low-resource environments — no GPU, no model download, no inference latency.

BM25, briefly

BM25 is TF-IDF with two corrections, and it is what you should use for retrieval rather than plain TF-IDF:

Term-frequency saturation. The contribution of repeated occurrences flattens, controlled by k₁ (around 1.2). A document mentioning a term twenty times is not twenty times more relevant.

Document-length normalisation relative to the corpus average, controlled by b (around 0.75). A long document will contain most query terms by chance.

Both corrections address ways raw TF-IDF over-rewards documents. For classification features TF-IDF is fine; for ranking search results, BM25 is meaningfully better.

Questions people ask

Do I need stop-word removal? Not really — IDF drives ubiquitous terms to zero. Removing them shrinks the vocabulary, which is the only benefit.

Should I stem or lemmatise? For TF-IDF, yes — it pools evidence across word forms and shrinks the vocabulary. Never for transformers.

What min_df should I use? 2–5 for most corpora. It removes the long tail of typos and one-offs.

Is TF-IDF outdated? No. It is a strong baseline, the basis of keyword retrieval, and interpretable in ways embeddings are not.

Should I use it with embeddings? Yes — hybrid retrieval combining BM25 with dense search beats either alone on most real corpora.

Why is my matrix enormous? Bigrams without min_df or max_features. Cap both.

Recap in one screen

• TF-IDF multiplies how often a term appears here by how rare it is across the corpus.
• A term in every document gets an IDF of zero — automatic, corpus-derived stop-word removal.
• The logarithm stops rare terms getting absurd weights; sublinear TF does the same for repetition.
• TF-IDF plus logistic regression is a fast, interpretable baseline worth measuring before anything heavier.
• It cannot relate similar words, which is what embeddings add — and BM25 is the version to use for retrieval.