TF-IDF
Two numbers multiplied. Term frequency rewards a word for appearing often in this document; inverse document frequency punishes it for appearing in every document. The product is high only for words that are frequent here and rare elsewhere — which is a workable definition of "what this document is about".
Overview
The two halves
Term frequency is how often a term occurs in a document, usually damped — a word appearing twenty times is not twenty times as relevant, so implementations take a logarithm or normalise by document length.
Inverse document frequency is log(N / df), where df is the number of documents containing the term. When a term is in every document that ratio is 1 and the log is 0, so the term contributes nothing at all.
That zero is the elegant part: stopwords are removed by the arithmetic rather than by a list you have to maintain per language.
Parameters
Visualisation
—Readout
What to watch
- A term in every document scores zero — the log sees to that.
- One occurrence of a rare word beats two of a common one.
- No stopword list is needed; the maths removes them.
TF-IDF: A Practical Guide
How does TF-IDF decide which words in a document actually matter?
Where it falls short
No length normalisation by default. A long document contains more of everything, so raw TF favours it. Dividing by length overcorrects and favours very short ones. BM25's b parameter exists to tune between the two.
Unbounded term frequency. TF keeps growing with repetition. BM25 saturates it, so the tenth occurrence adds far less than the second — which matches how relevance actually behaves.
No understanding of meaning. "car" and "automobile" are unrelated terms. That is what dense embeddings fix, and why modern retrieval runs both and fuses the results — see hybrid search.
Why it still matters in a RAG system
Exact terms still win on identifiers, error codes, product names and rare jargon — precisely the queries where an embedding model has seen too little to place the token meaningfully. A vector-only pipeline reliably fails on "error TS2345".
TF-IDF is also the honest baseline. If a dense retriever cannot beat it on your evaluation set, the problem is the embedding model or the chunking, not the ranking.
Things to try
- Turn idf off and query the. Every document scores, and the ranking is driven by a word that distinguishes nothing.
- Query the cat with idf on. Only cat contributes — the stopword cancels itself without a stopword list.
- Turn tf damping off and watch the document that repeats a term run away with the score. That runaway is what BM25's saturation fixes.
What to remember
TF-IDF weights a term by how often it appears here times how rare it is everywhere. A term in every document has an idf of zero and drops out by arithmetic rather than by a list. It knows nothing about meaning, which is why a synonym scores zero and why dense retrieval runs alongside it.