If “the cat sat” appears 100 times and “the cat sat on” appears 60, then P(on | the cat, sat) = 0.6. There is no training in the gradient-descent sense — the model is a table of counts, built in one pass.
The underlying simplification is the Markov assumption: that only the last n−1 words matter. It is plainly false for language and works surprisingly well anyway.
Generated N-grams
0 ITEMS
Enter text to generate sequences
List ScrollVIEW
Tip: Use the slider to scroll through the list if it overflows.
N-gram Explainer: A Practical Guide
Predict the next word from the previous n-1 words, by counting how often that sequence appeared. Simple, fast, interpretable - and defeated by the fact that most valid sentences have never been written down.
Choosing n
n
Behaviour
1
No word order at all; smallest feature set
2
Captures negation and common phrases; usually worth it
3
Some benefit, considerably sparser
4+
Rarely justified for features; used in language models with heavy smoothing
For classification features, (1, 2) is the standard choice and the marginal gain from trigrams is usually small relative to the extra columns.
For a count-based language model, trigrams to 5-grams with Kneser-Ney smoothing was the state of the art for years, and it needed large corpora to estimate.
The sparsity wall
The counts grow catastrophically. With a 50,000-word vocabulary there are 2.5 billion possible bigrams, 1.25 × 1014 trigrams, and 3 × 1023 5-grams. No corpus covers a meaningful fraction of them.
So most n-grams have a count of zero, and a zero count means a probability of zero — the model declares a perfectly ordinary sentence impossible because it happens not to have seen one four-word span before. Worse, one zero anywhere makes the probability of the whole sentence zero.
The fixes are all forms of smoothing. Add-one (Laplace) smoothing adds a pseudocount to everything, which is simple and crude. Backoff falls back to a shorter n-gram when the longer one is unseen. Kneser-Ney, the best of the classical methods, discounts observed counts and redistributes the mass according to how many distinct contexts a word appears in — capturing that Francisco is common but only ever after San.
Counting sequences of words
An n-gram is a contiguous run of n items. For "the cat sat on the mat":
n
Name
Grams
1
Unigram
the, cat, sat, on, the, mat
2
Bigram
the cat, cat sat, sat on, on the, the mat
3
Trigram
the cat sat, cat sat on, sat on the, on the mat
Why bother beyond unigrams? Because word order carries meaning that a bag of individual words discards. "Not good" and "good not" have identical unigrams; as bigrams they are distinguishable. Sentiment classification improves measurably from bigrams for exactly this reason.
The cost is the vocabulary. With 10,000 distinct words there are up to 100 million possible bigrams and a trillion trigrams. Almost all are absent from any real corpus, so the feature matrix becomes enormous and extremely sparse.
That trade — more context against exponentially more features — is the whole subject.
As a language model
Before neural networks, next-word prediction was done by counting. An n-gram language model estimates:
P(wᵗ | w₁ … wᵗ₋₁) ≈ P(wᵗ | wᵗ₋ₙ₊₁ … wᵗ₋₁)
The approximation is the Markov assumption: only the previous n−1 words matter. For a trigram model, the probability of the next word depends on the two before it and nothing further back.
Estimation is division: count how often "sat on the" was followed by "mat", divide by how often "sat on the" appeared at all.
Two problems arise immediately, and the fixes are classic:
Unseen sequences get probability zero, which zeroes the probability of the whole sentence. Smoothing fixes it — add-one (Laplace) is the simplest, and Kneser-Ney is the one that actually works well, because it accounts for how many different contexts a word appears in.
Longer n means better context and sparser counts.Backoff and interpolation handle it by falling back to shorter n-grams when the longer one has no data.
Why neural models replaced them
Two limitations are structural, not fixable by better smoothing.
No generalisation across similar words. A trigram model that has seen "the black cat" learns nothing about "the dark cat" — the two contexts are entirely separate counts. An embedding-based model places "black" and "dark" near each other, so evidence transfers.
No long-range dependency. A 5-gram model cannot connect a pronoun to a noun seven words earlier. Increasing n makes the counts too sparse to estimate long before the range becomes useful.
Neural language models solved both — distributed representations generalise across words, and recurrence or attention reaches arbitrary distances. That is the whole reason the field moved.
Interactive Exploration Guide
Start at n = 1. Set the n slider to 1 and read the output. With no context the model produces frequency-ordered nonsense.
Step up to 2 and 3. Raise n and watch local fluency appear. Pairs and triples look like real language, even though the sentence as a whole still drifts.
Push to 6. Set n to 6. Now most contexts have been seen once or not at all, so the model either repeats the training text verbatim or has nothing to offer — overfitting and sparsity in the same picture.
Scroll through the corpus. Use the view slider and watch which sequences have high counts. Almost all the probability mass sits on a small number of common patterns; the tail is enormous and nearly empty.
Traps worth knowing
No smoothing. Any unseen n-gram makes the whole sentence probability zero. Smoothing is not optional.
Multiplying raw probabilities. Multiplying hundreds of small numbers underflows to zero in floating point. Sum log-probabilities instead.
Raising n to fix quality. It usually makes things worse by increasing sparsity. Better smoothing beats a larger n.
Forgetting sentence boundaries. Without explicit start and end tokens the model cannot represent which words begin or end a sentence.
In one line
An n-gram model estimates the next word by counting how often each continuation followed the previous n−1 words, which makes it fast, interpretable, and trainable in a single pass. It is defeated by sparsity: the number of possible n-grams grows exponentially with n, so most sequences are never observed and smoothing is mandatory. Neural language models replaced it precisely because embeddings let them generalise across similar contexts rather than requiring each one to have been seen.
Where n-grams are still used
They have not disappeared, because counting is cheap, interpretable and needs no training:
Features for classical classifiers. TF-IDF over unigrams and bigrams, plus logistic regression, remains a strong baseline for text classification — and it trains in seconds.
Character n-grams for language identification, authorship attribution and handling misspellings. Character 3-grams are remarkably effective at identifying a language from a short string.
Autocomplete and query suggestion, where a count-based model over query logs is fast and adequate.
Spelling correction and fuzzy matching, using character n-gram overlap.
Evaluation metrics. BLEU and ROUGE, still standard for translation and summarisation, are n-gram overlap measures.
Plagiarism and duplicate detection, via shared n-gram fingerprints.
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer(ngram_range=(1, 2), # unigrams and bigrams
min_df=3, # drop very rare grams
max_features=50_000,
sublinear_tf=True)
X = vec.fit_transform(docs)
min_df and max_features are what keep this tractable — without them, bigrams alone can produce millions of columns, almost all appearing once.
For character n-grams, analyzer="char_wb", ngram_range=(3, 5) is a strong setting for noisy or multilingual text.
Count, predict, and watch it fall apart
An n-gram model is a table of counts. Building one shows both why it works at all and the two walls it hits -- sparsity and no generalisation.
example_01.pyNumPy
import numpy as np
from collections import Counter, defaultdict
corpus = (
"the cat sat on the mat the cat ate the fish the dog sat on the rug "
"the dog ate the bone the cat sat on the rug the bird sat on the branch "
"the cat chased the mouse the dog chased the cat the mouse ate the cheese"
).split()
print("a corpus of %d words, %d distinct." % (len(corpus), len(set(corpus))))
print()
def ngrams(tokens, n):
return [tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)]
print("%6s %14s %16s %20s" % ("n", "distinct n-grams", "possible", "coverage"))
V = len(set(corpus))
for n in (1, 2, 3, 4):
seen = len(set(ngrams(corpus, n)))
print("%6d %14d %16s %19.6f%%"
% (n, seen, "{:,}".format(V ** n), 100 * seen / V ** n))
print()
print("that last column is the sparsity problem, and it is not a small one.")
print("the number of possible n-grams grows as V^n while the number you have")
print("ever SEEN grows at best linearly with your corpus. by n=4 you have")
print("observed a rounding error's worth of the space.")
print()
bigrams = defaultdict(Counter)
for a, b in ngrams(corpus, 2):
bigrams[a][b] += 1
print("the model itself is a table. what follows 'the'?")
total = sum(bigrams["the"].values())
for w, c in bigrams["the"].most_common(6):
print(" %-8s %3d / %d = %.4f %s"
% (w, c, total, c / total, "#" * int(40 * c / total)))
print()
def generate(start, n_words, seed=0):
rng = np.random.default_rng(seed)
out = [start]
for _ in range(n_words - 1):
opts = bigrams[out[-1]]
if not opts:
break
words = list(opts)
probs = np.array([opts[w] for w in words], float)
out.append(words[rng.choice(len(words), p=probs / probs.sum())])
return " ".join(out)
print("sample from it and you get text that is locally plausible:")
for seed in range(4):
print(" %s" % generate("the", 10, seed))
print(" each pair of adjacent words is real. the sentences are not.")
print()
trigrams = defaultdict(Counter)
for a, b, c in ngrams(corpus, 3):
trigrams[(a, b)][c] += 1
print("a trigram model has more context and sounds better:")
def gen3(start, n_words, seed=0):
rng = np.random.default_rng(seed)
out = list(start)
for _ in range(n_words - 2):
opts = trigrams[tuple(out[-2:])]
if not opts:
break
ws = list(opts)
ps = np.array([opts[w] for w in ws], float)
out.append(ws[rng.choice(len(ws), p=ps / ps.sum())])
return " ".join(out)
for seed in range(3):
print(" %s" % gen3(("the", "cat"), 10, seed))
print(" but look at how often it simply runs out of options:")
dead = sum(1 for k in trigrams if len(trigrams[k]) == 1)
print(" %d of %d trigram contexts have exactly ONE continuation."
% (dead, len(trigrams)))
print(" the model is not predicting there, it is reciting.")
print()
print("WALL 1 -- unseen n-grams have probability zero:")
for test in (("the", "cat"), ("the", "elephant"), ("cat", "sat")):
c = bigrams[test[0]][test[1]]
tot = sum(bigrams[test[0]].values())
print(" P(%-8s | %-4s) = %d/%d = %.4f%s"
% (test[1], test[0], c, tot, c / tot if tot else 0,
" <- zero. the whole sentence is now impossible." if not c else ""))
print()
print(" one unseen pair makes an entire sentence's probability zero, which")
print(" is why smoothing exists. add-one is the simplest:")
for test in (("the", "elephant"),):
c = bigrams[test[0]][test[1]]
tot = sum(bigrams[test[0]].values())
print(" unsmoothed : %d / %d = %.6f" % (c, tot, c / tot))
print(" add-one : (%d+1) / (%d+%d) = %.6f"
% (c, tot, V, (c + 1) / (tot + V)))
print(" Kneser-Ney and backoff are better versions of the same repair:")
print(" move a little probability mass from what you saw to what you did not.")
print()
print("WALL 2 -- and this one no smoothing fixes. the model has no idea that")
print("two words are related:")
tri_counts = Counter(ngrams(corpus, 3))
def times(n):
return "%d time%s" % (n, "" if n == 1 else "s")
print(" it has seen 'the cat sat' %s and 'the dog sat' %s."
% (times(tri_counts[("the", "cat", "sat")]),
times(tri_counts[("the", "dog", "sat")])))
print(" it has never seen 'the bird ate'. what does it predict?")
print(" P(ate | the bird) = %.4f"
% (trigrams[("the", "bird")]["ate"]
/ max(sum(trigrams[("the", "bird")].values()), 1)))
print(" a human knows birds eat, because bird is like cat and dog. an")
print(" n-gram model cannot know that: 'cat', 'dog' and 'bird' are three")
print(" unrelated symbols with nothing in common.")
print()
print("that second wall is precisely what embeddings were invented to knock")
print("down. represent a word as a vector rather than an index, and words")
print("used in similar contexts end up nearby -- so what the model learns")
print("about one transfers to the others.")
print()
print("n-grams are not obsolete, though. they are fast, need no training,")
print("and are still used for spelling correction, language identification,")
print("and as a baseline that neural models are expected to beat.")
Output
Questions people ask
Are n-grams obsolete? As language models, largely. As features, as evaluation metrics and for character-level tasks, no.
Why does adding bigrams help sentiment analysis? Because "not good" is the signal, and unigrams cannot represent it.
What is the Markov assumption? That only the last n−1 items matter. It is false and useful.
How do I handle unseen n-grams? Smoothing and backoff for a language model; min_df and a fixed vocabulary for features.
Should I use word or character n-grams? Words for clean text in a known language; characters for noisy text, misspellings, multiple languages and short strings.
Is BLEU an n-gram metric? Yes — it measures overlap of n-grams up to 4 between a candidate and reference, with a brevity penalty.
Recap in one screen
An n-gram is a run of n consecutive items; going beyond unigrams reintroduces word order.
Feature count grows explosively with n, so min_df and a capped vocabulary are essential.
As a language model, n-grams assume only the last n−1 words matter, and need smoothing for unseen sequences.
They cannot generalise across similar words or reach long distances — which is why neural models replaced them.
Still used for classical features, character-level tasks, autocomplete, and BLEU/ROUGE evaluation.
Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Features for classical classifiers” here?
TF-IDF over unigrams and bigrams, plus logistic regression, remains a strong baseline for text classification — and it trains in seconds.
What is meant by “Character n-grams” here?
for language identification, authorship attribution and handling misspellings. Character 3-grams are remarkably effective at identifying a language from a short string.
What is meant by “Autocomplete and query suggestion,” here?
where a count-based model over query logs is fast and adequate.
What is meant by “Spelling correction” here?
and fuzzy matching, using character n-gram overlap.
Cheat sheet
N-gram Explainer
If “the cat sat” appears 100 times and “the cat sat on” appears 60, then P(on | the cat, sat) = 0.6. There is no training in the gradient-descent sense — the model is a table of counts, built in one pass.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.