Interactive visualization of how text is broken down into contiguous sequences of N items for language modeling.
Enter text to generate sequences
Tip: Use the slider to scroll through the list if it overflows.
An n-gram model estimates the chance of a word from the n−1 words before it, using nothing but counts from a corpus:
P(wt | wt−1, …) ≈ count(wt−n+1 … wt) / count(wt−n+1 … wt−1)
If “the cat sat” appears 100 times and “the cat sat on” appears 60. Then P(on | the cat, sat) = 0.6. There is no training in the gradient-descent sense. The model is a table of counts, built in one pass.
The underlying simplification is the Markov assumption: that only the last n−1 words matter. It is plainly false for language and works surprisingly well anyway.
This page shows how the model is built, how to choose n, and why sparsity. The fact that most valid sentences have never been written down. Is the basic limit. It also covers smoothing methods that redistribute chance mass to unseen n-grams, and how n-gram counts are still used as features in text classification and as baselines for evaluating neural language models. The visualisation above lets you type a sentence and watch the n-gram chances update from a live corpus. The code blocks below build a trigram model from scratch so you can inspect the count tables right.
Interactive visualization of how text is broken down into contiguous sequences of N items for language modeling.
Predict the next word from the previous n-1 words, by counting how often that sequence appeared. Simple, fast, interpretable - and defeated by the fact that most valid sentences have never been written down.
An n-gram model estimates the chance of a word from the n−1 words before it, using nothing but counts from a corpus:
P(wt | wt−1, …) ≈ count(wt−n+1 … wt) / count(wt−n+1 … wt−1)
If “the cat sat” appears 100 times and “the cat sat on” appears 60. Then P(on | the cat, sat) = 0.6. There is no training in the gradient-descent sense. The model is a table of counts, built in one pass.
The underlying simplification is the Markov assumption: that only the last n−1 words matter. It is plainly false for language and works surprisingly well anyway.
This page shows how the model is built, how to choose n, and why sparsity. The fact that most valid sentences have never been written down. Is the basic limit. It also covers smoothing methods that redistribute chance mass to unseen n-grams, and how n-gram counts are still used as features in text classification and as baselines for evaluating neural language models. The visualisation above lets you type a sentence and watch the n-gram chances update from a live corpus. The code blocks below build a trigram model from scratch so you can inspect the count tables right.
| n | Behaviour |
|---|---|
| 1 | No word order at all; smallest feature set |
| 2 | Captures negation and common phrases; usually worth it |
| 3 | Some benefit, considerably sparser |
| 4+ | Rarely justified for features; used in language models with heavy smoothing |
For classification features, (1, 2) is the standard choice and the marginal gain from trigrams is often small next to the extra columns.
For a count-based language model, trigrams to 5-grams with Kneser-Ney smoothing was the state of the art for years. And it needed large corpora to estimate.
The counts grow catastrophically. With a 50,000-word vocabulary there are 2.5 billion doable bigrams, 1.25 × 1014 trigrams, and 3 × 1023 5-grams. No corpus covers a meaningful fraction of them.
So most n-grams have a count of zero, and a zero count means a chance of zero. The model declares a perfectly ordinary sentence not doable because it happens not to have seen one four-word span before. Worse, one zero anywhere makes the chance of the whole sentence zero.
The fixes are all forms of smoothing. Add-one (Laplace) smoothing adds a pseudocount to all of it, which is simple and crude. Backoff falls back to a shorter n-gram when the longer one is unseen. Kneser-Ney, the best of the classical methods, discounts observed counts and redistributes the mass according to how many distinct contexts a word appears in. Capturing that Francisco is common but only ever after San.
An n-gram is a contiguous run of n items. For "the cat sat on the mat":
| n | Name | Grams |
|---|---|---|
| 1 | Unigram | the, cat, sat, on, the, mat |
| 2 | Bigram | the cat, cat sat, sat on, on the, the mat |
| 3 | Trigram | the cat sat, cat sat on, sat on the, on the mat |
Why bother beyond unigrams? Because word order carries meaning that a bag of single words discards. "Not good" and "good not" have the same unigrams; as bigrams they are distinguishable. Sentiment classification improves measurably from bigrams for just this reason.
The cost is the vocabulary. With 10,000 distinct words there are up to 100 million doable bigrams and a trillion trigrams. Almost all are absent from any real corpus, so the feature matrix becomes enormous and very sparse.
That trade — more context against fast more features — is the whole subject.
Before neural networks, next-word prediction was done by counting. An n-gram language model estimates:
P(wᵗ | w₁ … wᵗ₋₁) ≈ P(wᵗ | wᵗ₋ₙ₊₁ … wᵗ₋₁)
The estimate is the Markov assumption: only the last n−1 words matter. For a trigram model, the chance of the next word depends on the two before it and nothing further back.
Estimation is division: count how often "sat on the" was followed by "mat", divide by how often "sat on the" appeared at all.
Two problems arise at once, and the fixes are classic:
Unseen sequences get chance zero, which zeroes the chance of the whole sentence. Smoothing fixes it — add-one (Laplace) is the simplest, and Kneser-Ney is the one that really works well,. This is because it accounts for how many different contexts a word appears in.
Longer n means better context and sparser counts. Backoff and interpolation handle it by falling back to shorter n-grams when the longer one has no data.
Two limitations are structural, not fixable by better smoothing.
No extension across close words. A trigram model that has seen "the black cat" learns nothing about "the dark cat". The two contexts are fully separate counts. An embedding-based model places "black" and "dark" near each other, so evidence transfers.
No long-range dependency. A 5-gram model cannot connect a pronoun to a noun seven words earlier. Raising n makes the counts too sparse to estimate long before the range becomes useful.
Neural language models solved both — distributed forms generalise across words, and recurrence or attention reaches arbitrary distances. That is the whole reason the field moved.
An n-gram model estimates the next word by counting how often each follow-up followed the last n−1 words. This makes it fast, interpretable, and trainable in a single pass. It is defeated by sparsity: the number of doable n-grams grows fast with n. So most sequences are never observed and smoothing is mandatory. Neural language models replaced it just because embeddings let them generalise across close contexts rather than needing each one to have been seen.
They have not disappeared, because counting is cheap, interpretable and needs no training:
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer(ngram_range=(1, 2), # unigrams and bigrams
min_df=3, # drop very rare grams
max_features=50_000,
sublinear_tf=True)
X = vec.fit_transform(docs)min_df and max_features are what keep this tractable — without them, bigrams alone can give millions of columns, almost all appearing once.
For character n-grams, analyzer="char_wb", ngram_range=(3, 5) is a strong setting for noisy or multilingual text.
An n-gram model is a table of counts. Building one shows both why it works at all and the two walls it hits. Sparsity and no extension.
Building on the same setup, the next block shows the left cases.
The below block covers the left steps.
The next block goes on with the left demonstrations.
Building on the same setup, the next block shows the left cases.
Are n-grams obsolete? As language models, largely. As features, as evaluation metrics and for character-level tasks, no.
Why does adding bigrams help sentiment review? Because "not good" is the sign, and unigrams cannot represent it.
What is the Markov assumption? That only the last n−1 items matter. It is false and useful.
How do I handle unseen n-grams? Smoothing and backoff for a language model; min_df and a fixed vocabulary for features.
Should I use word or character n-grams? Words for clean text in a known language; characters for noisy text, misspellings, multiple languages and short strings.
Is BLEU an n-gram metric? Yes — it measures overlap of n-grams up to 4 between a candidate and guide, with a brevity cost.
min_df and a capped vocabulary are key.Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Features for classical classifiers” here?
What is meant by “Character n-grams” here?
What is meant by “Autocomplete and query suggestion,” here?
What is meant by “Spelling correction” here?
Slide an n-gram window across text and see how counting short sequences lets a model guess which word comes next.