vizlearn
Home / Natural Language Processing

Stemming vs Lemmatization

Natural Language ProcessingBy Updated

Compare how NLP algorithms reduce words to their base forms using heuristic vs dictionary approaches.

Input Text

Tip: Use words like "running", "better", "feet", or "is" to see the most dramatic differences.

Transformation Matrix

0 Words
Original Stemmed Lemma

Key Logic

Stemming (Porte's)

A crude heuristic process that chops off the ends of words. It often results in non-dictionary roots (e.g., "studies" → "studi").

Lemmatization

A sophisticated process using vocabulary and morphological analysis. It always returns a valid dictionary word (e.g., "are" → "be").

Differences

Rule-Based
Context-Aware
No Change

Overview

Compare how NLP algorithms reduce words to their base forms using heuristic vs dictionary approaches.

In this guide
  1. The Goal: Text Normalization
  2. Stemming: The Fast and Crude Approach
  3. Lemmatization: The Smart and Exact Approach
  4. Two ways to collapse word forms
  5. When each is right
  6. Why neither belongs in a transformer pipeline
  7. Chopping suffixes against looking words up
  8. Things to try
  9. Key Takeaway: Speed vs. Precision
  10. The stemmers you will meet
  11. What it buys, measured
  12. Common mistakes
  13. Questions people ask
  14. Recap in one screen

Stemming vs. Lemmatization: A Practical Comparison

Explore two key NLP techniques for text normalization and understand when to use each one.

The Goal: Text Normalization

In Natural Language Handling (NLP), we often need to treat different forms of a word as the same. For case, "run", "running", and "ran" all refer to the same basic concept. The process of reducing these changes down to a common base form is called text normalization. Stemming and lemmatization are two popular methods for reaching this.

Stemming strips suffixes with handwritten rules. It is fast, needs no dictionary, and often gives non-words: "studies" turns "studi", "better" stays "better". The Porter stemmer is the most widely used. And its output is meant for indexing and grouping, not for display.

Lemmatization uses a dictionary and part-of-speech data to find the actual root word: "studies" turns "study", "better" turns "good", "mice" turns "mouse". It is slower and needs a POS tagger to work correctly. "saw" lemmatises to "see" as a verb and "saw" as a noun. But its output is always a real word.

The choice depends on the downstream task and the model. For bag-of-words models, TF-IDF pipelines and search indexes. Where vocabulary reduction is the goal, stemming is often enough. The non-word output does not matter because no human reads it. For tasks where the output must be readable. Text generation, summarisation, chatbot responses. Lemmatization is the right choice. For transformer-based models, neither is needed: the subword tokeniser handles morphological change internally, and applying stemming or lemmatization before the tokeniser removes data the model was trained to use.

Stemming: The Fast and Crude Approach

Stemming is a process that reduces words to their "stem" or root form by chopping off prefixes and suffixes. It uses a set of simple, rule-based heuristics and does not care if the coming stem is a real dictionary word.

  • Method: Coded, rule-based (e.g., remove "ing", "ed", "s").
  • Speed: Very fast, as it doesn't need to look up words in a dictionary.
  • Result: Often gives non-words. For case, "studies" might turn "studi".
  • Use Case: Ideal for uses where speed is critical and perfect precision isn't needed, such as search engine indexing or large-scale text review.

Lemmatization: The Smart and Exact Approach

Lemmatization is a more complex process that aims to find the true root form of a word, known as the lemma. It uses a dictionary and morphological review to think of the context of the word and its part of speech.

  • Method: Dictionary-based, thinks of the word's meaning and part of speech.
  • Speed: Slower than stemming. This is because it involves dictionary lookups.
  • Result: Always returns a valid dictionary word. For case, "better" is correctly found as having the lemma "good", and "are" turns "be".
  • Use Case: Best for uses needing high precision, such as chatbots, question-answering systems, and machine translation. Where grasp the correct meaning of words is key.

Two ways to collapse word forms

"Run", "runs", "running" and "ran" are four strings and one concept. To a bag-of-words model they are four unrelated features, each with a quarter of the evidence. Reducing them to a common form pools that evidence.

There are two methods, and they differ in how much they know.

Stemming chops suffixes according to rules. Fast, crude, and it gives strings that are not always words.

Lemmatisation looks the word up, using its part of speech, and returns the real dictionary form.

WordPorter stemmerLemmatiser
runningrunrun
runsrunrun
ranranrun
studiesstudistudy
betterbettergood
waswabe
caringcarecare

Two rows show the gap clearly. "Ran" and "better" need knowledge, not rules — a stemmer has no way to know that "ran" relates to "run" or that "better" is a form of "good". And "studi" and "wa" show what rule-based chopping gives. Fragments that work as index keys and read as errors.

When each is right

Stemming is right when the output is a lookup key nobody reads. Search indexing is the canonical case: if "studi" is the key for both "study" and "studies", a query for either matches. And the user never sees the stem.

Lemmatisation is right when the output is shown to someone, or when correctness matters. Topic modelling where you display the top terms, linguistic review, a thing with an audit trail.

 StemmingLemmatisation
MethodSuffix rulesDictionary plus part of speech
OutputSometimes not a wordAlways a real word
SpeedVery fastSlower — needs POS tagging
Handles irregularsNoYes
Needs language resourcesNoYes, per language
from nltk.stem import PorterStemmer, WordNetLemmatizer

PorterStemmer().stem("studies")                        # 'studi'
WordNetLemmatizer().lemmatize("studies")               # 'study'
WordNetLemmatizer().lemmatize("better", pos="a")       # 'good'

That last line matters: without the part of speech, a lemmatiser assumes noun and gets adjectives and verbs wrong. lemmatize("running") returns "running" unless you tell it the word is a verb. In practice, run a POS tagger first — spaCy does this by itself.

Why neither belongs in a transformer pipeline

Both methods exist to solve a problem transformers do not have.

A subword tokeniser now splits "running" into "run" + "ning", so the shared stem is shown and the suffix is available as separate data. The model sees both the root and the inflection, and it can use the inflection. Tense, number and aspect all carry meaning.

Stemming before a transformer so does two harmful things. It destroys grammatical data the model was trained to use. And it creates strings ("studi", "wa") that are not in the vocabulary and tokenise into odd fragments.

The rule is simple. Stem or lemmatise for classical bag-of-words models. Never for transformers.

Chopping suffixes against looking words up

A stemmer applies rules and can give nonsense. A lemmatiser consults a dictionary and needs to know the part of speech. Both are implemented here on the same words, including the cases where each one fails.

example_01.pyNumPy
Output
The next block goes on with the left demonstrations.
example_02.pyNumPy
Output
The next block goes on with the left demonstrations.
example_03.pyNumPy
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
Output
The next block goes on with the left demonstrations.
example_05.pyNumPy
Output

                

                

                    
Things to try

                    
Use the interactive tool above to see the gap firsthand:
  1. Analyze the Default Text: Click the "Normalize Text" button with the default sentence. Look at the results in the "Transformation Matrix". 
       
    • Notice how for "foxes", both stemming and lemmatization give "fox".
      •  
    • For "jumping", stemming gives "jump" and lemmatization also gives "jump". Often, they agree.
      •  
    • But for "studying", stemming gives "studi" (not a real word). While lemmatization correctly returns "study".
      •  
  2. Test Irregular Verbs: Clear the text and type "He went to the best restaurants. They were good." Click "Normalize". 
       
    • Stemming fails on these: "went" stays "went", "best" stays "best", "were" turns "wer".
      •  
    • Lemmatization shines: "went" turns "go", "best" turns "good", and "were" turns "be". It understands the underlying lemma.
      •  
  3. The "Caring" vs. "Cars" Test: Type "caring for cars". 
       
    • Stemming will likely reduce both to "car". It loses the meaning and context.
      •  
    • Lemmatization will correctly find "caring" -> "care" and "cars" -> "car".
      •  

                

                

                    
Key Takeaway: Speed vs. Precision

                    
The choice between stemming and lemmatization is a trade-off. Stemming is fast and good enough for many uses, but it can be wrong. Lemmatization is more exact and gives meaningful root words. But it comes at a higher compute cost. Choose the tool that best fits the needs of your set NLP task.

                

                

                    
The stemmers you will meet

                    
Porter (1980) is the classic for English — five phases of suffix rules, well understood, moderately aggressive. Still the default in many pipelines.
Snowball (Porter2) is Porter's own improvement, and it covers many languages. The better default when you have a choice.
Lancaster is more aggressive and gives shorter stems, sometimes conflating words that should stay distinct.
For lemmatisation, WordNet (via NLTK) is the traditional English option, and spaCy is the real one. It tags parts of speech and lemmatises in one pass, across many languages, and is much faster.
import spacy
nlp = spacy.load("en_core_web_sm")

doc = nlp("The studies were better than we ran previously")
[t.lemma_ for t in doc]
# ['the', 'study', 'be', 'well', 'than', 'we', 'run', 'previously']
Note "be" from "were" and "run" from "ran" — the irregulars a stemmer cannot reach.

                

                

                    
What it buys, measured

                    
The gain is vocabulary reduction, and it is large for classical models.
On a typical English corpus, stemming reduces the distinct term count by roughly 30–50%. That means more instances per feature, a smaller and better-conditioned matrix, and less overfitting on rare forms.
Whether it improves precision is less certain and task-dependent. For topic classification and search recall it often helps. For sentiment it can hurt, because it flattens distinctions ("caring" and "cared" carry different implications). For a thing where tense or number is the sign, it clearly hurts.
The honest guidance: treat it as a hyperparameter and test it, rather than applying it because a tutorial did.

                

                

                    
Common mistakes

                    
  • Lemmatising without a part of speech, so verbs and adjectives are treated as nouns and left unchanged.
  • Using an English stemmer on other languages. Suffix rules are language-set, and applying English ones to German or Turkish gives nonsense.
  • Stemming before a transformer. Destroys data and creates out-of-vocabulary fragments.
  • Stemming the query but not the index (or vice versa) in a search system, so nothing matches.
  • Assuming lemmatisation is always better. It is more correct and slower. And for a search index the stem is fine.
  • Applying it to named entities, where "Rogers" turning "Roger" changes the referent.

                

                

                    
Questions people ask

                    
Which should I use? Lemmatisation when output grade matters or is shown to users; stemming for search indexes and quick baselines; neither for transformers.
Is lemmatisation much slower? Meaningfully so, because it needs POS tagging and dictionary lookups. Roughly an order of size in NLTK, less with spaCy's pipeline.
Do I need stop-word cutting too? For classical bag-of-words models, often yes. For transformers, no.
What about other languages? Snowball covers many for stemming; spaCy has lemmatisers for a wide range. Morphologically rich languages gain more from lemmatisation than English does.
Does it help search? Yes, for recall — matching "running" to a query for "run". Modern search engines do this, alongside embedding-based retrieval.
Why does the Porter stemmer give non-words? Because it applies suffix rules without any dictionary. The stems are index keys, not words.

                

                

                    
Recap in one screen

                    
  • Both reduce inflected forms to a common form so evidence pools across them.
  • Stemming chops with rules — fast, and it gives non-words and misses irregulars.
  • Lemmatisation uses a dictionary and part of speech — correct, slower. And it needs the POS to be right.
  • Use stemming for index keys, lemmatisation for a thing readable, neither before a transformer.
  • Vocabulary shrinks by 30–50%. Precision gains are task-dependent, so test rather than assume.

                

            


        

    



        
    



Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. What is meant by “Lemmatising without a part of speech,” here?
  2. What is meant by “Using an English stemmer on other languages” here?
  3. What is meant by “Stemming before a transformer” here?
  4. What is meant by “Stemming the query but not the index” here?





 Cheat sheet
Stemming vs Lemmatization
Compare a stemmer chopping suffixes against a lemmatizer resolving proper dictionary forms of the very same words.
NLP · vizlearn.in/natural_language_processing/stemming_vs_lemmatization.html
More in NLP
NLPN-gram ExplainerNLPText Classification with Naive BayesNLPWord Cloud GeneratorNLPWhy Text Encoding is Needed in NLP
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact