vizlearn
Home / Natural Language Processing

Stemming vs Lemmatization

Natural Language ProcessingBy Updated

Compare how NLP algorithms reduce words to their base forms using heuristic vs dictionary approaches.

Overview

The Goal: Text Normalization

In Natural Language Processing (NLP), we often need to treat different forms of a word as the same. For example, "run", "running", and "ran" all refer to the same basic concept. The process of reducing these variations down to a common base form is called text normalization. Stemming and lemmatization are two popular techniques for achieving this.

Input Text

Tip: Use words like "running", "better", "feet", or "is" to see the most dramatic differences.

Transformation Matrix

0 Words
Original Stemmed Lemma

Key Logic

Stemming (Porte's)

A crude heuristic process that chops off the ends of words. It often results in non-dictionary roots (e.g., "studies" → "studi").

Lemmatization

A sophisticated process using vocabulary and morphological analysis. It always returns a valid dictionary word (e.g., "are" → "be").

Differences

Rule-Based
Context-Aware
No Change

Stemming vs. Lemmatization: A Practical Comparison

Explore two key NLP techniques for text normalization and understand when to use each one.

Stemming: The Fast and Crude Approach

Stemming is a process that reduces words to their "stem" or root form by chopping off prefixes and suffixes. It uses a set of simple, rule-based heuristics and does not care if the resulting stem is a real dictionary word.

  • Method: Algorithmic, rule-based (e.g., remove "ing", "ed", "s").
  • Speed: Very fast, as it doesn't need to look up words in a dictionary.
  • Result: Often produces non-words. For example, "studies" might become "studi".
  • Use Case: Ideal for applications where speed is critical and perfect accuracy isn't necessary, such as search engine indexing or large-scale text analysis.

Lemmatization: The Smart and Accurate Approach

Lemmatization is a more sophisticated process that aims to find the true root form of a word, known as the lemma. It uses a dictionary and morphological analysis to consider the context of the word and its part of speech.

  • Method: Dictionary-based, considers the word's meaning and part of speech.
  • Speed: Slower than stemming because it involves dictionary lookups.
  • Result: Always returns a valid dictionary word. For example, "better" is correctly identified as having the lemma "good", and "are" becomes "be".
  • Use Case: Best for applications requiring high accuracy, such as chatbots, question-answering systems, and machine translation, where understanding the correct meaning of words is essential.

Two ways to collapse word forms

"Run", "runs", "running" and "ran" are four strings and one concept. To a bag-of-words model they are four unrelated features, each with a quarter of the evidence. Reducing them to a common form pools that evidence.

There are two methods, and they differ in how much they know.

Stemming chops suffixes according to rules. Fast, crude, and it produces strings that are not always words.

Lemmatisation looks the word up, using its part of speech, and returns the real dictionary form.

WordPorter stemmerLemmatiser
runningrunrun
runsrunrun
ranranrun
studiesstudistudy
betterbettergood
waswabe
caringcarecare

Two rows show the difference clearly. "Ran" and "better" need knowledge, not rules — a stemmer has no way to know that "ran" relates to "run" or that "better" is a form of "good". And "studi" and "wa" show what rule-based chopping produces: fragments that work as index keys and read as errors.

When each is right

Stemming is appropriate when the output is a lookup key nobody reads. Search indexing is the canonical case: if "studi" is the key for both "study" and "studies", a query for either matches, and the user never sees the stem.

Lemmatisation is appropriate when the output is shown to someone, or when correctness matters — topic modelling where you display the top terms, linguistic analysis, anything with an audit trail.

 StemmingLemmatisation
MethodSuffix rulesDictionary plus part of speech
OutputSometimes not a wordAlways a real word
SpeedVery fastSlower — needs POS tagging
Handles irregularsNoYes
Needs language resourcesNoYes, per language
from nltk.stem import PorterStemmer, WordNetLemmatizer

PorterStemmer().stem("studies")                        # 'studi'
WordNetLemmatizer().lemmatize("studies")               # 'study'
WordNetLemmatizer().lemmatize("better", pos="a")       # 'good'

That last line matters: without the part of speech, a lemmatiser assumes noun and gets adjectives and verbs wrong. lemmatize("running") returns "running" unless you tell it the word is a verb. In practice, run a POS tagger first — spaCy does this automatically.

Why neither belongs in a transformer pipeline

Both techniques exist to solve a problem transformers do not have.

A subword tokeniser already splits "running" into "run" + "ning", so the shared stem is represented and the suffix is available as separate information. The model sees both the root and the inflection, and it can use the inflection — tense, number and aspect all carry meaning.

Stemming before a transformer therefore does two harmful things: it destroys grammatical information the model was trained to use, and it creates strings ("studi", "wa") that are not in the vocabulary and tokenise into odd fragments.

The rule is simple. Stem or lemmatise for classical bag-of-words models. Never for transformers.

Chopping suffixes against looking words up

A stemmer applies rules and can produce nonsense. A lemmatiser consults a dictionary and needs to know the part of speech. Both are implemented here on the same words, including the cases where each one fails.

example_01.pyNumPy
Output

                

                

                    
Things to try

                    
Use the interactive tool above to see the difference firsthand:
  1. Analyze the Default Text: Click the "Normalize Text" button with the default sentence. Look at the results in the "Transformation Matrix". 
       
    • Notice how for "foxes", both stemming and lemmatization produce "fox".
      •  
    • For "jumping", stemming gives "jump" and lemmatization also gives "jump". In many cases, they agree.
      •  
    • But for "studying", stemming produces "studi" (not a real word), while lemmatization correctly returns "study".
      •  
  2. Test Irregular Verbs: Clear the text and type "He went to the best restaurants. They were good." Click "Normalize". 
       
    • Stemming fails on these: "went" stays "went", "best" stays "best", "were" becomes "wer".
      •  
    • Lemmatization shines: "went" becomes "go", "best" becomes "good", and "were" becomes "be". It understands the underlying lemma.
      •  
  3. The "Caring" vs. "Cars" Test: Type "caring for cars". 
       
    • Stemming will likely reduce both to "car". It loses the meaning and context.
      •  
    • Lemmatization will correctly identify "caring" -> "care" and "cars" -> "car".
      •  

                

                

                    
Key Takeaway: Speed vs. Accuracy

                    
The choice between stemming and lemmatization is a trade-off. Stemming is fast and good enough for many applications, but it can be inaccurate. Lemmatization is more accurate and provides meaningful root words, but it comes at a higher computational cost. Choose the tool that best fits the needs of your specific NLP task.

                

                

                    
The stemmers you will meet

                    
Porter (1980) is the classic for English — five phases of suffix rules, well understood, moderately aggressive. Still the default in many pipelines.
Snowball (Porter2) is Porter's own improvement, and it covers many languages. The better default when you have a choice.
Lancaster is more aggressive and produces shorter stems, sometimes conflating words that should stay distinct.
For lemmatisation, WordNet (via NLTK) is the traditional English option, and spaCy is the practical one — it tags parts of speech and lemmatises in one pass, across many languages, and is considerably faster.
import spacy
nlp = spacy.load("en_core_web_sm")

doc = nlp("The studies were better than we ran previously")
[t.lemma_ for t in doc]
# ['the', 'study', 'be', 'well', 'than', 'we', 'run', 'previously']
Note "be" from "were" and "run" from "ran" — the irregulars a stemmer cannot reach.

                

                

                    
What it buys, measured

                    
The benefit is vocabulary reduction, and it is substantial for classical models.
On a typical English corpus, stemming reduces the distinct term count by roughly 30–50%. That means more occurrences per feature, a smaller and better-conditioned matrix, and less overfitting on rare forms.
Whether it improves accuracy is less certain and task-dependent. For topic classification and search recall it usually helps. For sentiment it can hurt, because it flattens distinctions ("caring" and "cared" carry different implications). For anything where tense or number is the signal, it clearly hurts.
The honest guidance: treat it as a hyperparameter and test it, rather than applying it because a tutorial did.

                

                

                    
Common mistakes

                    
  • Lemmatising without a part of speech, so verbs and adjectives are treated as nouns and left unchanged.
  • Using an English stemmer on other languages. Suffix rules are language-specific, and applying English ones to German or Turkish produces nonsense.
  • Stemming before a transformer. Destroys information and creates out-of-vocabulary fragments.
  • Stemming the query but not the index (or vice versa) in a search system, so nothing matches.
  • Assuming lemmatisation is always better. It is more correct and slower, and for a search index the stem is fine.
  • Applying it to named entities, where "Rogers" becoming "Roger" changes the referent.

                

                

                    
Questions people ask

                    
Which should I use? Lemmatisation when output quality matters or is shown to users; stemming for search indexes and quick baselines; neither for transformers.
Is lemmatisation much slower? Meaningfully so, because it needs POS tagging and dictionary lookups — roughly an order of magnitude in NLTK, less with spaCy's pipeline.
Do I need stop-word removal too? For classical bag-of-words models, often yes. For transformers, no.
What about other languages? Snowball covers many for stemming; spaCy has lemmatisers for a wide range. Morphologically rich languages benefit more from lemmatisation than English does.
Does it help search? Yes, for recall — matching "running" to a query for "run". Modern search engines do this, alongside embedding-based retrieval.
Why does the Porter stemmer produce non-words? Because it applies suffix rules without any dictionary. The stems are index keys, not words.

                

                

                    
Recap in one screen

                    
  • Both reduce inflected forms to a common form so evidence pools across them.
  • Stemming chops with rules — fast, and it produces non-words and misses irregulars.
  • Lemmatisation uses a dictionary and part of speech — correct, slower, and it needs the POS to be right.
  • Use stemming for index keys, lemmatisation for anything readable, neither before a transformer.
  • Vocabulary shrinks by 30–50%; accuracy gains are task-dependent, so test rather than assume.

                

            


        

    



        
    



Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. What is meant by “Lemmatising without a part of speech,” here?
  2. What is meant by “Using an English stemmer on other languages” here?
  3. What is meant by “Stemming before a transformer” here?
  4. What is meant by “Stemming the query but not the index” here?





 Cheat sheet
Stemming vs Lemmatization
In Natural Language Processing (NLP), we often need to treat different forms of a word as the same. For example, "run", "running", and "ran" all refer to the same basic concept. The process of reducing these variations down to a common base form is called text normalization. Stemming and lemmatization are two popular techniques for achieving this.
NLP · vizlearn.in/natural_language_processing/stemming_vs_lemmatization.html
More in NLP
NLPN-gram ExplainerNLPWord Cloud GeneratorNLPWhy Text Encoding is Needed in NLPNLPText Encoding Techniques in NLP
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact