Stemming vs Lemmatization

After tokenization, you’ll often notice that the same word appears in multiple forms—run, running, runs, ran. For a machine, these are completely different tokens unless you do something about it. This is where stemming and lemmatization come into play. Both aim to reduce words to a common base form, but they do it in very different ways.

Stemming is the simpler and faster approach. It works by chopping off word endings using a set of crude rules. For example, “playing,” “played,” and “player” might all be reduced to “play.” While this can reduce vocabulary size quickly, it often produces stems that aren’t real words, such as “stud” from “studies.” For many tasks, that’s acceptable, but it can look messy and occasionally misleading.

Lemmatization takes a more informed approach. Instead of blindly trimming words, it uses vocabulary and linguistic rules to map words back to their dictionary form, known as a lemma. For instance, “better” becomes “good,” and “running” becomes “run.” Lemmatization is slower and requires more linguistic knowledge, but the results are cleaner and more interpretable.

Choosing between the two depends on your goal. If you’re working on large-scale text analysis where speed matters more than precision, stemming may be enough. If your task involves meaning, readability, or downstream interpretation—such as question answering or sentiment analysis—lemmatization is usually the better choice.

The key takeaway is not that one method is always superior, but that both are tools. Good NLP practitioners choose based on context, data size, and the problem they’re trying to solve—not habit or convenience.