Bag of Words and N-gram Models

Before deep learning entered the picture, most NLP systems relied on simple but powerful ideas to represent text numerically. One of the earliest and most influential of these is the Bag of Words (BoW) model. Despite the name, it’s not as casual as it sounds.

In a Bag of Words approach, a document is represented by the frequency of words, ignoring grammar and word order. If a sentence contains the word “data” three times, that count becomes a feature. Every unique word in the vocabulary becomes a column, and each document turns into a long vector of counts. The model doesn’t know where the word appeared—only that it appeared. Surprisingly, this works well for many tasks like spam detection or topic classification.

However, BoW has a clear limitation: it loses context. The sentences “not good” and “good” may look very similar if both contain the word “good.” This is where n-gram models improve things. Instead of counting single words (unigrams), n-grams count sequences of words—bigrams (“not good”) or trigrams (“not at all good”). By capturing short word sequences, n-grams preserve some local context and meaning.

Of course, there’s a trade-off. As you increase the value of n, the number of features grows rapidly, making models heavier and more memory-intensive. This is why practical NLP systems carefully choose which n-grams to keep, often using frequency thresholds or limits.

Even today, Bag of Words and n-grams are far from obsolete. They’re fast, interpretable, and often strong baselines. Understanding them builds intuition for why more advanced representations exist—and when simple methods are still the smarter choice.