TF-IDF and Feature Weighting

Bag of Words and n-grams treat all words equally, but in real text, not all words are equally important. This is where TF-IDF (Term Frequency–Inverse Document Frequency) comes in. TF-IDF is a simple idea with a big impact: it rewards words that are important in a document while down-weighting words that appear everywhere.

The term frequency (TF) part measures how often a word appears in a document. Intuitively, words that occur more frequently are more likely to be relevant to that document. But frequency alone can be misleading—common words like “data,” “system,” or “use” may appear often across many documents without saying much.

That’s where inverse document frequency (IDF) helps. IDF reduces the weight of words that appear in many documents and boosts words that are relatively rare. A term that appears in almost every document gets a low IDF score, while a term that appears in only a few documents gets a higher one. When TF and IDF are combined, you get a score that highlights words that are both frequent and distinctive.

In practice, TF-IDF produces vectors that are usually more informative than raw word counts. It often improves performance in tasks like document classification, search, and information retrieval, especially when working with large text collections.

Even though modern NLP uses embeddings and transformers, TF-IDF is still widely used. It’s fast, easy to interpret, and surprisingly strong as a baseline. Many production systems still rely on it—either on its own or alongside more complex models.