# Bag-of-words model

The bag-of-words model is a representation of text in which a document is reduced to the multiset of terms it contains: the ordering of words and their syntax are discarded, and only the number of occurrences of each term is kept<sup>[1](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html)</sup><sup> • </sup><sup>[2](https://technav.ieee.org/topic/bag-of-words-model/)</sup>. Under this view, the sentence "Mary is quicker than John" and "John is quicker than Mary" produce identical representations<sup>[1](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html)</sup>. The model underlies classical information retrieval scoring, text classification, and many natural language processing pipelines, where it turns variable-length text into fixed-length numerical vectors that machine learning algorithms can consume<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup>.

| Key fact | Detail |
|---|---|
| Definition | A document is represented by its terms and their counts; word order and syntax are ignored<sup>[1](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html)</sup> |
| Basic weight | Term frequency \( \mathrm{tf}_{t,d} \), the number of occurrences of term \( t \) in document \( d \)<sup>[1](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html)</sup> |
| Standard weighting | \( \mathrm{tf\text{-}idf}_{t,d} = \mathrm{tf}_{t,d} \times \mathrm{idf}_t \), highest for terms frequent in few documents and lowest for terms in nearly all documents<sup>[4](https://nlp.stanford.edu/IR-book/html/htmledition/tf-idf-weighting-1.html)</sup> |
| Similarity measure | Cosine similarity between document vectors, which compensates for differences in document length<sup>[5](https://nlp.stanford.edu/IR-book/html/htmledition/dot-products-1.html)</sup> |
| Sparsity | The document-term matrix is typically more than 99% zeros; 10,000 short documents can yield a vocabulary of about 100,000 words while each document uses only 100 to 1,000 unique words<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup> |
| Main limitation | Unigram bag-of-words cannot capture phrases, multi-word expressions, or word order dependence<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup> |
| Common extension | Bag of n-grams, counting pairs or longer runs of consecutive words to recover partial local order information<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup><sup> • </sup><sup>[6](https://aclanthology.org/E17-2068.pdf)</sup> |

## Construction

Building a bag-of-words representation involves three steps: tokenizing strings and assigning an integer identifier to each possible token, counting the occurrences of tokens in each document, and optionally normalizing or weighting the counts<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup>. The result is a document-term matrix with one row per document and one column per token in the corpus vocabulary<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup>.

The simplest weight for a term is its raw count in the document, a scheme called term frequency and denoted \( \mathrm{tf}_{t,d} \)<sup>[1](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html)</sup>. Any function that maps the number of occurrences of \( t \) in \( d \) to a positive real value can serve as the weight, and the resulting vector acts as a quantitative digest of the document<sup>[1](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html)</sup>.

## Weighting: from counts to tf-idf

Raw counts treat all terms as equally important, which is a problem for retrieval: in a collection about the auto industry, the term "auto" appears in almost every document and has little discriminating power<sup>[7](https://nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf)</sup>. The standard correction is inverse document frequency weighting, based on the document frequency \( \mathrm{df}_t \), the number of documents in the collection that contain term \( t \)<sup>[8](https://nlp.stanford.edu/IR-book/html/htmledition/inverse-document-frequency-1.html)</sup>.

Combining the two gives the tf-idf weight:

\[ \mathrm{tf\text{-}idf}_{t,d} = \mathrm{tf}_{t,d} \times \mathrm{idf}_t \]

A common concrete form assigns

\[ w_{t,d} = (1 + \log \mathrm{tf}_{t,d}) \cdot \log \frac{N}{\mathrm{df}_t} \]

where \( N \) is the number of documents in the collection<sup>[9](https://nlp.stanford.edu/IR-book/essir2011/pdf/vspace.pdf)</sup>. The tf-idf weight of a term is highest when the term occurs many times within a small number of documents, lower when it occurs fewer times or in many documents, and lowest when it occurs in virtually all documents<sup>[4](https://nlp.stanford.edu/IR-book/html/htmledition/tf-idf-weighting-1.html)</sup>.

## The vector space view

With a weight for each term, each document becomes a vector with one component per dictionary term, and the collection becomes a set of vectors in a space with one axis per term<sup>[4](https://nlp.stanford.edu/IR-book/html/htmledition/tf-idf-weighting-1.html)</sup><sup> • </sup><sup>[5](https://nlp.stanford.edu/IR-book/html/htmledition/dot-products-1.html)</sup>. This representation loses the relative ordering of terms, which is precisely the bag-of-words property<sup>[5](https://nlp.stanford.edu/IR-book/html/htmledition/dot-products-1.html)</sup>.

Because two documents with identical term distributions but different lengths would show a large raw vector difference, the standard similarity measure is the cosine similarity of the two vectors, the dot product divided by the product of their Euclidean lengths<sup>[5](https://nlp.stanford.edu/IR-book/html/htmledition/dot-products-1.html)</sup>. A query can be treated as a very short document, so the same cosine score ranks documents against a query, and a document can score highly even if it does not contain every query term<sup>[10](https://nlp.stanford.edu/IR-book/html/htmledition/queries-as-vectors-1.html)</sup>.

## Use in text classification

In supervised text classification, the bag-of-words vector is the input to a classifier. The multinomial variant of naive Bayes, one of the two classic naive Bayes variants for text, takes word count vectors as input, although tf-idf vectors are also known to work well in practice<sup>[11](https://scikit-learn.org/stable/modules/naive_bayes.html)</sup>. The Bernoulli variant instead requires binary feature vectors recording whether each term occurs, and it explicitly penalizes the non-occurrence of a term that indicates a class, which the multinomial variant ignores; it may perform better on datasets with shorter documents<sup>[11](https://scikit-learn.org/stable/modules/naive_bayes.html)</sup>.

A worked example on the 20 newsgroups dataset illustrates typical scale and performance. Tokenizing 2,257 posts from four categories produced a count matrix of 2,257 documents by 35,788 features. A pipeline of counting, tf-idf transformation, and multinomial naive Bayes reached 83.5% accuracy on the held-out posts; replacing the classifier with a linear support vector machine raised accuracy to 91.3%<sup>[12](https://scikit-learn.org/0.20/tutorial/text_analytics/working_with_text_data.html)</sup>. Linear models are the usual partners for bag-of-words features because the representation is high-dimensional and sparse, and sparse vectors make both storage and linear algebra tractable<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup>. On a 20 newsgroups subset, tf-idf vectors averaged about 159 non-zero components per document in a space of more than 30,000 dimensions, under 0.5% non-zero features<sup>[13](https://scikit-learn.org/stable/datasets/real_world.html)</sup>.

Despite its simplicity, the representation has been shown to be effective across a broad range of NLP tasks, including word sense disambiguation, text categorization, and sentiment analysis<sup>[14](https://www-nlp.stanford.edu/~wcmac/papers/nli-diss.pdf)</sup>. Preprocessing choices also matter: a systematic study over all combinations of five or six basic preprocessing methods on four benchmark corpora, using Bayes networks, an SVM variant, and random forests with bag-of-words unigrams, found that specific preprocessing combinations significantly improved classification results on particular datasets<sup>[15](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0232525)</sup>.

## Limitations and extensions

The central limitation is the loss of order. A collection of unigrams cannot capture phrases and multi-word expressions, and the model does not account for misspellings or word derivations<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup>. Two partial remedies stay within the counting framework:

- **Word n-grams.** Counting bigrams or longer runs of consecutive words preserves some local positioning information. The fastText classifier, for example, uses a bag of n-grams as additional features because explicitly modeling word order is computationally expensive, and this achieves results comparable to order-aware methods<sup>[6](https://aclanthology.org/E17-2068.pdf)</sup>.
- **[Character](https://www.edgepedia.org/character-computing) n-grams.** Counting character sequences makes the representation resilient to misspellings and derivations; for the corpus \([\text{words}, \text{wprds}]\), a unigram model sees two entirely distinct documents, while a character 2-gram representation finds the documents matching in 4 of 8 features<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup>.

Even with n-grams, bag-of-words and bag-of-n-grams representations destroy most of the inner structure of a document and the meaning carried by that structure, so tasks requiring natural language understanding need models that take sentence and paragraph structure into account<sup>[3](https://scikit-learn.org/1.1/modules/feature_extraction.html)</sup>. Continuous representations such as word2vec and doc2vec, which encode words and documents as dense vectors, are the main alternatives compared against bag-of-words in document classification studies<sup>[16](https://arxiv.org/abs/1707.01321)</sup>.

## References

1. [Term frequency and weighting, Introduction to Information Retrieval (Manning, Raghavan, Schütze), Stanford NLP](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html)
2. [Bag of words model, IEEE Technology Navigator](https://technav.ieee.org/topic/bag-of-words-model/)
3. [6.2. Feature extraction, scikit-learn 1.1.3 documentation](https://scikit-learn.org/1.1/modules/feature_extraction.html)
4. [Tf-idf weighting, Introduction to Information Retrieval, Stanford NLP](https://nlp.stanford.edu/IR-book/html/htmledition/tf-idf-weighting-1.html)
5. [Dot products, Introduction to Information Retrieval, Stanford NLP](https://nlp.stanford.edu/IR-book/html/htmledition/dot-products-1.html)
6. [Joulin, A. et al. (2016). Bag of Tricks for Efficient Text Classification, EACL 2017](https://aclanthology.org/E17-2068.pdf)
7. [Introduction to Information Retrieval, full text PDF, Stanford NLP](https://nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf)
8. [Inverse document frequency, Introduction to Information Retrieval, Stanford NLP](https://nlp.stanford.edu/IR-book/html/htmledition/inverse-document-frequency-1.html)
9. [IIR 6&7: Vector Space Model, lecture slides, Stanford NLP](https://nlp.stanford.edu/IR-book/essir2011/pdf/vspace.pdf)
10. [Queries as vectors, Introduction to Information Retrieval, Stanford NLP](https://nlp.stanford.edu/IR-book/html/htmledition/queries-as-vectors-1.html)
11. [1.9. Naive Bayes, scikit-learn 1.9.1 documentation](https://scikit-learn.org/stable/modules/naive_bayes.html)
12. [Working With Text Data, scikit-learn 0.20.4 tutorial](https://scikit-learn.org/0.20/tutorial/text_analytics/working_with_text_data.html)
13. [9.2. Real world datasets, scikit-learn 1.9.1 documentation](https://scikit-learn.org/stable/datasets/real_world.html)
14. [MacCartney, B. Natural Language Inference, Stanford dissertation](https://www-nlp.stanford.edu/~wcmac/papers/nli-diss.pdf)
15. [The influence of preprocessing on text classification using a bag-of-words representation, PLOS One (2020)](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0232525)
16. [The Influence of Feature Representation of Text on the Performance of Document Classification, arXiv:1707.01321](https://arxiv.org/abs/1707.01321)

---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Language and vision AI › Natural language processing › NLP tasks and methods › Text classification and categorization*

*Initially written Sep 17, 2026 · Reviewed: Oct 15, 2026 · Edited: Oct 15, 2026 · Last review: Oct 15, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license