= Document frequency
{title2=$\mathrm{df}_t$}
The <document frequency> $\mathrm{df}_t$ of term $t$ is the number of documents in a specified collection that contain it at least once. Repeating a term many times inside one document contributes only one to its <document frequency>.<Introduction to Information Retrieval by Christopher D. Manning and others>[§6.2.1, Inverse document frequency, definition of document frequency and Figure 6.7 comparison with total occurrences]{ref}
For a collection $D$, write the count using an <indicator function>:
$$
\mathrm{df}_t=\sum_{d\in D}\mathbf1_{\{\mathrm{tf}_{t,d}>0\}},\qquad 0\leq\mathrm{df}_t\leq |D|.
$$
Here $\mathrm{tf}_{t,d}$ is the raw <term frequency>. When an <inverted index> has one document record per containing document, the number of those records in the term's <postings list> equals $\mathrm{df}_t$. A record's position list can still contain several occurrences.
Consider `red red boat`, `boat red red` and `blue boat`. Red occurs four times but has <document frequency> two. Boat occurs only three times but has <document frequency> three. Thus the term with more total occurrences need not appear in more documents. Blue has <document frequency> one.
Adding another hundred red occurrences to the first document increases its <term frequency> without changing red's <document frequency>. Adding a new document containing red increases the latter by one. The collection boundary and preprocessing rules are therefore part of the statistic's definition. In <information retrieval>, this distinction lets scoring use evidence about how widely a term is distributed, rather than confusing repetition in one document with coverage of many documents.
Back to article page
