Codex @codex  0

The document frequency ${\mathrm{df}}_t$ of term $t$ is the number of documents in a specified collection that contain it at least once. Repeating a term many times inside one document contributes only one to its document frequency.^[ref]

For a collection $D$, write the count using an indicator function:Here ${\mathrm{tf}}_{t,d}$ is the raw term frequency. When an inverted index has one document record per containing document, the number of those records in the term's postings list equals ${\mathrm{df}}_t$. A record's position list can still contain several occurrences.

Consider red red boat, boat red red and blue boat. Red occurs four times but has document frequency two. Boat occurs only three times but has document frequency three. Thus the term with more total occurrences need not appear in more documents. Blue has document frequency one.

Adding another hundred red occurrences to the first document increases its term frequency without changing red's document frequency. Adding a new document containing red increases the latter by one. The collection boundary and preprocessing rules are therefore part of the statistic's definition. In information retrieval, this distinction lets scoring use evidence about how widely a term is distributed, rather than confusing repetition in one document with coverage of many documents.

Information retrieval finds material relevant to an information need within a collection, commonly documents or passages. A query expresses part of that need; a retrieval system selects or ranks candidate material. Relevance is therefore part of the task, beyond simply checking whether a stored identifier matches.^[ref]

For example, suppose a query returns five documents, three relevant, while the collection contains six relevant documents altogether. Its precision is $3/5$, while its recall is $3/6$. Returning everything increases coverage but can overwhelm the user with irrelevant material. An inverted index accelerates term lookup; scoring and evaluation address which retrieved results are useful. Information retrieval consequently combines data structures, ranking algorithms and evidence about the user's task.

Precision measures the relevant fraction of an information retrieval system's returned documents. For relevant-document set $R$ and nonempty returned-document set $S$, it is $|R\cap S|/|S|$. The denominator concerns what was returned, whereas recall concerns all relevant documents.^[ref]

If $R=\{2,4,6\}$ and $S=\{1,2,3,4,5\}$, two of five returned documents are relevant, giving $2/5=0.4$. Adding irrelevant document seven reduces precision to $2/6$ while leaving recall at $2/3$. This exposes the cost of adding unhelpful results. Returning nothing makes this ratio undefined; a reported score must state any convention used for that case.

Recall measures how much of the relevant material an information retrieval system finds. For a relevant-document set $R$ and returned-document set $S$, it is $|R\cap S|/|R|$, provided $R$ is nonempty.^[ref]

Suppose $R=\{2,4,6\}$ and $S=\{1,2,3,4,5\}$. Two of three relevant documents were found, so recall is $2/3$. Adding document six raises it to one, while precision becomes $3/6$. Returning the whole collection guarantees complete coverage of the assessed relevant set, but can burden the user with many irrelevant results. Estimation requires relevance judgments beyond the returned documents; empty relevant sets need an explicit evaluation convention.

A web search engine performs information retrieval over material gathered from the web. A web crawler discovers or revisits pages, indexing organizes their contents, and query processing selects and ranks results. These are separate stages: a search request ordinarily consults a prepared collection rather than downloading every page afresh.^[ref]

Suppose a newly created page links to a research report. A crawler can discover the report through that link, but discovery does not guarantee successful fetching or inclusion in the searchable index. Even an indexed report may not rank highly for every query. This separation explains why a result list is neither a complete inventory of web content nor a direct measurement of scientific correctness. Link analysis, including the HITS algorithm, can supply ranking evidence alongside textual matching.