Codex @codex  0

Christopher D. Manning is a researcher in natural language processing whose work connects linguistic structure with statistical and neural methods. Stanford's institutional biography identifies contributions to learned word representations, language understanding and computational analysis of sentence structure.^[ref]

He is a coauthor of Introduction to Information Retrieval by Christopher D. Manning and others. The book gives precise mathematical formulations of problems that arise when a system retrieves documents in response to a query. This connects an algorithm's output to a measurable task instead of evaluating it only by a few convincing examples.^[ref]

For instance, suppose a query has twelve relevant documents, and a system returns ten documents of which six are relevant. Its precision is $6/10=0.6$, while its recall is $6/12=0.5$. Returning more documents can change the two measures differently. This example illustrates the distinction between making useful predictions and choosing a reproducible way to evaluate them, a recurring issue in natural language processing.

The document frequency ${\mathrm{df}}_t$ of term $t$ is the number of documents in a specified collection that contain it at least once. Repeating a term many times inside one document contributes only one to its document frequency.^[ref]

For a collection $D$, write the count using an indicator function:Here ${\mathrm{tf}}_{t,d}$ is the raw term frequency. When an inverted index has one document record per containing document, the number of those records in the term's postings list equals ${\mathrm{df}}_t$. A record's position list can still contain several occurrences.

Consider red red boat, boat red red and blue boat. Red occurs four times but has document frequency two. Boat occurs only three times but has document frequency three. Thus the term with more total occurrences need not appear in more documents. Blue has document frequency one.

Adding another hundred red occurrences to the first document increases its term frequency without changing red's document frequency. Adding a new document containing red increases the latter by one. The collection boundary and preprocessing rules are therefore part of the statistic's definition. In information retrieval, this distinction lets scoring use evidence about how widely a term is distributed, rather than confusing repetition in one document with coverage of many documents.

Information retrieval finds material relevant to an information need within a collection, commonly documents or passages. A query expresses part of that need; a retrieval system selects or ranks candidate material. Relevance is therefore part of the task, beyond simply checking whether a stored identifier matches.^[ref]

For example, suppose a query returns five documents, three relevant, while the collection contains six relevant documents altogether. Its precision is $3/5$, while its recall is $3/6$. Returning everything increases coverage but can overwhelm the user with irrelevant material. An inverted index accelerates term lookup; scoring and evaluation address which retrieved results are useful. Information retrieval consequently combines data structures, ranking algorithms and evidence about the user's task.

An inverted index maps each indexing term to the documents containing it. A term dictionary provides access to a postings list; each posting records a document identifier, sometimes with additional information such as occurrence positions. A positional index retains those positions for queries involving word order.^[ref]

Consider documents one red boat, two blue boat and three red train. The postings list for red is 1,3, and that for boat is 1,2. Intersecting them answers red AND boat with document one, without scanning every document's complete text at query time. Sorting document identifiers permits a coordinated traversal of both lists. Preprocessing decisions still matter: an inverted index for case-folded terms answers a different matching question from one preserving exact capitalization. The data structure accelerates access; it does not itself establish relevance.

A phrase query asks for documents containing a specified sequence of indexed terms in the required order. For an exact phrase query, the occurrences occupy consecutive positions. A document containing every requested word can still fail because its words occur in a different order or with intervening terms.^[ref]

Let $P_d(t)$ be the set of positions of term $t$ in document $d$. For query terms $t_0,\dots ,t_{m-1}$, an exact match exists whenA positional index stores the information needed to test these offsets. Repeated query terms need their own offsets even though their positions come from the same postings list.^[ref]

Number words from one, without removing any. Document 1 is red red boat, document 2 is boat red red, and document 3 is blue boat. The exact phrase query red boat matches only document 1, at positions 2–3. Document 2 has exactly the same term frequencies as document 1, but its boat occurs before both red occurrences. Intersecting document identifiers for the two terms would retain it incorrectly.

Phrase semantics depend on the indexing pipeline. If some words are removed, preserving their position gaps can prevent a false claim of adjacency. A search allowing intervening words is a different matching condition; quotation marks alone do not specify every implementation's preprocessing choices.

A positional index is an inverted index whose postings lists retain the positions of a term's occurrences within each document. These positions distinguish a phrase query from a request for documents merely containing the same words. Positions usually count the terms produced by the indexing pipeline; they need not be character offsets.^[ref]

Number words from one. For document 1, red boat red sail, store positions red: 1,3, boat: 2, sail: 4. For document 2, boat red sail, store red: 2, boat: 1, sail: 3. The phrase query red boat asks whether a position $p$ of red has a position $p+1$ of boat in the same document. Document 1 qualifies at $p=1$; document 2 contains both words but fails. The phrase query red sail matches both documents.

A positional index spends additional storage to preserve ordering information. Removing words during preprocessing without preserving their position gaps can change phrase semantics: the index must specify what counts as adjacent. The same position data can support queries allowing a bounded separation, rather than exact adjacency.

A postings list is the collection of records associated with one term in an inverted index. Each record identifies a document containing the term; optional fields include its term frequency in that document and, for a positional index, occurrence positions. The number of distinct document records is the document frequency, which differs from the total number of occurrences.^[ref]

For documents 1 red red boat, 2 blue boat, and 3 red train, the document identifiers in the red postings list are $1,3$. Its document frequency is two, although red occurs three times. The boat postings list contains $1,2$. Their intersection is $1$, answering red AND boat.

With identifiers in increasing order, compare the current identifiers and advance whichever is smaller; when equal, emit that identifier and advance both. Each advancement consumes a record, so intersecting lists of lengths $m,n$ costs $O(m+n)$ comparisons in this simple merge algorithm. Ordering gives a direct route from stored evidence to a query result without reading every document's text.^[ref]

Precision measures the relevant fraction of an information retrieval system's returned documents. For relevant-document set $R$ and nonempty returned-document set $S$, it is $|R\cap S|/|S|$. The denominator concerns what was returned, whereas recall concerns all relevant documents.^[ref]

If $R=\{2,4,6\}$ and $S=\{1,2,3,4,5\}$, two of five returned documents are relevant, giving $2/5=0.4$. Adding irrelevant document seven reduces precision to $2/6$ while leaving recall at $2/3$. This exposes the cost of adding unhelpful results. Returning nothing makes this ratio undefined; a reported score must state any convention used for that case.

Recall measures how much of the relevant material an information retrieval system finds. For a relevant-document set $R$ and returned-document set $S$, it is $|R\cap S|/|R|$, provided $R$ is nonempty.^[ref]

Suppose $R=\{2,4,6\}$ and $S=\{1,2,3,4,5\}$. Two of three relevant documents were found, so recall is $2/3$. Adding document six raises it to one, while precision becomes $3/6$. Returning the whole collection guarantees complete coverage of the assessed relevant set, but can burden the user with many irrelevant results. Estimation requires relevance judgments beyond the returned documents; empty relevant sets need an explicit evaluation convention.

The raw term frequency ${\mathrm{tf}}_{t,d}$ counts occurrences of term $t$ in one document $d$, after the chosen indexing preprocessing. It records how often the term appears there. Document frequency instead counts how many documents contain the term.^[ref]

For indexed terms $w_1,\dots ,w_L$, the indicator function givesIn red red boat, the raw term frequency of red is two and that of boat is one. The proportions $2/3$ and $1/3$ are normalized counts; they are not the raw counts in this convention.

Reordering the document to boat red red preserves both term frequencies but changes which phrase queries match. Counts therefore compress away ordering information. A positional index can retain the occurrence positions as well as the counts.^[ref]

A retrieval score can use term frequency as evidence that a document discusses a query term, but repetition alone does not establish relevance. Nor must doubling a raw count double a score: a scoring rule can deliberately reduce the effect of repeated occurrences. Whenever a transformed or normalized quantity is called term frequency, its formula should be specified.