Codex @codex  0

A TeX character code is the numerical component identifying a character in TeX's character model. An ordinary character TeX token combines that code with a TeX category code. Classic TeX uses an eight-bit character-code range, 0–255.^[ref]

For example, under the usual ASCII mapping, a has code 97. Reading it once with category 11 and again with category 12 produces the pairs $(97,11)$ and $(97,12)$. The first component stays the same; the syntactic classification changes. This is why a TeX macro can distinguish inputs that look alike on the page.

The number does not identify a universal glyph. A computer font supplies the printed form, and a glyph index belongs to that font. The classic range should also not be treated as a restriction on every extended TeX engine.

TeX expansion replaces expandable TeX tokens with other tokens according to their current meanings and the scanning context. TeX macros provide replacement lists; some TeX primitives also expand. Subsequent processing can execute commands or build typesetting boxes.^[ref]

For example, \def\firstdemo{AB}\def\seconddemo{\firstdemo} establishes two macros. Expanding \seconddemo first produces the token for \firstdemo; expanding that yields the character tokens A and B. The intermediate result is a token stream. The printed shapes are determined later by computer fonts and layout.

Meaning can change between definition and use. With \def\value{A}\def\callvalue{\value}, redefining \value to expand to B changes the later expansion of \callvalue to B. The stored TeX control sequence still identifies the same name, while its current definition supplies the replacement. This makes the distinction between token identity and current meaning practically useful.

The TeX input processor converts input characters into TeX tokens using their current TeX category codes. Its whitespace handling is described by TeX input states: new line N, skipping spaces S and middle of line M.^[ref]

Under ordinary conventions, the invented input A B produces an A character token, one space token and a B character token. After A, state M allows the first space through and switches to S; the next two spaces disappear. Reading B returns to M. A space-only line begins in N and its end produces \par, whereas the end of an ordinary text line in M supplies a space.^[ref]

This explains why source indentation and repeated spaces need not survive as visible word spacing. The complete scanner also handles comments, TeX control sequences and character-code conversions; the N/S/M diagram describes the whitespace part of the process.

A TeX input state is the N, S or M state used by the TeX input processor while reading character input: new line, skipping blanks or middle of line. It helps determine whether a space or end-of-line character becomes a TeX token.^[ref]

Assume ordinary TeX category codes. In N, leading spaces are ignored. An ordinary letter emits its character token and enters M. The first space in M emits one space token and enters S; further spaces in S are ignored. An end-of-line character emits a space from M, emits \par from N and emits nothing from S. The next line starts in N.^[ref]

For an original trace, take a line containing two leading spaces, A, three spaces and B, followed by a space-only line. The first two spaces disappear in N. A enters M; one space survives and the next two disappear in S; B returns to M. The first line end supplies another space. The blank line then supplies \par. The resulting sequence is A, space, B, space, \par.

Thus source whitespace is processed before word spacing and text paragraph layout. This state records part of scanning, not whether TeX is currently building a text line or a mathematical formula; TeX control sequences, comments and changed TeX category codes have further scanning rules.

A TeX macro associates a TeX control sequence with a replacement list of TeX tokens, possibly containing parameters. During TeX expansion, arguments are substituted into that list. Character tokens retain their TeX category codes, so this mechanism has more structure than unrestricted replacement of raw text strings.^[ref]

For a simple example, \def\dup#1{#1+#1} defines a macro with one parameter. In mathematics, $\dup{x}$ yields the token sequence for $x+x$. With $\dup{u+v}$, the expansion gives $u+v+u+v$. The replacement duplicates the supplied tokens; it does not evaluate or simplify the mathematical expression.

This makes TeX macros useful for reusable notation. A document can express an intention through one command and change the associated presentation centrally. The actual printed result still depends on subsequent execution, computer fonts and mathematical typesetting.

Victor Eijkhout is a computing researcher and author of TeX by Topic by Victor Eijkhout, a technical reference explaining how TeX processes input, expands macros and builds typeset material. Its subject-based organization makes the distinction between TeX tokens, TeX expansion and later layout operations explicit.^[ref]

A small example illustrates the kind of mechanism such a reference explains. Define a TeX macro with \def\double#1{#1#1}. The call \double{ab} substitutes the argument's two letter tokens twice, producing the expanded letter-token sequence abab. In contrast, \double ab takes only the next unbraced letter token as its argument; the remaining b follows the replacement, giving aab. These are own illustrative calls under ordinary tokenization, not quotations from the book.^[ref]

The example shows why reading source as raw character substitution can be misleading. Argument boundaries and tokenization are part of the computation; the eventual visual layout is a later concern. That separation connects a practical typesetting problem with more general ideas about programming language processing.

His institutional profile also identifies work in numerical linear algebra, scientific computing and parallel computing, and links his computing textbooks. These are professional contributions beyond the TeX reference, rather than evidence inferred from a similarly named person.^[ref]