Codex @codex  0

A TeX control symbol is a TeX control sequence formed by an escape followed by one character whose TeX category code is not letter. Under ordinary conventions, \% is one such command.^[ref]

For an invented example, define \def\!{Q} and then read \! X. The resulting replacement starts with Q, one ordinary space token and X. The nonletter ends the command name immediately; the first following space is processed in state M and survives. Compare \def\word{Q} followed by \word X, which produces QX.

Control space, written as an escape followed by a space, has special handling: it enters state S. Thus a rule that discards every space after every command would predict the wrong result for the \! example.^[ref]

A TeX control word is a TeX control sequence scanned from an escape followed by a maximal run of characters with TeX category code 11. The scanner then enters the space-skipping TeX input state S.^[ref]

For example, after \def\word{Q}, the input \word X yields QX; the spaces terminate the name and are skipped. The input \word X does the same. By contrast, \wordX names a single different command, so removing the separator does not preserve the token sequence.

Token identity and current meaning are separate. Define \def\call{\word}, then redefine \word to yield R. Expanding \call now yields R, because its stored command token still names word. This is useful when reading TeX macros: the spelling identifies the command, while its current definition controls expansion.

TeX expansion replaces expandable TeX tokens with other tokens according to their current meanings and the scanning context. TeX macros provide replacement lists; some TeX primitives also expand. Subsequent processing can execute commands or build typesetting boxes.^[ref]

For example, \def\firstdemo{AB}\def\seconddemo{\firstdemo} establishes two macros. Expanding \seconddemo first produces the token for \firstdemo; expanding that yields the character tokens A and B. The intermediate result is a token stream. The printed shapes are determined later by computer fonts and layout.

Meaning can change between definition and use. With \def\value{A}\def\callvalue{\value}, redefining \value to expand to B changes the later expansion of \callvalue to B. The stored TeX control sequence still identifies the same name, while its current definition supplies the replacement. This makes the distinction between token identity and current meaning practically useful.

The TeX input processor converts input characters into TeX tokens using their current TeX category codes. Its whitespace handling is described by TeX input states: new line N, skipping spaces S and middle of line M.^[ref]

Under ordinary conventions, the invented input A B produces an A character token, one space token and a B character token. After A, state M allows the first space through and switches to S; the next two spaces disappear. Reading B returns to M. A space-only line begins in N and its end produces \par, whereas the end of an ordinary text line in M supplies a space.^[ref]

This explains why source indentation and repeated spaces need not survive as visible word spacing. The complete scanner also handles comments, TeX control sequences and character-code conversions; the N/S/M diagram describes the whitespace part of the process.

A TeX input state is the N, S or M state used by the TeX input processor while reading character input: new line, skipping blanks or middle of line. It helps determine whether a space or end-of-line character becomes a TeX token.^[ref]

Assume ordinary TeX category codes. In N, leading spaces are ignored. An ordinary letter emits its character token and enters M. The first space in M emits one space token and enters S; further spaces in S are ignored. An end-of-line character emits a space from M, emits \par from N and emits nothing from S. The next line starts in N.^[ref]

For an original trace, take a line containing two leading spaces, A, three spaces and B, followed by a space-only line. The first two spaces disappear in N. A enters M; one space survives and the next two disappear in S; B returns to M. The first line end supplies another space. The blank line then supplies \par. The resulting sequence is A, space, B, space, \par.

Thus source whitespace is processed before word spacing and text paragraph layout. This state records part of scanning, not whether TeX is currently building a text line or a mathematical formula; TeX control sequences, comments and changed TeX category codes have further scanning rules.

A TeX macro associates a TeX control sequence with a replacement list of TeX tokens, possibly containing parameters. During TeX expansion, arguments are substituted into that list. Character tokens retain their TeX category codes, so this mechanism has more structure than unrestricted replacement of raw text strings.^[ref]

For a simple example, \def\dup#1{#1+#1} defines a macro with one parameter. In mathematics, $\dup{x}$ yields the token sequence for $x+x$. With $\dup{u+v}$, the expansion gives $u+v+u+v$. The replacement duplicates the supplied tokens; it does not evaluate or simplify the mathematical expression.

This makes TeX macros useful for reusable notation. A document can express an intention through one command and change the associated presentation centrally. The actual printed result still depends on subsequent execution, computer fonts and mathematical typesetting.

A TeX primitive is an operation supplied by TeX itself, such as defining a TeX macro, changing a register or constructing a typesetting box. A TeX control sequence can name a primitive or a macro. The distinction concerns the current meaning of the command, rather than the appearance of its name.^[ref]

For example, \hbox constructs a horizontal typesetting box. The assignment \let\mkbox=\hbox gives another name to its current meaning. Defining a macro with \def\label{sample} creates a replacement list instead. Some primitives participate in TeX expansion, while others perform operations during later processing; primitive does not mean unexpandable. This distinction helps explain why a command can appear in an expansion trace without immediately drawing a glyph.

A TeX token is an internal syntactic unit used by TeX. Ordinary character tokens pair a TeX character code with a TeX category code; TeX control sequence tokens identify commands. Macro processing also uses TeX parameter tokens.^[ref]

For example, an a read with category 11 and an a read with category 12 are different character tokens, even though both can select the same printed glyph in ordinary text. Changing a category assignment later does not retokenize a previously stored TeX macro replacement list. A stored control-sequence token can, however, acquire a different current meaning through redefinition.

The TeX input processor performs the relevant lexical analysis. The later TeX expansion stage operates on the resulting tokens. This separation explains why changing the spelling of input and changing the meaning of an existing command are different operations.