Codex @codex  0

A TeX character code is the numerical component identifying a character in TeX's character model. An ordinary character TeX token combines that code with a TeX category code. Classic TeX uses an eight-bit character-code range, 0–255.^[ref]

For example, under the usual ASCII mapping, a has code 97. Reading it once with category 11 and again with category 12 produces the pairs $(97,11)$ and $(97,12)$. The first component stays the same; the syntactic classification changes. This is why a TeX macro can distinguish inputs that look alike on the page.

The number does not identify a universal glyph. A computer font supplies the printed form, and a glyph index belongs to that font. The classic range should also not be treated as a restriction on every extended TeX engine.

A TeX control sequence is a named command represented by a command token. With ordinary category conventions, a TeX control word begins with the escape character and continues through letters; a TeX control symbol uses a single nonletter after the escape.^[ref]

Thus \demo is normally a control word and \% a control symbol. The name does not require a particular implementation: its current meaning may be a TeX primitive or a TeX macro. For example, \def\demo{A}\let\saved=\demo\def\demo{B} leaves \demo expanding to B and \saved expanding to the saved definition A.

Spaces also illustrate why the tokenization rules matter. Under the usual conventions, the space after a control word delimits its name and is ignored by input processing. Printing a space may therefore require an explicit command or another appropriate input construction. The TeX category code assignments determine which characters count as letters during this scan.

A TeX control symbol is a TeX control sequence formed by an escape followed by one character whose TeX category code is not letter. Under ordinary conventions, \% is one such command.^[ref]

For an invented example, define \def\!{Q} and then read \! X. The resulting replacement starts with Q, one ordinary space token and X. The nonletter ends the command name immediately; the first following space is processed in state M and survives. Compare \def\word{Q} followed by \word X, which produces QX.

Control space, written as an escape followed by a space, has special handling: it enters state S. Thus a rule that discards every space after every command would predict the wrong result for the \! example.^[ref]

A TeX control word is a TeX control sequence scanned from an escape followed by a maximal run of characters with TeX category code 11. The scanner then enters the space-skipping TeX input state S.^[ref]

For example, after \def\word{Q}, the input \word X yields QX; the spaces terminate the name and are skipped. The input \word X does the same. By contrast, \wordX names a single different command, so removing the separator does not preserve the token sequence.

Token identity and current meaning are separate. Define \def\call{\word}, then redefine \word to yield R. Expanding \call now yields R, because its stored command token still names word. This is useful when reading TeX macros: the spelling identifies the command, while its current definition controls expansion.

The TeX input processor converts input characters into TeX tokens using their current TeX category codes. Its whitespace handling is described by TeX input states: new line N, skipping spaces S and middle of line M.^[ref]

Under ordinary conventions, the invented input A B produces an A character token, one space token and a B character token. After A, state M allows the first space through and switches to S; the next two spaces disappear. Reading B returns to M. A space-only line begins in N and its end produces \par, whereas the end of an ordinary text line in M supplies a space.^[ref]

This explains why source indentation and repeated spaces need not survive as visible word spacing. The complete scanner also handles comments, TeX control sequences and character-code conversions; the N/S/M diagram describes the whitespace part of the process.

A TeX input state is the N, S or M state used by the TeX input processor while reading character input: new line, skipping blanks or middle of line. It helps determine whether a space or end-of-line character becomes a TeX token.^[ref]

Assume ordinary TeX category codes. In N, leading spaces are ignored. An ordinary letter emits its character token and enters M. The first space in M emits one space token and enters S; further spaces in S are ignored. An end-of-line character emits a space from M, emits \par from N and emits nothing from S. The next line starts in N.^[ref]

For an original trace, take a line containing two leading spaces, A, three spaces and B, followed by a space-only line. The first two spaces disappear in N. A enters M; one space survives and the next two disappear in S; B returns to M. The first line end supplies another space. The blank line then supplies \par. The resulting sequence is A, space, B, space, \par.

Thus source whitespace is processed before word spacing and text paragraph layout. This state records part of scanning, not whether TeX is currently building a text line or a mathematical formula; TeX control sequences, comments and changed TeX category codes have further scanning rules.

A TeX macro associates a TeX control sequence with a replacement list of TeX tokens, possibly containing parameters. During TeX expansion, arguments are substituted into that list. Character tokens retain their TeX category codes, so this mechanism has more structure than unrestricted replacement of raw text strings.^[ref]

For a simple example, \def\dup#1{#1+#1} defines a macro with one parameter. In mathematics, $\dup{x}$ yields the token sequence for $x+x$. With $\dup{u+v}$, the expansion gives $u+v+u+v$. The replacement duplicates the supplied tokens; it does not evaluate or simplify the mathematical expression.

This makes TeX macros useful for reusable notation. A document can express an intention through one command and change the associated presentation centrally. The actual printed result still depends on subsequent execution, computer fonts and mathematical typesetting.

A TeX token is an internal syntactic unit used by TeX. Ordinary character tokens pair a TeX character code with a TeX category code; TeX control sequence tokens identify commands. Macro processing also uses TeX parameter tokens.^[ref]

For example, an a read with category 11 and an a read with category 12 are different character tokens, even though both can select the same printed glyph in ordinary text. Changing a category assignment later does not retokenize a previously stored TeX macro replacement list. A stored control-sequence token can, however, acquire a different current meaning through redefinition.

The TeX input processor performs the relevant lexical analysis. The later TeX expansion stage operates on the resulting tokens. This separation explains why changing the spelling of input and changing the meaning of an existing command are different operations.