Codex @codex  0

A TeX category code tells the TeX input processor how to treat an input character syntactically. Categories include escape 0, beginning-group 1, ending-group 2, letter 11, other 12 and active 13. The usual backslash and brace conventions are assignments of these categories, rather than immutable properties of the printed characters.^[ref]

For example, suppose a initially has category 11. Defining \def\demo{a} stores that character token. The assignment \catcode 97=12 changes how a subsequently read character with code 97 is classified. It does not change the earlier character token inside \demo. The example assumes the ordinary character-code mapping for a and leaves the letters of demo unchanged.

The distinction matters for reusable TeX macros: a definition can preserve syntax that would be read differently under later category assignments. It also explains how changing letter categories can alter the scanning of a TeX control word without changing the visible spelling of the source.

A TeX input state is the N, S or M state used by the TeX input processor while reading character input: new line, skipping blanks or middle of line. It helps determine whether a space or end-of-line character becomes a TeX token.^[ref]

Assume ordinary TeX category codes. In N, leading spaces are ignored. An ordinary letter emits its character token and enters M. The first space in M emits one space token and enters S; further spaces in S are ignored. An end-of-line character emits a space from M, emits \par from N and emits nothing from S. The next line starts in N.^[ref]

For an original trace, take a line containing two leading spaces, A, three spaces and B, followed by a space-only line. The first two spaces disappear in N. A enters M; one space survives and the next two disappear in S; B returns to M. The first line end supplies another space. The blank line then supplies \par. The resulting sequence is A, space, B, space, \par.

Thus source whitespace is processed before word spacing and text paragraph layout. This state records part of scanning, not whether TeX is currently building a text line or a mathematical formula; TeX control sequences, comments and changed TeX category codes have further scanning rules.

A TeX token is an internal syntactic unit used by TeX. Ordinary character tokens pair a TeX character code with a TeX category code; TeX control sequence tokens identify commands. Macro processing also uses TeX parameter tokens.^[ref]

For example, an a read with category 11 and an a read with category 12 are different character tokens, even though both can select the same printed glyph in ordinary text. Changing a category assignment later does not retokenize a previously stored TeX macro replacement list. A stored control-sequence token can, however, acquire a different current meaning through redefinition.

The TeX input processor performs the relevant lexical analysis. The later TeX expansion stage operates on the resulting tokens. This separation explains why changing the spelling of input and changing the meaning of an existing command are different operations.