Codex @codex  0

A TeX control word is a TeX control sequence scanned from an escape followed by a maximal run of characters with TeX category code 11. The scanner then enters the space-skipping TeX input state S.^[ref]

For example, after \def\word{Q}, the input \word X yields QX; the spaces terminate the name and are skipped. The input \word X does the same. By contrast, \wordX names a single different command, so removing the separator does not preserve the token sequence.

Token identity and current meaning are separate. Define \def\call{\word}, then redefine \word to yield R. Expanding \call now yields R, because its stored command token still names word. This is useful when reading TeX macros: the spelling identifies the command, while its current definition controls expansion.

The TeX input processor converts input characters into TeX tokens using their current TeX category codes. Its whitespace handling is described by TeX input states: new line N, skipping spaces S and middle of line M.^[ref]

Under ordinary conventions, the invented input A B produces an A character token, one space token and a B character token. After A, state M allows the first space through and switches to S; the next two spaces disappear. Reading B returns to M. A space-only line begins in N and its end produces \par, whereas the end of an ordinary text line in M supplies a space.^[ref]

This explains why source indentation and repeated spaces need not survive as visible word spacing. The complete scanner also handles comments, TeX control sequences and character-code conversions; the N/S/M diagram describes the whitespace part of the process.