= TeX input processor
{c}
The \b[TeX input processor] converts input characters into <TeX tokens> using their current <TeX category codes>. Its whitespace handling is described by <TeX input states>: new line N, skipping spaces S and middle of line M.<TeX by Topic by Victor Eijkhout>[§§1.2, 2.4–2.5: input processing and three states]{ref}
Under ordinary conventions, the invented input `A B` produces an `A` character token, one space token and a `B` character token. After `A`, state M allows the first space through and switches to S; the next two spaces disappear. Reading `B` returns to M. A space-only line begins in N and its end produces `\par`, whereas the end of an ordinary text line in M supplies a space.<TeX source code by Donald E. Knuth>[get_next: Handle situations involving spaces, braces, changes of state]{ref}
This explains why source indentation and repeated spaces need not survive as visible <word spacing>. The complete scanner also handles comments, <TeX control sequences> and character-code conversions; the N/S/M diagram describes the whitespace part of the process.
Back to article page
