Word segmentation

Word segmentation identifies the units treated as words in a particular linguistic analysis. Written spaces provide one possible cue, but an orthographic unit need not correspond to one syntactic unit. The boundary convention must therefore be stated before counting words or assigning syntactic dependencies.^[ref]

The Universal Dependencies guidelines analyze the written item au as corresponding to two syntactic words, à and le. Extending that example to au village, an orthographic count gives two units, while the syntactic analysis gives three. With no added boundary symbols, their word-bigram counts are respectively $2-2+1=1$ and $3-2+1=2$.

This is a change in representation, not evidence that the utterance became longer. Word segmentation affects n-gram counts, lemma assignment and linguistic annotation. A reproducible text corpus records how boundaries were chosen and how original written forms map to the analyzed units.