Word segmentation
= Word segmentation
<Word segmentation> identifies the units treated as words in a particular linguistic analysis. Written spaces provide one possible cue, but an orthographic unit need not correspond to one syntactic unit. The boundary convention must therefore be stated before counting words or assigning <syntactic dependencies>.<Tokenization and Word Segmentation in Universal Dependencies>[orthographic tokens versus syntactic words; documented contraction examples]{ref}
The <Universal Dependencies> guidelines analyze the written item `au` as corresponding to two syntactic words, `à` and `le`. Extending that example to `au village`, an orthographic count gives two units, while the syntactic analysis gives three. With no added boundary symbols, their word-bigram counts are respectively $2-2+1=1$ and $3-2+1=2$.
This is a change in representation, not evidence that the utterance became longer. <Word segmentation> affects <n-gram> counts, <lemma (linguistics)> assignment and <linguistic annotation>. A reproducible <text corpus> records how boundaries were chosen and how original written forms map to the analyzed units.