Linguistic annotation

Linguistic annotation adds explicit analytical information to language data, such as word boundaries, part of speech, grammatical number or syntactic dependencies. The original text and the assigned labels are distinct: an annotation records an analysis under stated conventions, rather than being an uninterpreted observation.

In the sentence Birds fly, one analysis assigns the lemma forms bird and fly, the categories noun and verb, and a subject relationship connecting the first word to the second. Universal Dependencies represents such information in CoNLL-U, separating word forms, lemma forms, grammatical features and dependency fields.^[ref]

Keeping those layers separate permits a revised grammatical analysis without silently changing the underlying recording. It also makes disagreement inspectable: two annotators may agree on the text but assign different categories. A reusable annotated text corpus therefore needs its conventions and ambiguous cases documented alongside the labels.

Word segmentation identifies the units treated as words in a particular linguistic analysis. Written spaces provide one possible cue, but an orthographic unit need not correspond to one syntactic unit. The boundary convention must therefore be stated before counting words or assigning syntactic dependencies.^[ref]

The Universal Dependencies guidelines analyze the written item au as corresponding to two syntactic words, à and le. Extending that example to au village, an orthographic count gives two units, while the syntactic analysis gives three. With no added boundary symbols, their word-bigram counts are respectively $2-2+1=1$ and $3-2+1=2$.

This is a change in representation, not evidence that the utterance became longer. Word segmentation affects n-gram counts, lemma assignment and linguistic annotation. A reproducible text corpus records how boundaries were chosen and how original written forms map to the analyzed units.

An interlinear gloss aligns a linguistic example with an analysis of the meanings or grammatical properties of its words and morphemes. It commonly combines an original-language line, a segmented line, analytical labels and a freer translation. The aligned analysis provides information that the translation alone can conceal.

The Leipzig Glossing Rules by Bernard Comrie and others specify word alignment, matching hyphens at segmentable morpheme boundaries, and abbreviated grammatical labels.^[ref] In the original English illustration walk-ed / walk-PST, PST identifies past grammatical tense; the matching hyphens connect the ending with its proposed contribution. The unsegmented surface form is walked.

A matching number of boundaries does not prove the segmentation correct. The rules explicitly distinguish the gloss from the original data: different analyses can describe the same recorded form. An interlinear gloss therefore makes claims about linguistic morphology inspectable while preserving the need for independent evidence about those claims.

CoNLL-U is a plain-text format for exchanging Universal Dependencies linguistic annotation. A word line has ten tab-separated fields: identifier, form, lemma, universal and optional local part of speech, features, basic head, basic relation, enhanced dependencies and miscellaneous information. Blank lines separate sentences; hash-prefixed comments can describe each sentence.^[ref]

An original two-word example, with literal tab separators, is:The first word depends on word two as its subject; word two is attached to the notional root numbered zero. Underscores indicate unspecified values. A saved sentence ends with its blank separator, including at the end of a file.

The specification also requires UTF-8 text with NFC Unicode normalization and LF line endings. Range identifiers represent multiword tokens and decimal identifiers represent empty nodes under additional rules. Consequently, counting ten columns is useful but insufficient for validating every CoNLL-U record.

Universal Dependencies is a framework for comparable grammatical annotation across languages. It combines part of speech labels, morphological features and typed syntactic dependencies, while allowing documented language-specific choices. The framework supplies a common representation rather than a claim that all languages express every distinction identically.^[ref]

For Birds fly, a compatible analysis identifies Birds as a noun with plural grammatical number, gives it the lemma bird, and links it to fly with the subject relation nsubj. These are distinct annotation decisions: changing a lexical convention need not require changing the sentence's dependency structure.

Basic dependencies form a rooted tree, whereas an enhanced representation can contain additional relations and need not remain a tree.^[ref] Keeping the layers explicit helps compare parser output and manually analyzed text corpus resources. A valid interchange file and an empirically justified grammatical analysis remain different achievements.