Text corpus

A text corpus is a collection of language samples assembled according to stated criteria for systematic analysis. It can contain written material or transcribed speech; selection, recording conventions and linguistic annotation determine what questions it can answer. A collection's size alone does not make it representative of every speaker or situation.^[ref]

Suppose a toy text corpus contains 20,000 word tokens and 60 occurrences of a chosen form. Its observed rate is $60/20000=0.003$, or 3,000 per million tokens. A second collection needs a comparable denominator before counts are compared. Keeping interview and news samples separately identifiable helps distinguish a difference in language use from a difference in the mixture of text types. A text corpus thus turns examples into countable evidence while preserving the need to justify sampling.

An n-gram is a contiguous ordered sequence of $n$ units under a specified segmentation, commonly words or characters. A one-unit sequence is a unigram and a two-unit sequence a bigram. In a text corpus, counting n-grams retains local ordering that individual-word totals discard.^[ref]

For the word sequence red boat red boat, the three bigram occurrences are red boat, boat red and red boat. Thus red boat accounts for $2/3$ of these bigram windows, although the same sequence occurs twice. For $L$ units and $1\le n\le L$, the number of unpadded windows is $L-n+1$.

The units and denominator are part of the definition of a measured rate. Character bigrams differ from word bigrams, and word segmentation, capitalization and sentence boundaries can change the count. A reproducible analysis states those choices before comparing time series or language varieties.

Linguistic annotation adds explicit analytical information to language data, such as word boundaries, part of speech, grammatical number or syntactic dependencies. The original text and the assigned labels are distinct: an annotation records an analysis under stated conventions, rather than being an uninterpreted observation.

In the sentence Birds fly, one analysis assigns the lemma forms bird and fly, the categories noun and verb, and a subject relationship connecting the first word to the second. Universal Dependencies represents such information in CoNLL-U, separating word forms, lemma forms, grammatical features and dependency fields.^[ref]

Keeping those layers separate permits a revised grammatical analysis without silently changing the underlying recording. It also makes disagreement inspectable: two annotators may agree on the text but assign different categories. A reusable annotated text corpus therefore needs its conventions and ambiguous cases documented alongside the labels.

Word segmentation identifies the units treated as words in a particular linguistic analysis. Written spaces provide one possible cue, but an orthographic unit need not correspond to one syntactic unit. The boundary convention must therefore be stated before counting words or assigning syntactic dependencies.^[ref]

The Universal Dependencies guidelines analyze the written item au as corresponding to two syntactic words, à and le. Extending that example to au village, an orthographic count gives two units, while the syntactic analysis gives three. With no added boundary symbols, their word-bigram counts are respectively $2-2+1=1$ and $3-2+1=2$.

This is a change in representation, not evidence that the utterance became longer. Word segmentation affects n-gram counts, lemma assignment and linguistic annotation. A reproducible text corpus records how boundaries were chosen and how original written forms map to the analyzed units.

An interlinear gloss aligns a linguistic example with an analysis of the meanings or grammatical properties of its words and morphemes. It commonly combines an original-language line, a segmented line, analytical labels and a freer translation. The aligned analysis provides information that the translation alone can conceal.

The Leipzig Glossing Rules by Bernard Comrie and others specify word alignment, matching hyphens at segmentable morpheme boundaries, and abbreviated grammatical labels.^[ref] In the original English illustration walk-ed / walk-PST, PST identifies past grammatical tense; the matching hyphens connect the ending with its proposed contribution. The unsegmented surface form is walked.

A matching number of boundaries does not prove the segmentation correct. The rules explicitly distinguish the gloss from the original data: different analyses can describe the same recorded form. An interlinear gloss therefore makes claims about linguistic morphology inspectable while preserving the need for independent evidence about those claims.

CoNLL-U is a plain-text format for exchanging Universal Dependencies linguistic annotation. A word line has ten tab-separated fields: identifier, form, lemma, universal and optional local part of speech, features, basic head, basic relation, enhanced dependencies and miscellaneous information. Blank lines separate sentences; hash-prefixed comments can describe each sentence.^[ref]

An original two-word example, with literal tab separators, is:The first word depends on word two as its subject; word two is attached to the notional root numbered zero. Underscores indicate unspecified values. A saved sentence ends with its blank separator, including at the end of a file.

The specification also requires UTF-8 text with NFC Unicode normalization and LF line endings. Range identifiers represent multiword tokens and decimal identifiers represent empty nodes under additional rules. Consequently, counting ten columns is useful but insufficient for validating every CoNLL-U record.

Universal Dependencies is a framework for comparable grammatical annotation across languages. It combines part of speech labels, morphological features and typed syntactic dependencies, while allowing documented language-specific choices. The framework supplies a common representation rather than a claim that all languages express every distinction identically.^[ref]

For Birds fly, a compatible analysis identifies Birds as a noun with plural grammatical number, gives it the lemma bird, and links it to fly with the subject relation nsubj. These are distinct annotation decisions: changing a lexical convention need not require changing the sentence's dependency structure.

Basic dependencies form a rooted tree, whereas an enhanced representation can contain additional relations and need not remain a tree.^[ref] Keeping the layers explicit helps compare parser output and manually analyzed text corpus resources. A valid interchange file and an empirically justified grammatical analysis remain different achievements.