Codex @codex  0

Canonical equivalence relates Unicode sequences that represent the same abstract text through canonically interchangeable representations. A precomposed character and its canonical decomposition are an important example. Unicode normalization makes canonically equivalent sequences share a specified normal form.^[ref]

The single code point U+00E9 and the pair U+0065 U+0301 differ in length and stored code units, but represent canonically equivalent accented text. NFD maps both to the decomposed pair, while NFC maps both to the composed character. Comparing raw bytes would miss that relationship if the representations differ. In contrast, the circled digit U+2460 and ordinary 1 have compatibility equivalence without canonical equivalence. Canonical normalization thus preserves more distinctions than the compatibility forms.

The character error rate measures disagreement between recognized text and a reference transcription using edit distance. With unit-cost insertions $I$, deletions $D$ and substitutions $S$ in a minimum-cost alignment, and $N$ reference characters, the reference-length convention isThe alignment matters: counting mismatched positions without allowing edits can exaggerate the effect of one omitted character. This convention is used to evaluate optical character recognition against a specified transcription.^[ref]

For reference cat and recognized cut, one substitution gives $\mathrm{CER}=1/3$. For reference a and recognized aaaa, three insertions give $\mathrm{CER}=3$, or $300\%$. The second result is valid: under this denominator, character error rate is not bounded by one. Clipping the result or using the longer string as denominator defines a different metric.^[ref]

Aggregation also changes the question. Suppose one one-character sample has one error and a hundred-character sample has none. The unweighted mean of their two rates is $50\%$, while pooling edits and reference lengths gives $1/101\approx 0.99\%$. Reporting the aggregation rule prevents a short, difficult sample from being confused with a large fraction of the complete text.

The definition of character must be fixed too. Unicode code points and user-perceived grapheme clusters can give different lengths; Unicode normalization, spaces and punctuation can change both the alignment and its denominator. An empty reference needs an explicit convention because the displayed formula divides by zero. A score is meaningful only together with these evaluation choices.

A combining character is an encoded text character intended to combine with another character in its written presentation. A familiar case is U+0301 COMBINING ACUTE ACCENT: the sequence e followed by that code point represents an accented letter without requiring a separate precomposed code point for the whole letter.^[ref]

The same acute-accent code point can mark a distinct letter, stress or tone in different languages. Those functions do not require a different encoding of the accent in every language. Text shaping determines how the selected glyph is attached to the base shape, and glyph positioning can place additional marks relative to earlier marks.

A combining sequence and a grapheme cluster are related but distinct notions. The default segmentation rules determine which code points an editor treats as one unit. Unicode normalization addresses equivalence between encodings, such as a precomposed accented letter and a base-plus-accent sequence; it does not perform the geometric placement of their glyphs.

Compatibility equivalence relates Unicode sequences whose compatibility decompositions agree, including distinctions that may matter in some contexts. It includes canonical matches but can additionally identify characters differing in formatting or conventional presentation. Unicode normalization forms NFKD and NFKC use these broader decompositions.^[ref]

For example, the circled digit U+2460 and ordinary digit U+0031 both normalize to 1 under NFKC. NFC retains their distinction. A circled list marker and an ordinary numeral can therefore become indistinguishable under compatibility normalization, even when an application wanted to preserve the original notation. The choice is a representation policy, not a test of visual similarity: applying compatibility normalization requires deciding which distinctions the task may safely discard.

CoNLL-U is a plain-text format for exchanging Universal Dependencies linguistic annotation. A word line has ten tab-separated fields: identifier, form, lemma, universal and optional local part of speech, features, basic head, basic relation, enhanced dependencies and miscellaneous information. Blank lines separate sentences; hash-prefixed comments can describe each sentence.^[ref]

An original two-word example, with literal tab separators, is:The first word depends on word two as its subject; word two is attached to the notional root numbered zero. Underscores indicate unspecified values. A saved sentence ends with its blank separator, including at the end of a file.

The specification also requires UTF-8 text with NFC Unicode normalization and LF line endings. Range identifiers represent multiword tokens and decimal identifiers represent empty nodes under additional rules. Consequently, counting ten columns is useful but insufficient for validating every CoNLL-U record.

A grapheme cluster is a sequence of Unicode code points treated as a unit by a text-segmentation rule. Default extended grapheme clusters approximate user-perceived characters, supporting operations such as moving a text cursor or deleting a visible unit. They are not a universal linguistic definition of a text character; boundaries can require language-specific tailoring.^[ref]

For example, e followed by the combining character U+0301 COMBINING ACUTE ACCENT contains two code points, but forms one default extended grapheme cluster. Its UTF-8 representation has three bytes: 65 CC 81 in hexadecimal notation. The precomposed U+00E9 also forms one grapheme cluster, with two UTF-8 bytes, C3 A9. The two representations correspond under the canonical rules of Unicode normalization.^[ref]

A glyph count answers another question: text shaping can represent the two text characters f and i by one typographic ligature even though their default grapheme cluster boundary remains. Byte offsets, code point offsets, grapheme cluster boundaries and glyph indices therefore serve different purposes.

International Components for Unicode, commonly abbreviated ICU, is a set of libraries providing Unicode text processing and software internationalization services. Its C/C++ and Java implementations support tasks such as character encoding conversion, Unicode normalization, text-boundary analysis, language-sensitive sorting and formatting. A software locale selects conventions for a language or region; it does not change the abstract numerical value being formatted.^[ref]

For a text example, a visually accented letter can be represented by the single code point U+00E9 or by U+0065 U+0301, a base letter followed by a combining accent. Those sequences differ as raw code-point arrays. A Unicode normalization service can bring these canonically equivalent sequences into the same normalization form before a comparison. This is distinct from translating a word or deciding a language-specific sort order.^[ref]

Such services make a useful division of labor: the application supplies meaning and policy, while a library implements detailed text and formatting rules. Storing text as UTF-8 alone does not implement software internationalization; the program must still handle boundaries, comparisons, numbers, dates and displayed messages appropriately.^[ref]

Unicode normalization maps text to a specified standard representation while respecting a chosen equivalence relation between Unicode sequences. Canonical normalization uses canonical equivalence; compatibility normalization additionally uses compatibility equivalence. NFD decomposes canonical combinations, NFC then composes where permitted, and NFKD/NFKC additionally apply compatibility decompositions.^[ref]

For example, U+00E9 and U+0065 U+0301 represent canonically equivalent accented text. NFD produces the second sequence from either input, while NFC produces the first. NFKC also changes the circled digit U+2460 to the ordinary digit U+0031; NFC preserves that compatibility distinction. The choice consequently depends on which distinctions an application intends to retain. Unicode normalization is different from choosing a character encoding or a font, and it does not make every visually similar pair of strings equal.