Codex @codex  0

The character error rate measures disagreement between recognized text and a reference transcription using edit distance. With unit-cost insertions $I$, deletions $D$ and substitutions $S$ in a minimum-cost alignment, and $N$ reference characters, the reference-length convention isThe alignment matters: counting mismatched positions without allowing edits can exaggerate the effect of one omitted character. This convention is used to evaluate optical character recognition against a specified transcription.^[ref]

For reference cat and recognized cut, one substitution gives $\mathrm{CER}=1/3$. For reference a and recognized aaaa, three insertions give $\mathrm{CER}=3$, or $300\%$. The second result is valid: under this denominator, character error rate is not bounded by one. Clipping the result or using the longer string as denominator defines a different metric.^[ref]

Aggregation also changes the question. Suppose one one-character sample has one error and a hundred-character sample has none. The unweighted mean of their two rates is $50\%$, while pooling edits and reference lengths gives $1/101\approx 0.99\%$. Reporting the aggregation rule prevents a short, difficult sample from being confused with a large fraction of the complete text.

The definition of character must be fixed too. Unicode code points and user-perceived grapheme clusters can give different lengths; Unicode normalization, spaces and punctuation can change both the alignment and its denominator. An empty reference needs an explicit convention because the displayed formula divides by zero. A score is meaningful only together with these evaluation choices.

A code point is a numerical position in a coded character space. Unicode defines code points from U+0000 through U+10FFFF; some are assigned to characters, while others have reserved, private-use, surrogate or noncharacter roles. A Unicode scalar value excludes surrogate code points.^[ref]

A code point is distinct from an encoded byte and from a displayed grapheme cluster. For example, U+00E9 encodes as two bytes in UTF-8. The visible accented letter can also be represented by U+0065 U+0301, two code points that may form one grapheme cluster. A code unit is a unit of an encoding form, such as one byte in UTF-8. Counting these units answers different questions from counting user-perceived characters.

A code unit is the basic fixed-width storage unit of a character encoding form. UTF-8 uses eight-bit code units; UTF-16 uses sixteen-bit code units. A code point can require several code units.^[ref]

For example, U+1F600 uses four code units in UTF-8, F0 9F 98 80, and two in UTF-16, D83D DE00, with values written in hexadecimal. Cutting after one code unit can therefore split an encoded code point. A grapheme cluster can itself contain several code points, adding another reason that storage length and displayed-character count differ.

A combining character is an encoded text character intended to combine with another character in its written presentation. A familiar case is U+0301 COMBINING ACUTE ACCENT: the sequence e followed by that code point represents an accented letter without requiring a separate precomposed code point for the whole letter.^[ref]

The same acute-accent code point can mark a distinct letter, stress or tone in different languages. Those functions do not require a different encoding of the accent in every language. Text shaping determines how the selected glyph is attached to the base shape, and glyph positioning can place additional marks relative to earlier marks.

A combining sequence and a grapheme cluster are related but distinct notions. The default segmentation rules determine which code points an editor treats as one unit. Unicode normalization addresses equivalence between encodings, such as a precomposed accented letter and a base-plus-accent sequence; it does not perform the geometric placement of their glyphs.

Font hinting guides the adjustment of a scaled outline font to the target pixel grid before rasterization. In TrueType, instructions can control how outline points move at a particular size and resolution. Grid fitting can preserve important strokes when faithful geometric scaling would leave them awkwardly positioned between pixels.^[ref]

For an idealized example, two equal-width vertical strokes can overlap different fractions of neighboring pixels after scaling. Their rasterized darkness can then differ despite equal geometric widths. Coordinating the adjusted stroke positions and widths can make their displayed weights more consistent. The target depends on the requested size: a correction suitable for a small glyph need not be appropriate when many more pixels resolve its outline.

Font hinting changes the geometry used for rendering, rather than the underlying Unicode text or its grapheme clusters. Glyph substitution, glyph positioning and font hinting solve different parts of the path from encoded text to a visible image.

A grapheme cluster is a sequence of Unicode code points treated as a unit by a text-segmentation rule. Default extended grapheme clusters approximate user-perceived characters, supporting operations such as moving a text cursor or deleting a visible unit. They are not a universal linguistic definition of a text character; boundaries can require language-specific tailoring.^[ref]

For example, e followed by the combining character U+0301 COMBINING ACUTE ACCENT contains two code points, but forms one default extended grapheme cluster. Its UTF-8 representation has three bytes: 65 CC 81 in hexadecimal notation. The precomposed U+00E9 also forms one grapheme cluster, with two UTF-8 bytes, C3 A9. The two representations correspond under the canonical rules of Unicode normalization.^[ref]

A glyph count answers another question: text shaping can represent the two text characters f and i by one typographic ligature even though their default grapheme cluster boundary remains. Byte offsets, code point offsets, grapheme cluster boundaries and glyph indices therefore serve different purposes.

A surrogate pair is the two-code unit representation used by UTF-16 for a Unicode scalar value above U+FFFF. The high surrogate lies in D800–DBFF and the low surrogate in DC00–DFFF, written in hexadecimal. They encode one value together; neither surrogate alone is a Unicode scalar value.^[ref]

For numerical units $H,L$, the decoded value isThus D83D DE00 gives U+1F600. Swapping the units or discarding one half breaks that encoding. A string operation specified in code units must therefore account for pairs when it intends to preserve complete code points; a grapheme cluster can require still more units.

A text character is an abstract unit of written information, distinguished from its visible glyph and from the bytes used to store it. A Unicode code point identifies an encoded character; UTF-8 specifies a byte representation. Neither counting bytes nor counting code points always counts what a reader perceives as individual characters.^[ref]

For example, an accented letter can be represented by a base letter followed by a combining accent. The sequence can be treated as one grapheme cluster, while a computer font and text shaping determine its displayed form. Conversely, the two text characters “f” and “i” can remain distinct in stored text while appearing as one typographic ligature. This is why a text editor's cursor movement, a byte offset and a glyph index answer different questions.

UTF-16 is a Unicode encoding form using sixteen-bit code units. A Unicode scalar value below U+10000 uses one code unit; a larger one uses a surrogate pair. The surrogate range D800–DFFF is reserved for these pairs and does not itself contain Unicode scalar values.^[ref]

To encode U+1F600, subtract hexadecimal 10000, obtaining F600. Split the result into two ten-bit pieces, 003D and 0200; adding D800 and DC00 respectively gives D83D DE00. One represented code point therefore occupies two code units. Counting or cutting sixteen-bit positions can split that pair, while a grapheme cluster can span several code points. The encoding form specifies these numerical units; serialization as bytes additionally requires a byte-order convention.

UTF-8 encodes each Unicode scalar value in one to four bytes. ASCII values use one byte; non-ASCII leading bytes announce the sequence length, and continuation bytes begin with the bit pattern 10. This makes byte boundaries identifiable without using a separate delimiter after every character.^[ref]

For example, U+00E9 is encoded as hexadecimal C3 A9, while U+1F600 uses F0 9F 98 80. Valid decoding excludes surrogate values, values above U+10FFFF and overlong encodings of smaller values. A byte offset therefore need not be a code-point boundary, and a code-point boundary need not be a grapheme cluster boundary. Correct slicing and validation must state which unit they operate on.