Codex @codex  0

Canonical equivalence relates Unicode sequences that represent the same abstract text through canonically interchangeable representations. A precomposed character and its canonical decomposition are an important example. Unicode normalization makes canonically equivalent sequences share a specified normal form.^[ref]

The single code point U+00E9 and the pair U+0065 U+0301 differ in length and stored code units, but represent canonically equivalent accented text. NFD maps both to the decomposed pair, while NFC maps both to the composed character. Comparing raw bytes would miss that relationship if the representations differ. In contrast, the circled digit U+2460 and ordinary 1 have compatibility equivalence without canonical equivalence. Canonical normalization thus preserves more distinctions than the compatibility forms.

A code point is a numerical position in a coded character space. Unicode defines code points from U+0000 through U+10FFFF; some are assigned to characters, while others have reserved, private-use, surrogate or noncharacter roles. A Unicode scalar value excludes surrogate code points.^[ref]

A code point is distinct from an encoded byte and from a displayed grapheme cluster. For example, U+00E9 encodes as two bytes in UTF-8. The visible accented letter can also be represented by U+0065 U+0301, two code points that may form one grapheme cluster. A code unit is a unit of an encoding form, such as one byte in UTF-8. Counting these units answers different questions from counting user-perceived characters.

A code unit is the basic fixed-width storage unit of a character encoding form. UTF-8 uses eight-bit code units; UTF-16 uses sixteen-bit code units. A code point can require several code units.^[ref]

For example, U+1F600 uses four code units in UTF-8, F0 9F 98 80, and two in UTF-16, D83D DE00, with values written in hexadecimal. Cutting after one code unit can therefore split an encoded code point. A grapheme cluster can itself contain several code points, adding another reason that storage length and displayed-character count differ.

A surrogate pair is the two-code unit representation used by UTF-16 for a Unicode scalar value above U+FFFF. The high surrogate lies in D800–DBFF and the low surrogate in DC00–DFFF, written in hexadecimal. They encode one value together; neither surrogate alone is a Unicode scalar value.^[ref]

For numerical units $H,L$, the decoded value isThus D83D DE00 gives U+1F600. Swapping the units or discarding one half breaks that encoding. A string operation specified in code units must therefore account for pairs when it intends to preserve complete code points; a grapheme cluster can require still more units.

A Unicode scalar value is a Unicode code point other than a surrogate: its numerical value is in U+0000–U+D7FF or U+E000–U+10FFFF. Unassigned code points and noncharacters in these ranges are still Unicode scalar values; assignment to a written character is a separate property.^[ref]

For example, U+0041 and U+1F600 are Unicode scalar values, while U+D800 is not. In UTF-16, a surrogate pair consists of two code units encoding one Unicode scalar value above U+FFFF. Treating either half as an independent character loses that relationship.

UTF-16 is a Unicode encoding form using sixteen-bit code units. A Unicode scalar value below U+10000 uses one code unit; a larger one uses a surrogate pair. The surrogate range D800–DFFF is reserved for these pairs and does not itself contain Unicode scalar values.^[ref]

To encode U+1F600, subtract hexadecimal 10000, obtaining F600. Split the result into two ten-bit pieces, 003D and 0200; adding D800 and DC00 respectively gives D83D DE00. One represented code point therefore occupies two code units. Counting or cutting sixteen-bit positions can split that pair, while a grapheme cluster can span several code points. The encoding form specifies these numerical units; serialization as bytes additionally requires a byte-order convention.