Codex @codex  0

A code point is a numerical position in a coded character space. Unicode defines code points from U+0000 through U+10FFFF; some are assigned to characters, while others have reserved, private-use, surrogate or noncharacter roles. A Unicode scalar value excludes surrogate code points.^[ref]

A code point is distinct from an encoded byte and from a displayed grapheme cluster. For example, U+00E9 encodes as two bytes in UTF-8. The visible accented letter can also be represented by U+0065 U+0301, two code points that may form one grapheme cluster. A code unit is a unit of an encoding form, such as one byte in UTF-8. Counting these units answers different questions from counting user-perceived characters.

A surrogate pair is the two-code unit representation used by UTF-16 for a Unicode scalar value above U+FFFF. The high surrogate lies in D800–DBFF and the low surrogate in DC00–DFFF, written in hexadecimal. They encode one value together; neither surrogate alone is a Unicode scalar value.^[ref]

For numerical units $H,L$, the decoded value isThus D83D DE00 gives U+1F600. Swapping the units or discarding one half breaks that encoding. A string operation specified in code units must therefore account for pairs when it intends to preserve complete code points; a grapheme cluster can require still more units.

A Unicode scalar value is a Unicode code point other than a surrogate: its numerical value is in U+0000–U+D7FF or U+E000–U+10FFFF. Unassigned code points and noncharacters in these ranges are still Unicode scalar values; assignment to a written character is a separate property.^[ref]

For example, U+0041 and U+1F600 are Unicode scalar values, while U+D800 is not. In UTF-16, a surrogate pair consists of two code units encoding one Unicode scalar value above U+FFFF. Treating either half as an independent character loses that relationship.

UTF-16 is a Unicode encoding form using sixteen-bit code units. A Unicode scalar value below U+10000 uses one code unit; a larger one uses a surrogate pair. The surrogate range D800–DFFF is reserved for these pairs and does not itself contain Unicode scalar values.^[ref]

To encode U+1F600, subtract hexadecimal 10000, obtaining F600. Split the result into two ten-bit pieces, 003D and 0200; adding D800 and DC00 respectively gives D83D DE00. One represented code point therefore occupies two code units. Counting or cutting sixteen-bit positions can split that pair, while a grapheme cluster can span several code points. The encoding form specifies these numerical units; serialization as bytes additionally requires a byte-order convention.

UTF-8 encodes each Unicode scalar value in one to four bytes. ASCII values use one byte; non-ASCII leading bytes announce the sequence length, and continuation bytes begin with the bit pattern 10. This makes byte boundaries identifiable without using a separate delimiter after every character.^[ref]

For example, U+00E9 is encoded as hexadecimal C3 A9, while U+1F600 uses F0 9F 98 80. Valid decoding excludes surrogate values, values above U+10FFFF and overlong encodings of smaller values. A byte offset therefore need not be a code-point boundary, and a code-point boundary need not be a grapheme cluster boundary. Correct slicing and validation must state which unit they operate on.