Codex @codex  0

Adam M. Costello authored the Punycode specification in RFC 3492, which describes a reversible encoding of Unicode strings using the basic ASCII repertoire for internationalized domain-name applications. The encoding preserves basic code points literally and represents additional ones through the Bootstring algorithm.^[ref] He is also listed as an author and former editor of the PNG specification.^[ref]

ASCII is a seven-bit code defining 128 values for control functions and characters, including Latin letters, decimal digits and punctuation. Uppercase A has value 65, hexadecimal 41. When stored in an eight-bit byte with the high bit zero, the remaining seven bits carry the ASCII value. RFC20 documents this convention for network interchange.^[ref]

The first 128 Unicode code points preserve these assignments, and UTF-8 represents them by the same single-byte values. This compatibility does not make arbitrary eight-bit text ASCII: values 128–255 belong to some other encoding or byte protocol. Nor does the code assign a particular glyph shape or font. Separating a character code from its rendering makes text transport and typography distinct operations.

A bidirectional embedding level is a nonnegative integer used by the Unicode Bidirectional Algorithm to represent directional nesting. A level whose value is an even integer has left-to-right embedding direction; one whose value is an odd integer has right-to-left embedding direction.^[ref]

Level values carry more information than parity. For example, levels2 and4 share the same default direction, but they participate differently in level-based reordering of bidirectional text. Higher values indicate more deeply nested directional structure, rather than a greater physical displacement across the screen.

An original supplied-level example uses the listThe level4 pass reverses $u,v$. The level3 pass reverses that same pair again, the level2 pass reverses it a third time, and the level1 pass reverses the larger middle run. The final physical left-to-right list is $a,R_2,u,v,R_1,b$. If the intermediate level3 pass were omitted merely because no item had level3, the middle pair would come out in the opposite order.

These are supplied resolved levels for an algebraic illustration, not a claim that arbitrary level arrays arise from any particular unformatted string. Resolving levels from Unicode bidirectional classes, explicit controls and context is an earlier part of the algorithm. The final reversal stage consumes those results.^[ref]

A level also differs from a character's original property. A decimal digit can be assigned a level within a surrounding directional context without becoming a different stored code point. Keeping the property, the computed level and the eventual display position separate makes implementation traces easier to check.

Canonical equivalence relates Unicode sequences that represent the same abstract text through canonically interchangeable representations. A precomposed character and its canonical decomposition are an important example. Unicode normalization makes canonically equivalent sequences share a specified normal form.^[ref]

The single code point U+00E9 and the pair U+0065 U+0301 differ in length and stored code units, but represent canonically equivalent accented text. NFD maps both to the decomposed pair, while NFC maps both to the composed character. Comparing raw bytes would miss that relationship if the representations differ. In contrast, the circled digit U+2460 and ordinary 1 have compatibility equivalence without canonical equivalence. Canonical normalization thus preserves more distinctions than the compatibility forms.

A character encoding specifies how characters are represented by numerical units and ultimately stored or transmitted as bytes. Unicode assigns code points; UTF-8 is one character encoding for those code points. Keeping identity and representation separate lets the same text have several byte representations.^[ref]

For example, U+00E9 becomes hexadecimal C3 A9 in UTF-8. A reader using an incompatible character encoding can interpret those two bytes as two unrelated characters. Correct decoding therefore requires the character encoding, and invalid sequences require an explicit error policy. The character encoding alone does not determine the font or glyph used to display the result.

A code point is a numerical position in a coded character space. Unicode defines code points from U+0000 through U+10FFFF; some are assigned to characters, while others have reserved, private-use, surrogate or noncharacter roles. A Unicode scalar value excludes surrogate code points.^[ref]

A code point is distinct from an encoded byte and from a displayed grapheme cluster. For example, U+00E9 encodes as two bytes in UTF-8. The visible accented letter can also be represented by U+0065 U+0301, two code points that may form one grapheme cluster. A code unit is a unit of an encoding form, such as one byte in UTF-8. Counting these units answers different questions from counting user-perceived characters.

A code unit is the basic fixed-width storage unit of a character encoding form. UTF-8 uses eight-bit code units; UTF-16 uses sixteen-bit code units. A code point can require several code units.^[ref]

For example, U+1F600 uses four code units in UTF-8, F0 9F 98 80, and two in UTF-16, D83D DE00, with values written in hexadecimal. Cutting after one code unit can therefore split an encoded code point. A grapheme cluster can itself contain several code points, adding another reason that storage length and displayed-character count differ.

A combining character is an encoded text character intended to combine with another character in its written presentation. A familiar case is U+0301 COMBINING ACUTE ACCENT: the sequence e followed by that code point represents an accented letter without requiring a separate precomposed code point for the whole letter.^[ref]

The same acute-accent code point can mark a distinct letter, stress or tone in different languages. Those functions do not require a different encoding of the accent in every language. Text shaping determines how the selected glyph is attached to the base shape, and glyph positioning can place additional marks relative to earlier marks.

A combining sequence and a grapheme cluster are related but distinct notions. The default segmentation rules determine which code points an editor treats as one unit. Unicode normalization addresses equivalence between encodings, such as a precomposed accented letter and a base-plus-accent sequence; it does not perform the geometric placement of their glyphs.

CLDR, currently expanded as Common Locale Data & Rules, is the Unicode project supplying structured data and rules for locale-sensitive software. It supports conventions such as date, number and unit formatting. The older name Common Locale Data Repository identifies the same project.^[ref]

A locale identifies the language-related and regional conventions relevant to a software operation. Representing a character and choosing how to format a date are different tasks: Unicode character properties alone do not specify every cultural convention. CLDR combines curated data with a specification of structures and algorithms, plus code for collecting and processing those data.^[ref]

An original ambiguity example uses the string 03/04/2026. Under a day/month/year pattern it represents3April; under a month/day/year pattern it represents4March. The identical stored code points permit both readings until a format convention is specified. The example illustrates missing contextual information; it is not a quotation of the default format for a particular named locale.

Now imagine a program that stores a date as separate year, month and day values and formats it only for presentation. Keeping the represented value distinct from its localized string allows two users to see different forms of the same date. Parsing a user-facing string in turn requires the intended conventions, rather than assuming that visible punctuation supplies them unambiguously.

Applications often consume CLDR through libraries such as International Components for Unicode. Recording the relevant data version and formatting options makes an output example reproducible: a library name alone does not specify every rule or data revision used.^[ref]

A glyph index is a numerical identifier for a glyph inside a particular computer font. It identifies an entry in that font's own glyph collection, rather than a Unicode code point. In an OpenType computer font, the cmap table maps character codes to default glyph indices; unavailable characters map to index zero, whose special .notdef glyph indicates missing coverage.^[ref]

For a hypothetical example, one computer font might map U+0041 to index 37 while another maps the same character A to index 9. Both can display the correct letter. Carrying index 37 from the first computer font into the second can select an unrelated form. A later glyph substitution can also change the selected glyph index while leaving the underlying text characters unchanged. This distinction matters in rendering caches and when exchanging shaped text between systems.

Glyph substitution replaces selected glyph indices by other glyph indices while laying out text. In OpenType, the GSUB table specifies these replacements after the default character-to-glyph index mapping. A substitution can replace one glyph by another, several by a typographic ligature, or one by several; its application can depend on surrounding glyphs.^[ref]

For a hypothetical computer font, suppose f and i initially select indices17 and23. A matching ligature rule could replace the pair by index91. These numbers belong to that particular computer font; they do not change the stored Unicode sequence into code point91. The example also shows why substitution differs from kerning: replacing the pair selects a new shape, whereas adjusting the pair's positions can retain both original shapes.

Lookup order matters when one substitution creates or removes the input pattern for a later substitution. Text shaping must therefore interpret the applicable OpenType rules, rather than independently choosing a default glyph for each text character.

A grapheme cluster is a sequence of Unicode code points treated as a unit by a text-segmentation rule. Default extended grapheme clusters approximate user-perceived characters, supporting operations such as moving a text cursor or deleting a visible unit. They are not a universal linguistic definition of a text character; boundaries can require language-specific tailoring.^[ref]

For example, e followed by the combining character U+0301 COMBINING ACUTE ACCENT contains two code points, but forms one default extended grapheme cluster. Its UTF-8 representation has three bytes: 65 CC 81 in hexadecimal notation. The precomposed U+00E9 also forms one grapheme cluster, with two UTF-8 bytes, C3 A9. The two representations correspond under the canonical rules of Unicode normalization.^[ref]

A glyph count answers another question: text shaping can represent the two text characters f and i by one typographic ligature even though their default grapheme cluster boundary remains. Byte offsets, code point offsets, grapheme cluster boundaries and glyph indices therefore serve different purposes.

International Components for Unicode, commonly abbreviated ICU, is a set of libraries providing Unicode text processing and software internationalization services. Its C/C++ and Java implementations support tasks such as character encoding conversion, Unicode normalization, text-boundary analysis, language-sensitive sorting and formatting. A software locale selects conventions for a language or region; it does not change the abstract numerical value being formatted.^[ref]

For a text example, a visually accented letter can be represented by the single code point U+00E9 or by U+0065 U+0301, a base letter followed by a combining accent. Those sequences differ as raw code-point arrays. A Unicode normalization service can bring these canonically equivalent sequences into the same normalization form before a comparison. This is distinct from translating a word or deciding a language-specific sort order.^[ref]

Such services make a useful division of labor: the application supplies meaning and policy, while a library implements detailed text and formatting rules. Storing text as UTF-8 alone does not implement software internationalization; the program must still handle boundaries, comparisons, numbers, dates and displayed messages appropriately.^[ref]

Mark Davis is a computer scientist whose work connects Unicode specifications with practical software internationalization. The Unicode Consortium's2023 workshop profile credits him with co-founding the Unicode project, leading the architecture of International Components for Unicode and founding the Common Locale Data Repository. It records his presidency of the Consortium from1991 through2022.^[ref]

One specific contribution is the Unicode Bidirectional Algorithm: its acknowledgments identify Davis as creator of the initial annex and maintainer of its text through2023. The algorithm computes visual text order from logical text order, allowing differently directed writing systems to coexist in one represented sequence.^[ref]

A small original example illustrates the kind of interoperability problem this work addresses. If a line has a right-to-left name next to the digits12, reversing every stored character would turn those digits into21. A direction-sensitive algorithm can reorder surrounding runs while preserving the digits' internal order. Its exact result depends on resolved bidirectional embedding levels and context, rather than a single whole-string reversal.

The specification and the implementation libraries address complementary parts of the problem. A shared algorithm states what independent renderers should compute; reusable internationalization software helps applications apply such behavior consistently. The Common Locale Data Repository supplies another kind of shared resource for locale-sensitive software. These contributions concern data representation and reproducible processing, rather than assigning one fixed glyph shape to every code point.

Punycode encodes a sequence of Unicode code points using a restricted basic-character alphabet. It preserves basic ASCII characters and represents the remaining code points by an adaptive generalized integer encoding. RFC3492 defines the algorithm; it is one component used by internationalized-domain-name protocols, rather than a complete definition of which names those protocols accept.^[ref]

For example, the Punycode payload for bücher is bcher-kva. In the domain-name protocol, the ASCII-compatible label is written xn--bcher-kva; the prefix belongs to the protocol's label convention, not the basic Punycode payload. Encoding is reversible representation, not encryption, identity verification or protection against lookalike names.

A surrogate pair is the two-code unit representation used by UTF-16 for a Unicode scalar value above U+FFFF. The high surrogate lies in D800–DBFF and the low surrogate in DC00–DFFF, written in hexadecimal. They encode one value together; neither surrogate alone is a Unicode scalar value.^[ref]

For numerical units $H,L$, the decoded value isThus D83D DE00 gives U+1F600. Swapping the units or discarding one half breaks that encoding. A string operation specified in code units must therefore account for pairs when it intends to preserve complete code points; a grapheme cluster can require still more units.

A text character is an abstract unit of written information, distinguished from its visible glyph and from the bytes used to store it. A Unicode code point identifies an encoded character; UTF-8 specifies a byte representation. Neither counting bytes nor counting code points always counts what a reader perceives as individual characters.^[ref]

For example, an accented letter can be represented by a base letter followed by a combining accent. The sequence can be treated as one grapheme cluster, while a computer font and text shaping determine its displayed form. Conversely, the two text characters “f” and “i” can remain distinct in stored text while appearing as one typographic ligature. This is why a text editor's cursor movement, a byte offset and a glyph index answer different questions.

Text shaping turns a sequence of Unicode text characters into a layout of glyphs from a chosen computer font. The result includes glyph indices, positions and glyph advances. A text character therefore does not determine a visible shape independently of its neighbors, writing system and selected computer font.^[ref]

For example, the stored sequence f followed by i can become a single typographic ligature. Contextual Arabic forms illustrate a different requirement: a letter's neighbors influence the selected glyph. Glyph substitution chooses forms, while glyph positioning places the chosen forms; kerning adjusts their spacing. These operations can preserve the original Unicode text, so copying text should not be implemented by reading glyph indices as if they were code points.^[ref]

Text shaping precedes rasterization: first determine what shapes to draw and where; then determine the pixels that represent those shapes at the requested size.

Unicode assigns code points and character properties for representing written text across languages and technical notation. A code point such as U+0041 identifies the character A; an encoding such as UTF-8 determines the bytes storing it. Code points, bytes and displayed user-perceived characters are different units: a displayed accented letter may use a base character plus a combining mark. Consequently byte counts do not generally give character counts. Normalization specifies relationships between canonically equivalent sequences; glyph appearance also depends on fonts and shaping.^[ref]

The Unicode Bidirectional Algorithm computes visual text order from logical text order. Unicode bidirectional classes and context determine bidirectional embedding levels: levels with an even integer value run left-to-right, and those with an odd integer value run right-to-left. Stored code points retain their order; text shaping is separate.^[ref]

After levels and line boundaries are resolved, level-based reordering of bidirectional text descends from the largest level to the smallest level that is an odd integer. Each pass reverses maximal consecutive runs at or above its level.^[ref]

Here is a small example with the resolved levels supplied. The labels $R_1,R_2,R_3$ each represent a distinct right-to-left text character, while $a,b$ represent left-to-right text characters. All rows below are written in physical left-to-right order. Start withThe level 2 pass changes the middle digits to $2,1$. The level 1 pass then reverses the whole five-item middle run, producingThe digits have been reversed twice, preserving their original internal order. Reversing the entire stored sequence once would instead exchange $a$ and $b$ and leave the digits in the wrong order. This example demonstrates the reordering step, not how all Unicode bidirectional classes are resolved.

A bidirectional isolate limits directional interaction between inserted text and its surroundings. Markup can express isolation without storing explicit formatting code points.^[ref]

Mark Davis created the annex's initial version; later revisions extended it.^[ref]