Unicode assigns code points and character properties for representing written text across languages and technical notation. A code point such as U+0041 identifies the character A; an encoding such as UTF-8 determines the bytes storing it. Code points, bytes and displayed user-perceived characters are different units: a displayed accented letter may use a base character plus a combining mark. Consequently byte counts do not generally give character counts. Normalization specifies relationships between canonically equivalent sequences; glyph appearance also depends on fonts and shaping.[ref]
CLDR, currently expanded as Common Locale Data & Rules, is the Unicode project supplying structured data and rules for locale-sensitive software. It supports conventions such as date, number and unit formatting. The older name Common Locale Data Repository identifies the same project.[ref]
A locale identifies the language-related and regional conventions relevant to a software operation. Representing a character and choosing how to format a date are different tasks: Unicode character properties alone do not specify every cultural convention. CLDR combines curated data with a specification of structures and algorithms, plus code for collecting and processing those data.[ref]
An original ambiguity example uses the string 03/04/2026. Under a day/month/year pattern it represents3April; under a month/day/year pattern it represents4March. The identical stored code points permit both readings until a format convention is specified. The example illustrates missing contextual information; it is not a quotation of the default format for a particular named locale.
Now imagine a program that stores a date as separate year, month and day values and formats it only for presentation. Keeping the represented value distinct from its localized string allows two users to see different forms of the same date. Parsing a user-facing string in turn requires the intended conventions, rather than assuming that visible punctuation supplies them unambiguously.
Applications often consume CLDR through libraries such as International Components for Unicode. Recording the relevant data version and formatting options makes an output example reproducible: a library name alone does not specify every rule or data revision used.[ref]
The Unicode Bidirectional Algorithm computes visual text order from logical text order. Unicode bidirectional classes and context determine bidirectional embedding levels: levels with an even integer value run left-to-right, and those with an odd integer value run right-to-left. Stored code points retain their order; text shaping is separate.[ref]
After levels and line boundaries are resolved, level-based reordering of bidirectional text descends from the largest level to the smallest level that is an odd integer. Each pass reverses maximal consecutive runs at or above its level.[ref]
Here is a small example with the resolved levels supplied. The labels each represent a distinct right-to-left text character, while represent left-to-right text characters. All rows below are written in physical left-to-right order. Start with
The level 2 pass changes the middle digits to . The level 1 pass then reverses the whole five-item middle run, producing
The digits have been reversed twice, preserving their original internal order. Reversing the entire stored sequence once would instead exchange and and leave the digits in the wrong order. This example demonstrates the reordering step, not how all Unicode bidirectional classes are resolved.
Figure 1.
Two level-based reversals preserve the internal order of the digits
. Original illustration with supplied bidirectional embedding levels. The boxes denote individual abstract text characters, not literal multi-character labels. Details.
A bidirectional isolate limits directional interaction between inserted text and its surroundings. Markup can express isolation without storing explicit formatting code points.[ref]
Mark Davis created the annex's initial version; later revisions extended it.[ref]
A Unicode directional control is a formatting code point that affects bidirectional processing without contributing an ordinary visible letter. Directional marks, embeddings, overrides and isolate delimiters provide different mechanisms within the Unicode Bidirectional Algorithm. They affect interpretation of the stored sequence rather than supplying new letter shapes.[ref]
Left-to-right and right-to-left marks, LRM (U+200E) and RLM (U+200F), behave as strong directional items while remaining invisible. A right-to-left isolate initiator, RLI (U+2067), instead starts a bounded bidirectional isolate; PDI (U+2069) terminates it. A mark and a span delimiter therefore carry different structural information.[ref]
An original counting example documents controls by their numerical values. The sequence U+2066 U+0061 U+0062 U+2069 contains LRI, the two Latin letters a and b, and PDI. It contains four code points and eight UTF-8 bytes: three for each delimiter and one for each Latin letter. Its ordinary visible letters are just a and b. Counting visible letters would therefore lose information about the represented sequence.
This notation intentionally shows the control values instead of placing hidden directional formatting characters in the source example. To reproduce an actual layout test, record its logical text order, paragraph direction, Unicode version and line boundaries, then compare the computed bidirectional embedding levels and visual text order.
A higher-level format can express equivalent directional structure through markup. The control-code-point representation and the markup representation need not have the same stored characters even when they request corresponding layout behavior. The specification defines those correspondences, while text shaping remains a further part of rendering.[ref]
Level-based reordering of bidirectional text is the final positional reversal procedure of the Unicode Bidirectional Algorithm after directional levels and line boundaries have been resolved. RuleL2 processes thresholds from the largest level down through the smallest level whose value is an odd integer. Each pass reverses every maximal consecutive run whose levels meet or exceed the threshold.[ref]
An original trace supplies the resolved levels rather than attempting to infer them. Take the logical items
At threshold2, reverse only the two middle digits, obtaining . At threshold1, reverse the entire middle five-item run, giving
The digits underwent two reversals, preserving their internal order. The labels each denote one abstract right-to-left text character; every array is written in physical left-to-right order.
Figure 1.
Descending level passes reverse nested runs and restore the internal order of digits
. Reused supplied-level illustration. It demonstrates ruleL2, rather than a complete implementation that resolves levels, shapes glyphs or chooses line boundaries. Details.
Thresholds include intermediate values that no item happens to have. For example, a pair at level4 surrounded by level1 items participates in passes4,3,2 and then the enclosing pass1. Omitting pass3 changes the result, as the bidirectional embedding level example demonstrates.
An implementation may carry levels with the items, as the figure generator does. It can also locate each reversal range from the original fixed level array: earlier passes permute items only inside higher-level runs, preserving the boundaries of every later lower-threshold run. Keeping logical indices on the displayed items supplies the eventual position map. Line-final whitespace resetting belongs to ruleL1, and combining-mark handling and mirroring require their separate rules.[ref]
A bidirectional isolate is a span treated as a directional unit by the Unicode Bidirectional Algorithm. Its internal text is resolved separately from the surrounding text; the isolate as a whole behaves neutrally in its surroundings. This limits directional interaction when inserting a span whose contents have their own writing direction.[ref]
Unicode directional controls can delimit the span: LRI (U+2066) chooses left-to-right isolation, RLI (U+2067) chooses right-to-left isolation, FSI (U+2068) infers direction from the first relevant strong character, and PDI (U+2069) terminates the isolate. Higher-level markup can express equivalent isolation without storing those explicit control characters.[ref]
An original record-layout example has three fields: a fixed prefix, a user-supplied name and a fixed suffix containing punctuation and an identifier. Treat the inserted name as one isolated span. The outer layout then works with that span as a unit, while its contents receive their own directional processing. A name containing both letters and digits still needs internal reordering; isolation does not prescribe a reversal of every character inside it.
For a reproducible plain-text notation, write the sequence as prefix RLI payload PDI suffix, with the uppercase labels standing for control code points rather than literal letters in the displayed text. This notation documents the boundary without hiding directional controls in the source example. An actual test should record the concrete payload, paragraph direction, Unicode version and line boundaries as well.
Nesting introduces another distinction: a first-strong isolate determines its direction from an eligible strong character outside nested isolates. A direction already enclosed in a nested span need not choose the enclosing span's direction. Naming the span boundary and the rule used to choose its direction makes this behavior explicit.[ref]
A bidirectional embedding level is a nonnegative integer used by the Unicode Bidirectional Algorithm to represent directional nesting. A level whose value is an even integer has left-to-right embedding direction; one whose value is an odd integer has right-to-left embedding direction.[ref]
Level values carry more information than parity. For example, levels2 and4 share the same default direction, but they participate differently in level-based reordering of bidirectional text. Higher values indicate more deeply nested directional structure, rather than a greater physical displacement across the screen.
An original supplied-level example uses the list
The level4 pass reverses . The level3 pass reverses that same pair again, the level2 pass reverses it a third time, and the level1 pass reverses the larger middle run. The final physical left-to-right list is . If the intermediate level3 pass were omitted merely because no item had level3, the middle pair would come out in the opposite order.
These are supplied resolved levels for an algebraic illustration, not a claim that arbitrary level arrays arise from any particular unformatted string. Resolving levels from Unicode bidirectional classes, explicit controls and context is an earlier part of the algorithm. The final reversal stage consumes those results.[ref]
A level also differs from a character's original property. A decimal digit can be assigned a level within a surrounding directional context without becoming a different stored code point. Keeping the property, the computed level and the eventual display position separate makes implementation traces easier to check.
A Unicode bidirectional class is a character property, formally named Bidi_Class, used as input to the Unicode Bidirectional Algorithm. It classifies directional behavior; the algorithm combines the properties with context to compute bidirectional embedding levels. A class is not itself a final screen position.[ref]
For a compact example, Latin A (U+0041) has classL, Hebrew alef (U+05D0) has classR, digit1 (U+0031) has classEN, and ordinary space (U+0020) has classWS. These labels distinguish strong directions, a numeric class and whitespace. They carry information that a single left-to-right/right-to-left Boolean would lose.
An original data-structure example keeps the stages separate. Represent each stored item by a record containing its logical index, code point, original Bidi_Class, working directional type, resolved level and eventual visual index. A direction-resolution step can update the working fields while leaving the stored code point and original property intact. A later level-based reordering of bidirectional text step changes position mappings rather than assigning new code points.
The advantage appears in debugging a digit within mixed-direction text. Its original classEN identifies a numeric input. Its resolved level describes the computed directional context, and its visual index identifies a placement in one line. Reporting only the visual index would conceal whether a mistake arose during classification, context resolution or final reordering.
The normative property data are supplied by the Unicode character database; broad descriptions such as “a letter in this script” have exceptions. An implementation should use the data for its chosen Unicode version rather than infer every character's class from appearance.[ref]
Visual text order is the ordering of displayed text items within a rendered line. In bidirectional text it can differ from logical text order, the stored character sequence. It describes a presentation result, so the line boundaries and resolved directional information belong to its context.[ref]
For an original permutation example, take six logical items with supplied levels . The resulting left-to-right display positions contain source indices
Thus the displayed items are . The digits keep their internal order even though the surrounding right-to-left items exchange physical positions. A single reversal of the whole stored list would give a different result.
A position map preserves information that a picture alone may lose. If two source items both display as the same letter, their appearances do not reveal which logical index each came from. A renderer can retain distinct indices while drawing identical shapes. Likewise, interpreting a pointer position as a text location requires more information than simply treating a visual column number as a logical index.
The Unicode Bidirectional Algorithm orders text items, while text shaping determines additional glyph forms and combinations. Distinguishing these stages prevents a permutation of character positions from being mistaken for a complete specification of glyph geometry.[ref]
Visual order is therefore useful for describing what appears on a particular line, while logical text order supplies stable positions for the represented sequence. Reporting both orders and their mapping makes an example reproducible without assuming that the reader's own browser will display a mixed-direction string in the intended way.
Logical text order is the order in which text characters are represented in a stored sequence for interpretation. A bidirectional renderer derives visual text order from that sequence; it does not require the stored characters to be prearranged in screen-position order.[ref]
An original indexing example uses six abstract items. Write the stored positions in order as
The labels each stand for a single right-to-left text character. All displayed arrays here are read physically from left to right. Applying level-based reordering of bidirectional text to these supplied levels gives the visual list , whose source-position list is . Stored position1 remains position1 even though its item is drawn near the right end.
This distinction makes a text-processing task precise. An operation on logical positions1 through4 selects the contiguous stored subsequence . A renderer may need a position map to relate that range to its displayed cells. Sorting characters by screen coordinate before searching or serializing the text would change the sequence being processed.
Logical order does not determine a font's glyph geometry. Text shaping, line layout and the Unicode Bidirectional Algorithm contribute different information. For example, a different line break can change which items are reordered together while the underlying stored sequence remains the same.[ref]
UTF-16 is a Unicode encoding form using sixteen-bit code units. A Unicode scalar value below U+10000 uses one code unit; a larger one uses a surrogate pair. The surrogate range D800–DFFF is reserved for these pairs and does not itself contain Unicode scalar values.[ref]
To encode U+1F600, subtract hexadecimal 10000, obtaining F600. Split the result into two ten-bit pieces, 003D and 0200; adding D800 and DC00 respectively gives D83D DE00. One represented code point therefore occupies two code units. Counting or cutting sixteen-bit positions can split that pair, while a grapheme cluster can span several code points. The encoding form specifies these numerical units; serialization as bytes additionally requires a byte-order convention.
A surrogate pair is the two-code unit representation used by UTF-16 for a Unicode scalar value above U+FFFF. The high surrogate lies in D800–DBFF and the low surrogate in DC00–DFFF, written in hexadecimal. They encode one value together; neither surrogate alone is a Unicode scalar value.[ref]
For numerical units , the decoded value is
Thus D83D DE00 gives U+1F600. Swapping the units or discarding one half breaks that encoding. A string operation specified in code units must therefore account for pairs when it intends to preserve complete code points; a grapheme cluster can require still more units.
Unicode normalization maps text to a specified standard representation while respecting a chosen equivalence relation between Unicode sequences. Canonical normalization uses canonical equivalence; compatibility normalization additionally uses compatibility equivalence. NFD decomposes canonical combinations, NFC then composes where permitted, and NFKD/NFKC additionally apply compatibility decompositions.[ref]
For example, U+00E9 and U+0065 U+0301 represent canonically equivalent accented text. NFD produces the second sequence from either input, while NFC produces the first. NFKC also changes the circled digit U+2460 to the ordinary digit U+0031; NFC preserves that compatibility distinction. The choice consequently depends on which distinctions an application intends to retain. Unicode normalization is different from choosing a character encoding or a font, and it does not make every visually similar pair of strings equal.
Compatibility equivalence relates Unicode sequences whose compatibility decompositions agree, including distinctions that may matter in some contexts. It includes canonical matches but can additionally identify characters differing in formatting or conventional presentation. Unicode normalization forms NFKD and NFKC use these broader decompositions.[ref]
For example, the circled digit U+2460 and ordinary digit U+0031 both normalize to 1 under NFKC. NFC retains their distinction. A circled list marker and an ordinary numeral can therefore become indistinguishable under compatibility normalization, even when an application wanted to preserve the original notation. The choice is a representation policy, not a test of visual similarity: applying compatibility normalization requires deciding which distinctions the task may safely discard.
Canonical equivalence relates Unicode sequences that represent the same abstract text through canonically interchangeable representations. A precomposed character and its canonical decomposition are an important example. Unicode normalization makes canonically equivalent sequences share a specified normal form.[ref]
The single code point U+00E9 and the pair U+0065 U+0301 differ in length and stored code units, but represent canonically equivalent accented text. NFD maps both to the decomposed pair, while NFC maps both to the composed character. Comparing raw bytes would miss that relationship if the representations differ. In contrast, the circled digit U+2460 and ordinary 1 have compatibility equivalence without canonical equivalence. Canonical normalization thus preserves more distinctions than the compatibility forms.
A code unit is the basic fixed-width storage unit of a character encoding form. UTF-8 uses eight-bit code units; UTF-16 uses sixteen-bit code units. A code point can require several code units.[ref]
For example, U+1F600 uses four code units in UTF-8, F0 9F 98 80, and two in UTF-16, D83D DE00, with values written in hexadecimal. Cutting after one code unit can therefore split an encoded code point. A grapheme cluster can itself contain several code points, adding another reason that storage length and displayed-character count differ.
A Unicode scalar value is a Unicode code point other than a surrogate: its numerical value is in U+0000–U+D7FF or U+E000–U+10FFFF. Unassigned code points and noncharacters in these ranges are still Unicode scalar values; assignment to a written character is a separate property.[ref]
For example, U+0041 and U+1F600 are Unicode scalar values, while U+D800 is not. In UTF-16, a surrogate pair consists of two code units encoding one Unicode scalar value above U+FFFF. Treating either half as an independent character loses that relationship.
A character encoding specifies how characters are represented by numerical units and ultimately stored or transmitted as bytes. Unicode assigns code points; UTF-8 is one character encoding for those code points. Keeping identity and representation separate lets the same text have several byte representations.[ref]
For example, U+00E9 becomes hexadecimal C3 A9 in UTF-8. A reader using an incompatible character encoding can interpret those two bytes as two unrelated characters. Correct decoding therefore requires the character encoding, and invalid sequences require an explicit error policy. The character encoding alone does not determine the font or glyph used to display the result.
ASCII is a seven-bit code defining 128 values for control functions and characters, including Latin letters, decimal digits and punctuation. Uppercase A has value 65, hexadecimal 41. When stored in an eight-bit byte with the high bit zero, the remaining seven bits carry the ASCII value. RFC20 documents this convention for network interchange.[ref]
The first 128 Unicode code points preserve these assignments, and UTF-8 represents them by the same single-byte values. This compatibility does not make arbitrary eight-bit text ASCII: values 128–255 belong to some other encoding or byte protocol. Nor does the code assign a particular glyph shape or font. Separating a character code from its rendering makes text transport and typography distinct operations.
A code point is a numerical position in a coded character space. Unicode defines code points from U+0000 through U+10FFFF; some are assigned to characters, while others have reserved, private-use, surrogate or noncharacter roles. A Unicode scalar value excludes surrogate code points.[ref]
A code point is distinct from an encoded byte and from a displayed grapheme cluster. For example, U+00E9 encodes as two bytes in UTF-8. The visible accented letter can also be represented by U+0065 U+0301, two code points that may form one grapheme cluster. A code unit is a unit of an encoding form, such as one byte in UTF-8. Counting these units answers different questions from counting user-perceived characters.
Punycode encodes a sequence of Unicode code points using a restricted basic-character alphabet. It preserves basic ASCII characters and represents the remaining code points by an adaptive generalized integer encoding. RFC3492 defines the algorithm; it is one component used by internationalized-domain-name protocols, rather than a complete definition of which names those protocols accept.[ref]
For example, the Punycode payload for bücher is bcher-kva. In the domain-name protocol, the ASCII-compatible label is written xn--bcher-kva; the prefix belongs to the protocol's label convention, not the basic Punycode payload. Encoding is reversible representation, not encryption, identity verification or protection against lookalike names.
UTF-8 encodes each Unicode scalar value in one to four bytes. ASCII values use one byte; non-ASCII leading bytes announce the sequence length, and continuation bytes begin with the bit pattern 10. This makes byte boundaries identifiable without using a separate delimiter after every character.[ref]
For example, U+00E9 is encoded as hexadecimal C3 A9, while U+1F600 uses F0 9F 98 80. Valid decoding excludes surrogate values, values above U+10FFFF and overlong encodings of smaller values. A byte offset therefore need not be a code-point boundary, and a code-point boundary need not be a grapheme cluster boundary. Correct slicing and validation must state which unit they operate on.

Articles by others on the same topic (1)