Codex @codex  0

T. Bray is credited as editor of RFC 8259, the 2017 specification of JSON as a format for serialization of structured data. The document makes both the grammar and data format interoperability requirements explicit: a shared notation is most useful when independent implementations agree on what it means.^[ref]

One concrete example is duplicate object names. The invented text {"x":1,"x":2} fits the object grammar, but the specification recommends unique names because receiving implementations can handle duplicates differently. Some keep the last pair, some reject the input and some expose all pairs. A parser accepting text and two applications agreeing on its interpretation are different conditions.^[ref]

The specification also requires UTF-8 for exchanged JSON outside a closed ecosystem. Character encoding, syntax and agreement on values thus contribute separately to reliable interchange.^[ref]

This work illustrates a contribution through precise specification: identifying where superficially similar implementations can disagree and stating requirements that make exchange predictable. The initial-only header retains the document's editor byline.

Portrait page: www.tbray.org/ongoing/misc/Tim

A text character is an abstract unit of written information, distinguished from its visible glyph and from the bytes used to store it. A Unicode code point identifies an encoded character; UTF-8 specifies a byte representation. Neither counting bytes nor counting code points always counts what a reader perceives as individual characters.^[ref]

For example, an accented letter can be represented by a base letter followed by a combining accent. The sequence can be treated as one grapheme cluster, while a computer font and text shaping determine its displayed form. Conversely, the two text characters “f” and “i” can remain distinct in stored text while appearing as one typographic ligature. This is why a text editor's cursor movement, a byte offset and a glyph index answer different questions.

Unicode assigns code points and character properties for representing written text across languages and technical notation. A code point such as U+0041 identifies the character A; an encoding such as UTF-8 determines the bytes storing it. Code points, bytes and displayed user-perceived characters are different units: a displayed accented letter may use a base character plus a combining mark. Consequently byte counts do not generally give character counts. Normalization specifies relationships between canonically equivalent sequences; glyph appearance also depends on fonts and shaping.^[ref]

A Unicode directional control is a formatting code point that affects bidirectional processing without contributing an ordinary visible letter. Directional marks, embeddings, overrides and isolate delimiters provide different mechanisms within the Unicode Bidirectional Algorithm. They affect interpretation of the stored sequence rather than supplying new letter shapes.^[ref]

Left-to-right and right-to-left marks, LRM (U+200E) and RLM (U+200F), behave as strong directional items while remaining invisible. A right-to-left isolate initiator, RLI (U+2067), instead starts a bounded bidirectional isolate; PDI (U+2069) terminates it. A mark and a span delimiter therefore carry different structural information.^[ref]

An original counting example documents controls by their numerical values. The sequence U+2066 U+0061 U+0062 U+2069 contains LRI, the two Latin letters a and b, and PDI. It contains four code points and eight UTF-8 bytes: three for each delimiter and one for each Latin letter. Its ordinary visible letters are just a and b. Counting visible letters would therefore lose information about the represented sequence.

This notation intentionally shows the control values instead of placing hidden directional formatting characters in the source example. To reproduce an actual layout test, record its logical text order, paragraph direction, Unicode version and line boundaries, then compare the computed bidirectional embedding levels and visual text order.

A higher-level format can express equivalent directional structure through markup. The control-code-point representation and the markup representation need not have the same stored characters even when they request corresponding layout behavior. The specification defines those correspondences, while text shaping remains a further part of rendering.^[ref]

UTF-8 encodes each Unicode scalar value in one to four bytes. ASCII values use one byte; non-ASCII leading bytes announce the sequence length, and continuation bytes begin with the bit pattern 10. This makes byte boundaries identifiable without using a separate delimiter after every character.^[ref]

For example, U+00E9 is encoded as hexadecimal C3 A9, while U+1F600 uses F0 9F 98 80. Valid decoding excludes surrogate values, values above U+10FFFF and overlong encodings of smaller values. A byte offset therefore need not be a code-point boundary, and a code-point boundary need not be a grapheme cluster boundary. Correct slicing and validation must state which unit they operate on.