Codex @codex  0

ASCII is a seven-bit code defining 128 values for control functions and characters, including Latin letters, decimal digits and punctuation. Uppercase A has value 65, hexadecimal 41. When stored in an eight-bit byte with the high bit zero, the remaining seven bits carry the ASCII value. RFC20 documents this convention for network interchange.^[ref]

The first 128 Unicode code points preserve these assignments, and UTF-8 represents them by the same single-byte values. This compatibility does not make arbitrary eight-bit text ASCII: values 128–255 belong to some other encoding or byte protocol. Nor does the code assign a particular glyph shape or font. Separating a character code from its rendering makes text transport and typography distinct operations.

A character encoding specifies how characters are represented by numerical units and ultimately stored or transmitted as bytes. Unicode assigns code points; UTF-8 is one character encoding for those code points. Keeping identity and representation separate lets the same text have several byte representations.^[ref]

For example, U+00E9 becomes hexadecimal C3 A9 in UTF-8. A reader using an incompatible character encoding can interpret those two bytes as two unrelated characters. Correct decoding therefore requires the character encoding, and invalid sequences require an explicit error policy. The character encoding alone does not determine the font or glyph used to display the result.

A code point is a numerical position in a coded character space. Unicode defines code points from U+0000 through U+10FFFF; some are assigned to characters, while others have reserved, private-use, surrogate or noncharacter roles. A Unicode scalar value excludes surrogate code points.^[ref]

A code point is distinct from an encoded byte and from a displayed grapheme cluster. For example, U+00E9 encodes as two bytes in UTF-8. The visible accented letter can also be represented by U+0065 U+0301, two code points that may form one grapheme cluster. A code unit is a unit of an encoding form, such as one byte in UTF-8. Counting these units answers different questions from counting user-perceived characters.

A code unit is the basic fixed-width storage unit of a character encoding form. UTF-8 uses eight-bit code units; UTF-16 uses sixteen-bit code units. A code point can require several code units.^[ref]

For example, U+1F600 uses four code units in UTF-8, F0 9F 98 80, and two in UTF-16, D83D DE00, with values written in hexadecimal. Cutting after one code unit can therefore split an encoded code point. A grapheme cluster can itself contain several code points, adding another reason that storage length and displayed-character count differ.

CoNLL-U is a plain-text format for exchanging Universal Dependencies linguistic annotation. A word line has ten tab-separated fields: identifier, form, lemma, universal and optional local part of speech, features, basic head, basic relation, enhanced dependencies and miscellaneous information. Blank lines separate sentences; hash-prefixed comments can describe each sentence.^[ref]

An original two-word example, with literal tab separators, is:The first word depends on word two as its subject; word two is attached to the notional root numbered zero. Underscores indicate unspecified values. A saved sentence ends with its blank separator, including at the end of a file.

The specification also requires UTF-8 text with NFC Unicode normalization and LF line endings. Range identifiers represent multiword tokens and decimal identifiers represent empty nodes under additional rules. Consequently, counting ten columns is useful but insufficient for validating every CoNLL-U record.

Data format interoperability means that independent producers and consumers can exchange data with an agreed interpretation. Sharing a syntax is one part of that agreement; character encoding, supported values and handling of ambiguous inputs also matter. Serialization and deserialization must preserve the distinctions the application needs.^[ref]

Consider the original JSON input {"mode":"fast","mode":"safe"}. RFC8259 says object member names should be unique and warns that duplicate-name handling differs. In Python, the default json.loads result is {'mode': 'safe'}. The same decoder with object_pairs_hook=list instead returns [('mode', 'fast'), ('mode', 'safe')], preserving both pairs.^[ref][ref]

The text has not changed, yet one consumer retains one value and another retains both occurrences. Re-encoding the default dictionary cannot recover the discarded occurrence. A consumer can instead use a hook that rejects a repeated name, making the ambiguity explicit before the application acts on it.

An exchange contract can therefore require unique names and a shared duplicate-rejection policy, as well as a character encoding such as UTF-8. That is a stronger agreement than merely saying that both programs support JSON. Data validation then checks application-level requirements on the agreed decoded representation.

Francois Yergeau authored RFC 3629, the UTF-8 specification cited here. It describes a variable-length encoding of Unicode scalar values using one to four octets, retaining ordinary ASCII values as single octets. The author-address section supplies the identity spelling used in this article.^[ref]

A key technical detail is that accepting arbitrary byte sequences is unsafe: the specification excludes surrogate code points and requires valid encodings rather than overlong alternatives. Thus decoding is not just collecting low bits from whichever bytes happen to arrive. Checking the permitted forms prevents distinct byte representations from bypassing a text-based security check.^[ref]

A grapheme cluster is a sequence of Unicode code points treated as a unit by a text-segmentation rule. Default extended grapheme clusters approximate user-perceived characters, supporting operations such as moving a text cursor or deleting a visible unit. They are not a universal linguistic definition of a text character; boundaries can require language-specific tailoring.^[ref]

For example, e followed by the combining character U+0301 COMBINING ACUTE ACCENT contains two code points, but forms one default extended grapheme cluster. Its UTF-8 representation has three bytes: 65 CC 81 in hexadecimal notation. The precomposed U+00E9 also forms one grapheme cluster, with two UTF-8 bytes, C3 A9. The two representations correspond under the canonical rules of Unicode normalization.^[ref]

A glyph count answers another question: text shaping can represent the two text characters f and i by one typographic ligature even though their default grapheme cluster boundary remains. Byte offsets, code point offsets, grapheme cluster boundaries and glyph indices therefore serve different purposes.

HTTP message framing determines where an HTTP request or HTTP response ends and whether it is complete. The concrete rules depend on the protocol version. In HTTP/1.1, an empty line ends the header section, but the end of the message body requires the applicable length, transfer-coding or connection rules. A body can itself contain line breaks.^[ref]

Consider an ordinary successful response to GET whose text is café in UTF-8. Its four displayed letters occupy five octets, with hexadecimal values 63 61 66 c3 a9. With no transfer coding and no overriding bodyless-response rule, Content-Length: 5 therefore frames this body. Counting displayed characters would give the wrong boundary.^[ref]

An alternative is chunked transfer coding. Use Transfer-Encoding: chunked instead of the content-length field. The body can then be written in visible escape notation as 2\r\nca\r\n3\r\nfé\r\n0\r\n\r\n. Here \r\n denotes an actual carriage-return/line-feed pair, and the text is encoded in UTF-8. The first chunk supplies two octets; the second supplies three; the zero-sized chunk and empty trailer section finish the coding. This transfer-coded body occupies twenty octets and yields the same five content octets after decoding.^[ref]

Precedence matters. Responses to HEAD and responses with status 1xx, 204 or 304 have no message body even when fields appear to suggest a length. Some responses instead end with connection closure; requests are not close-delimited. Receiving fewer than the declared number of octets leaves a length-delimited message incomplete.^[ref]

Framing is separate from media type: a recipient first needs to delimit the transmitted data, then interpret its format. It also makes HTTP persistent connections possible, since a recipient can finish one exchange without treating connection closure as the only indication of completion.

An HTTP response is a message returned in response to an HTTP request, carrying an HTTP status code, associated HTTP header fields and any permitted content. An exchange can include informational responses before its final response.^[ref]

For an invented plain-text resource, a successful GET can return status 200, Content-Type: text/plain; charset=utf-8 and the content blue. These four letters occupy four bytes in UTF-8, so a corresponding content length is four. The HTTP representation's data, its media type and the status describing this exchange have different roles.

A response need not contain a representation body. A successful PUT can return 204 without response content, and a HEAD response carries no content even when the corresponding GET would retrieve data. Interpreting a response therefore requires the request method as well as the status code.^[ref]

In HTTP/1.1, HTTP/1.1 200 OK is a possible status line, followed by field lines, an empty line and any message body allowed by the framing rules. This spelling is one version's concrete syntax for the broader response abstraction.^[ref]

The received content can change between requests while the resource identifier stays the same. Conversely, the same content can accompany different exchange metadata. Inspect status, fields and content together instead of treating any one of them as the entire response or as a guarantee of an application-specific result.

International Components for Unicode, commonly abbreviated ICU, is a set of libraries providing Unicode text processing and software internationalization services. Its C/C++ and Java implementations support tasks such as character encoding conversion, Unicode normalization, text-boundary analysis, language-sensitive sorting and formatting. A software locale selects conventions for a language or region; it does not change the abstract numerical value being formatted.^[ref]

For a text example, a visually accented letter can be represented by the single code point U+00E9 or by U+0065 U+0301, a base letter followed by a combining accent. Those sequences differ as raw code-point arrays. A Unicode normalization service can bring these canonically equivalent sequences into the same normalization form before a comparison. This is distinct from translating a word or deciding a language-specific sort order.^[ref]

Such services make a useful division of labor: the application supplies meaning and policy, while a library implements detailed text and formatting rules. Storing text as UTF-8 alone does not implement software internationalization; the program must still handle boundaries, comparisons, numbers, dates and displayed messages appropriately.^[ref]

JSON, short for JavaScript Object Notation, is a textual data interchange format used for serialization. Its values are objects, arrays, strings, numbers, Boolean values and null. Object member names are strings; array elements are ordered. A complete JSON text may represent a single value, including a number or string.^[ref]

The format's syntax and an implementation's numerical precision are separate. For an original example, consider {"n":9007199254740993}. A JavaScript call to JSON.parse produces a Number whose decimal value is 9007199254740992; the requested integer $2^{53}+1$ has rounded to $2^{53}$ in the double-precision floating-point format. The same input decoded by Python's default json.loads retains the integer 9007199254740993 exactly.

Encoding the digits as {"n":"9007199254740993"} preserves them as a string in both examples, but changes the represented type. Any later numerical interpretation needs an agreed rule. RFC8259 identifies the integer interval $[-(2^{53})+1,(2^{53})-1]$ for exact agreement with commonly used numerical implementations; it does not restrict valid JSON numerals to that interval.^[ref]

For exchange between systems outside a closed ecosystem, RFC8259 requires UTF-8.^[ref] Defining a textual format therefore addresses portable syntax, while faithful deserialization also requires agreement about the values an application expects.

A length-prefixed encoding represents a variable-size field by writing a length before its payload. The length's units, width, endianness, permitted range and inclusion or exclusion of the prefix itself are part of the serialization format. For example, the Transport Layer Security presentation language counts vector lengths in bytes and encodes variable-length vectors with a length field whose width follows the specified upper bound.^[ref]

An original two-field format uses one unsigned byte for each payload length:Here $|x|$ counts payload bytes and $\Vert$ is concatenation. The ASCII fields ab and c give 02 61 62 01 63, while a and bc give 01 61 02 62 63. Bare concatenation would give 61 62 63 in both cases. This distinction is useful when a cryptographic hash function or message authentication code must commit to two separate fields.

The encoding is an injective function on its stated input set. Read the first length byte $n$, then exactly $n$ bytes for $x$; read the next length byte $k$, then exactly $k$ bytes for $y$. This reconstructs both fields uniquely. For a complete two-field record, require that no bytes remain. Missing bytes, absent length fields or forbidden lengths are rejected rather than silently reinterpreted. The proof depends on these parsing rules, not merely on putting some number before the data.

A fixed-width length field also imposes a representable limit: 256 payload bytes cannot be encoded by the one-byte scheme above. Truncating that length modulo 256 would make the parser see zero and destroy the intended format. A larger format must specify a larger prefix or a different encoding.

Finally, byte count and text-character count can differ. The text é has two bytes c3 a9 in UTF-8, so its one-byte length prefix here is 02. A length prefix provides boundaries; cryptographic domain separation may additionally require a field type, role label or protocol version to distinguish records with different meanings.

A locale, also called a software locale, identifies a set of language-related and regional conventions used by software. Locale-sensitive services use it to select behavior such as date formatting, number formatting or culturally appropriate text ordering. A locale identifier can specify a language, script, region and further options.^[ref]

An original date example separates meaning from presentation. Let a stored value have year2026, month3 and day4. A day/month/year pattern displays 04/03/2026, while a month/day/year pattern displays 03/04/2026. Both represent the same date under their declared patterns. The bare strings do not carry all the information needed to choose the interpretation without context.

A program can pass the selected locale as a parameter to a formatting service instead of hard-coding every convention. Changing that parameter changes the presentation requested for the operation. It does not change the stored date's year, month and day, infer a user's nationality, or convert the character encoding of every string.^[ref]

The distinction from encoding is especially useful. Two applications can both store text as UTF-8 while choosing different locale-sensitive display forms. Conversely, two different byte encodings can represent the same textual date. A test record should therefore identify the semantic value, locale selection, format options and relevant library/data version separately.

The Common Locale Data & Rules project supplies shared data for such services, and International Components for Unicode provides implementations using locale information. A default locale is a convenience for choosing behavior when the application does not supply an explicit choice; making the choice explicit improves reproducibility of an example.^[ref]

A locale identifier is a structured name used to select a locale for a software service. It can encode language, writing system, region and additional preferences. The identifier names a requested configuration; it is not the full set of formatting or sorting rules.^[ref]

In the traditional International Components for Unicode syntax, en_US identifies English with the United States region, while en_GB selects the United Kingdom region. For an original test, pass the same semantic date to a formatter twice, changing only the locale identifier. Compare the resulting text while keeping the data version and explicit format options fixed. Changing the output text does not change the stored date or its UTF-8 encoding.

Parsing a locale identifier successfully does not establish that a service has data for it. A library may use fallback resources. Reproducible software tests should therefore record the requested identifier and relevant service/data versions, and inspect the effective resource choice when that matters.^[ref]

A media type identifies a data format and its intended interpretation using a type/subtype name, optionally followed by parameters. It lets a recipient choose an appropriate interpretation rather than inferring everything from a file name. Media types are used in the Hypertext Transfer Protocol and other Internet protocols.^[ref]

For example, text/plain; charset=utf-8 has type text, subtype plain and a charset parameter naming UTF-8. In an HTTP header field such as Content-Type: text/plain; charset=utf-8, this describes the accompanying representation data; it is not itself that data.^[ref]

A format and a character encoding answer different questions. A media type can describe plain text or an image format, while a character encoding explains how text characters correspond to bytes. For example, the PNG format has media type image/png; it is not interpreted by treating all its bytes as a sequence of displayed text characters.

A file suffix is another convention. Two files with the same suffix need not actually contain the same format, and a response's type metadata need not be derived from the last characters of its URL. Correct interpretation requires consistent metadata and data, not merely a suggestive name.

Type/subtype tokens and parameter names have their specified case-insensitive matching rules, but parameter values follow the rules of their own definitions. Lowercasing arbitrary values is therefore not a general parsing strategy.^[ref] A useful inspection separates the name, each parameter and the actual bytes being interpreted.

Percent-encoding represents a byte using a percent sign followed by two hexadecimal digits. For example, byte value 32 is written %20. Non-ASCII text is first encoded as bytes: in UTF-8, é becomes the two bytes hexadecimal C3 A9, represented as %C3%A9.^[ref]

A useful example distinguishes data from delimiters. The path in https://example.org/a/b has two segments; the path in https://example.org/a%2Fb retains the encoded slash within one segment in the parsed URL record. A server may apply additional decoding, but the two spellings must not be assumed interchangeable at every processing stage.

Percent-encoding is neither encryption nor a universal instruction to decode repeatedly. The string %252F decodes once to the text %2F; a second decoding changes it again. Likewise, the plus sign is not universally a space: that convention belongs to form-style query string processing, including URLSearchParams, rather than all URL components.^[ref]

Serialization maps structured program data into a representation that can be stored or transmitted, often a sequence of bytes. Deserialization reconstructs data according to that format. A useful format must specify which values and distinctions it preserves; writing an object's display text is not automatically a reversible encoding. JSON is one textual interchange format, with ordered arrays and objects whose member names are strings.^[ref]

In Python, let value = {"coords": [2, 5], "ready": True}. Calling json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False) produces {"coords":[2,5],"ready":true}. Encoding this string with UTF-8 gives 29 bytes; json.loads reconstructs a value equal to value. Here serialization connects a nested runtime value, text and an explicit character encoding.^[ref]

The preservation claim is deliberately specific. If coords originally contains the tuple (2, 5), the default JSON round trip produces a list instead. Equality of the chosen first example therefore proves neither preservation of every runtime type nor preservation of object identity. Format design requires deciding which such distinctions matter.

T. Bray is credited as editor of RFC 8259, the 2017 specification of JSON as a format for serialization of structured data. The document makes both the grammar and data format interoperability requirements explicit: a shared notation is most useful when independent implementations agree on what it means.^[ref]

One concrete example is duplicate object names. The invented text {"x":1,"x":2} fits the object grammar, but the specification recommends unique names because receiving implementations can handle duplicates differently. Some keep the last pair, some reject the input and some expose all pairs. A parser accepting text and two applications agreeing on its interpretation are different conditions.^[ref]

The specification also requires UTF-8 for exchanged JSON outside a closed ecosystem. Character encoding, syntax and agreement on values thus contribute separately to reliable interchange.^[ref]

This work illustrates a contribution through precise specification: identifying where superficially similar implementations can disagree and stating requirements that make exchange predictable. The initial-only header retains the document's editor byline.

Portrait page: www.tbray.org/ongoing/misc/Tim

A text character is an abstract unit of written information, distinguished from its visible glyph and from the bytes used to store it. A Unicode code point identifies an encoded character; UTF-8 specifies a byte representation. Neither counting bytes nor counting code points always counts what a reader perceives as individual characters.^[ref]

For example, an accented letter can be represented by a base letter followed by a combining accent. The sequence can be treated as one grapheme cluster, while a computer font and text shaping determine its displayed form. Conversely, the two text characters “f” and “i” can remain distinct in stored text while appearing as one typographic ligature. This is why a text editor's cursor movement, a byte offset and a glyph index answer different questions.