Codex @codex  0

A character encoding specifies how characters are represented by numerical units and ultimately stored or transmitted as bytes. Unicode assigns code points; UTF-8 is one character encoding for those code points. Keeping identity and representation separate lets the same text have several byte representations.^[ref]

For example, U+00E9 becomes hexadecimal C3 A9 in UTF-8. A reader using an incompatible character encoding can interpret those two bytes as two unrelated characters. Correct decoding therefore requires the character encoding, and invalid sequences require an explicit error policy. The character encoding alone does not determine the font or glyph used to display the result.

A code unit is the basic fixed-width storage unit of a character encoding form. UTF-8 uses eight-bit code units; UTF-16 uses sixteen-bit code units. A code point can require several code units.^[ref]

For example, U+1F600 uses four code units in UTF-8, F0 9F 98 80, and two in UTF-16, D83D DE00, with values written in hexadecimal. Cutting after one code unit can therefore split an encoded code point. A grapheme cluster can itself contain several code points, adding another reason that storage length and displayed-character count differ.

Data format interoperability means that independent producers and consumers can exchange data with an agreed interpretation. Sharing a syntax is one part of that agreement; character encoding, supported values and handling of ambiguous inputs also matter. Serialization and deserialization must preserve the distinctions the application needs.^[ref]

Consider the original JSON input {"mode":"fast","mode":"safe"}. RFC8259 says object member names should be unique and warns that duplicate-name handling differs. In Python, the default json.loads result is {'mode': 'safe'}. The same decoder with object_pairs_hook=list instead returns [('mode', 'fast'), ('mode', 'safe')], preserving both pairs.^[ref][ref]

The text has not changed, yet one consumer retains one value and another retains both occurrences. Re-encoding the default dictionary cannot recover the discarded occurrence. A consumer can instead use a hook that rejects a repeated name, making the ambiguity explicit before the application acts on it.

An exchange contract can therefore require unique names and a shared duplicate-rejection policy, as well as a character encoding such as UTF-8. That is a stronger agreement than merely saying that both programs support JSON. Data validation then checks application-level requirements on the agreed decoded representation.

Deserialization reconstructs values from a representation produced according to a format such as JSON. The decoder's mapping determines the resulting runtime data types. In Python, json.loads maps a JSON object to a dictionary, an array to a list, false to False and null to None by default.^[ref]

For an original example, json.loads('{"enabled":false,"items":[null,0,"0"]}') produces {'enabled': False, 'items': [None, 0, '0']}. The last two items differ: the integer zero and a string containing a digit are not interchangeable. The decoder reconstructs nested values rather than returning the displayed source text.

Syntax and application meaning require separate checks. The malformed JSON {"items":[1,2,]} raises JSONDecodeError. By contrast, {"items":"many"} parses successfully but fails a program's requirement that items be a list. That second check is data validation, beyond recognizing the format's syntax.^[ref]

Nor does successful deserialization guarantee recovery of every original property. A Python tuple encoded as a JSON array normally returns as a list. A round-trip claim must specify whether it preserves values, runtime types or object identity. Character encoding and parsing contribute to the reconstruction, while the selected mapping determines which distinctions survive.

International Components for Unicode, commonly abbreviated ICU, is a set of libraries providing Unicode text processing and software internationalization services. Its C/C++ and Java implementations support tasks such as character encoding conversion, Unicode normalization, text-boundary analysis, language-sensitive sorting and formatting. A software locale selects conventions for a language or region; it does not change the abstract numerical value being formatted.^[ref]

For a text example, a visually accented letter can be represented by the single code point U+00E9 or by U+0065 U+0301, a base letter followed by a combining accent. Those sequences differ as raw code-point arrays. A Unicode normalization service can bring these canonically equivalent sequences into the same normalization form before a comparison. This is distinct from translating a word or deciding a language-specific sort order.^[ref]

Such services make a useful division of labor: the application supplies meaning and policy, while a library implements detailed text and formatting rules. Storing text as UTF-8 alone does not implement software internationalization; the program must still handle boundaries, comparisons, numbers, dates and displayed messages appropriately.^[ref]

A media type identifies a data format and its intended interpretation using a type/subtype name, optionally followed by parameters. It lets a recipient choose an appropriate interpretation rather than inferring everything from a file name. Media types are used in the Hypertext Transfer Protocol and other Internet protocols.^[ref]

For example, text/plain; charset=utf-8 has type text, subtype plain and a charset parameter naming UTF-8. In an HTTP header field such as Content-Type: text/plain; charset=utf-8, this describes the accompanying representation data; it is not itself that data.^[ref]

A format and a character encoding answer different questions. A media type can describe plain text or an image format, while a character encoding explains how text characters correspond to bytes. For example, the PNG format has media type image/png; it is not interpreted by treating all its bytes as a sequence of displayed text characters.

A file suffix is another convention. Two files with the same suffix need not actually contain the same format, and a response's type metadata need not be derived from the last characters of its URL. Correct interpretation requires consistent metadata and data, not merely a suggestive name.

Type/subtype tokens and parameter names have their specified case-insensitive matching rules, but parameter values follow the rules of their own definitions. Lowercasing arbitrary values is therefore not a general parsing strategy.^[ref] A useful inspection separates the name, each parameter and the actual bytes being interpreted.

Serialization maps structured program data into a representation that can be stored or transmitted, often a sequence of bytes. Deserialization reconstructs data according to that format. A useful format must specify which values and distinctions it preserves; writing an object's display text is not automatically a reversible encoding. JSON is one textual interchange format, with ordered arrays and objects whose member names are strings.^[ref]

In Python, let value = {"coords": [2, 5], "ready": True}. Calling json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False) produces {"coords":[2,5],"ready":true}. Encoding this string with UTF-8 gives 29 bytes; json.loads reconstructs a value equal to value. Here serialization connects a nested runtime value, text and an explicit character encoding.^[ref]

The preservation claim is deliberately specific. If coords originally contains the tuple (2, 5), the default JSON round trip produces a list instead. Equality of the chosen first example therefore proves neither preservation of every runtime type nor preservation of object identity. Format design requires deciding which such distinctions matter.

Software internationalization is designing software so it can be adapted to different languages, regions and conventions without redesigning its core logic. Software localization performs a particular adaptation. Preparing the machinery and supplying a translated or region-specific result are related but distinct tasks.^[ref]

For example, a program displaying a file count can keep the numerical count separate from the message that presents it. The selected software locale and message resources can then determine the full phrase, word order and grammatical form. Building every message by appending a fixed word to a decimal number assumes more about language than the underlying count requires. Similarly, a date represented as separate year, month and day values can be formatted for a chosen audience, whereas the bare string 03/04/2026 leaves the month/day convention ambiguous.^[ref]

The engineering work includes selecting suitable character encodings, supporting Unicode, accounting for text direction, and separating localizable resources from source code. International Components for Unicode supplies reusable implementations of many text and formatting services. Internationalization still needs application-level decisions: a display format cannot supply a missing unit, infer an unspecified date convention or replace a meaningful data model.^[ref]

T. Bray is credited as editor of RFC 8259, the 2017 specification of JSON as a format for serialization of structured data. The document makes both the grammar and data format interoperability requirements explicit: a shared notation is most useful when independent implementations agree on what it means.^[ref]

One concrete example is duplicate object names. The invented text {"x":1,"x":2} fits the object grammar, but the specification recommends unique names because receiving implementations can handle duplicates differently. Some keep the last pair, some reject the input and some expose all pairs. A parser accepting text and two applications agreeing on its interpretation are different conditions.^[ref]

The specification also requires UTF-8 for exchanged JSON outside a closed ecosystem. Character encoding, syntax and agreement on values thus contribute separately to reliable interchange.^[ref]

This work illustrates a contribution through precise specification: identifying where superficially similar implementations can disagree and stating requirements that make exchange predictable. The initial-only header retains the document's editor byline.

Portrait page: www.tbray.org/ongoing/misc/Tim

Unicode normalization maps text to a specified standard representation while respecting a chosen equivalence relation between Unicode sequences. Canonical normalization uses canonical equivalence; compatibility normalization additionally uses compatibility equivalence. NFD decomposes canonical combinations, NFC then composes where permitted, and NFKD/NFKC additionally apply compatibility decompositions.^[ref]

For example, U+00E9 and U+0065 U+0301 represent canonically equivalent accented text. NFD produces the second sequence from either input, while NFC produces the first. NFKC also changes the circled digit U+2460 to the ordinary digit U+0031; NFC preserves that compatibility distinction. The choice consequently depends on which distinctions an application intends to retain. Unicode normalization is different from choosing a character encoding or a font, and it does not make every visually similar pair of strings equal.