UTF-8000: Unlimited UTF-8 for Arbitrarily Large Codepoints — Key

UTF-8 has long been the dominant character encoding on the web and in many systems, lauded for its compatibility with ASCII, variable-length design, and robust error recovery features. However, standard UTF-8, as defined by RFC 3629, is limited to encoding Unicode codepoints up to U+10FFFF, using a maximum of four bytes per character. While this range covers all currently assigned Unicode characters and most anticipated additions, what if we need to go beyond these bounds? What if we require truly unlimited capacity for future-proofing or novel applications?

Enter UTF-8000, a fascinating proposal and standalone project that extends UTF-8 to support arbitrarily large codepoints without introducing any new special cases. It embraces and builds upon the clever architecture originally designed by Ken Thompson and Rob Pike, demonstrating that UTF-8's core principles were inherently more flexible than its current four-byte implementation suggests. UTF-8000 maintains full compatibility with existing UTF-8 and ASCII, meaning ASCII ⊆ UTF-8 ⊆ UTF-8000.

The Elegant Extension: How UTF-8000 Works

The fundamental innovation of UTF-8000 lies in allowing multiple start bytes within a single code unit, a concept that was always implicitly possible within UTF-8's design, even if not utilized beyond the first byte in standard UTF-8. Let's break down the mechanics:

Core Byte Structure

Every byte in a UTF-8000 stream provides immediate clues about its role, a property known as self-synchronization:

• ASCII Bytes: Begin with a 0 bit (e.g., 0xxxxxxx). These are single-byte code units for codepoints U+0000 to U+007F, identical to standard ASCII.
• Multi-byte First Bytes: Begin with 11 (e.g., 110xxxxx, 1110xxxx). This marks the beginning of a multi-byte sequence.
• Continuation Bytes: Begin with 10 (e.g., 10xxxxxx). These bytes follow a multi-byte first byte.

This prefix-free design allows a decoder to instantly determine if it's at the start of a new character or in the middle of one, crucial for random access and error recovery. For instance, if a byte stream is corrupted, a decoder can simply discard invalid bytes until it encounters a valid first byte (0xxxxxxx or 11xxxxxx) to resynchronize.

Encoding Code Unit Length with Start Bits

For multi-byte code units, the length in bytes is indicated by a sequence of start bits. These bits form a unary code: n-2 1s followed by a terminating 0 bit. So, for an n-byte code unit, the sequence is 11...10. For example:

• 2-byte code unit: 110xxxxx (1 1 followed by a 0) - 1 start bit (ignoring the self-synchronization prefix 11 for a moment).
• 3-byte code unit: 1110xxxx (2 1s followed by a 0)
• 4-byte code unit: 11110xxx (3 1s followed by a 0)

In standard UTF-8, these start bits always fit within the very first byte. UTF-8000 extends this by allowing the start bits to stripe across subsequent bytes. For code units of 8 bytes or more, continuation bytes (10xxxxxx) can also function as start bytes, carrying additional 1 bits of the unary length code. For example, an 8-byte code unit starts 11111111``100xxxxx. The first byte 11111111 is followed by a continuation byte 100xxxxx, which, notably, has a 0 in its third bit position (10 0 xxxxx), completing the unary sequence for an 8-byte code unit (seven 1s followed by a 0).

This ingenious mechanism, derived directly from UTF-8's original design flexibility, means that the length of a code unit is always known by reading these initial bytes, a property called self-punctuation. You don't need to read past the end of a character to know its length.

Storing the Codepoint: Content Bits

The actual binary representation of the codepoint is stored in the content bits. The number of content bits varies based on the code unit length:

• ASCII (1-byte): 7 content bits (0xxxxxxx).
• Multi-byte (n > 1): 5n+1 content bits. This formula emerges from 8n total bits minus 2n for self-synchronization prefixes (two bits per byte) and n-1 for start bits (one bit for each 1 in the unary sequence plus the terminating 0). This simplifies to 8n - 2n - (n-1) = 5n+1.

For instance, a 2-byte code unit has 5*2+1 = 11 content bits, while a 3-byte code unit has 5*3+1 = 16 content bits.

Preventing Overlong Encoding: Mandatory Content Bits

To ensure each codepoint has a unique, canonical encoding and to prevent security vulnerabilities (like an attacker encoding a null byte U+0000 as a multi-byte sequence to bypass string termination checks in C), UTF-8000, like UTF-8, forbids overlong encodings. This means a codepoint must always be encoded using the shortest possible code unit.

This is enforced by checking mandatory content bits:

• ASCII (1-byte): 0 mandatory content bits, as it's the shortest possible encoding.
• 2-byte code units: The first 4 content bits must not all be zero. This is a special case inherited from UTF-8, effectively disallowing byte sequences like 0xC0 (11000000) and 0xC1 (11000001) from ever appearing in valid UTF-8000.
• n > 2-byte code units: The first 5 content bits must not all be zero. This is a consistent rule for all longer code units.

This careful handling of overlong encodings is a testament to the security-first design philosophy inherited by UTF-8000.

No New Special Cases: A Confirmation of UTF-8's Design

One of the most remarkable aspects of UTF-8000 is its claim to extend UTF-8 without introducing any new special cases. The two 'exceptions' — ASCII having 7 content bits (instead of 5n+1) and 2-byte units having 4 mandatory content bits (instead of 5) — are directly inherited from the original UTF-8 specification. This strongly suggests that UTF-8's original design was inherently extensible, and UTF-8000 merely unlocks this latent capability.

Information Rate and Practicality

The information rate of UTF-8000 (the proportion of content bits to total bits) for multi-byte code units is (5n+1) / 8n. As n grows, this value approaches 5/8, or 62.5%. This means that even for extremely long code units, a significant portion of the bytes is dedicated to encoding the codepoint itself, rather than overhead. This asymptotic behavior ensures that the encoding doesn't become prohibitively inefficient for very large codepoints.

While UTF-8000 is a fun, standalone proposal not endorsed by the Unicode Consortium, it serves as an excellent case study in robust, extensible encoding design. It highlights the genius of UTF-8's original architects and offers a glimpse into how future character encodings could evolve if the need for arbitrarily large codepoints ever arose.

Developers interested in experimenting can try out the reference implementation: $ pipx install UTF-8000.

FAQ

Q: Is UTF-8000 an official Unicode standard or endorsed by the Unicode Consortium?

A: No, UTF-8000 is explicitly a fun, standalone project and proposal. It is not endorsed by or representative of the Unicode Consortium.

Q: What is the main advantage of UTF-8000 over the current UTF-8 standard (RFC 3629)?

A: The primary advantage is its ability to encode arbitrarily large codepoints using an unlimited number of bytes per code unit, while preserving all the key properties of UTF-8, such as self-synchronization and self-punctuation, and introducing no new special cases.

Q: How does UTF-8000 handle byte-level synchronization and error recovery?

A: UTF-8000 inherits UTF-8's self-synchronization property through its byte prefixes. ASCII bytes start with 0, multi-byte first bytes with 11, and continuation bytes with 10. This allows a decoder to unambiguously identify the start of a code unit, enabling robust random access and error recovery in corrupted streams.