encoding - introspection + text and character codecs

Enable with use encoding;. Three groups of functions:

1. Introspection - rune count vs byte count, and an ASCII test.
2. Binary-to-text (toText / fromText) - hex, base64, and quoted-printable, bytes to a printable string and back.
3. Character codecs (encode / decode) - a Jennifer string to and from a single-byte legacy encoding (ISO-8859-, Windows-, EBCDIC).

The cross-kind UTF-8 codec ships with convert (convert.bytesFromString / convert.stringFromBytes); this library is where the rest of the codec proliferation lives.

use io;
use convert;
use encoding;

def src as bytes init convert.bytesFromString("café", "utf-8");
io.printf("%s\n", encoding.toText($src, "hex"));            # 636166c3a9
io.printf("%t\n", encoding.isAscii($src));                 # false
io.printf("%x\n", encoding.encode("café", "iso-8859-1"));  # 63 61 66 e9

Introspection

Binary-to-text: toText / fromText

One verb pair, the encoding named by a string. Reversible representation - these grow or reshape the bytes, they don’t reduce information (that’s compress).

Formats

Quoted-printable keeps printable ASCII literal, turns =, control, and 8-bit bytes into =XX, and soft-wraps lines to 76 columns with a trailing =. Decode reverses it, tolerant of both CRLF and LF soft breaks, and round-trips bytes.

Format names are exact (case-sensitive, no - / _ normalisation) - they’re the library’s own fixed set, not external standards with variant spellings. "base64" works; "BASE64" errors with the supported set listed.

Why a format string instead of encoding.hex() / encoding.base64()? Jennifer’s letters-only identifier rule rejects digits in method names, so encoding.base64 won’t parse; and the codec-table shape already matches the rest of this library plus convert, hash, and crc.

Character codecs: encode / decode

Convert a Jennifer string to and from a named single-byte encoding.

Codec set

Every codec below is single-byte: one byte maps to one rune. Bytes with no assignment in a codec (some Windows code pages have a few) are a positioned error on decode; runes with no byte are a positioned error on encode.

ASCII and EBCDIC

ISO/IEC 8859 (single-byte Latin and script families)

(There is no iso-8859-12; the draft was abandoned.)

Windows code pages

Codec names are exact

Codec names are exact-match - the one canonical spelling only, no case-folding, separator-stripping, or IANA aliases. "iso-8859-1" works; "latin-1", "ISO-8859-1", "iso88591", and "cp1252" all error with the known set listed. This is deliberate strictness (stance #2, explicit over implicit): a codec is named one way, and no lenient spelling hides a typo. Map an external name (an HTTP charset=ISO-8859-1, say) to the canonical form yourself before calling.

How the tables are built

Only ascii (7-bit, with its own out-of-range errors) and ebcdic (IBM-1047, not in the standard Unicode mapping path) are hand-written. Every ISO-8859 and Windows single-byte codec - including iso-8859-1 and windows-1252 - is generated from the Unicode Consortium mapping files, so every table comes from one authoritative source rather than being hand-transcribed:

go generate ./internal/lib/encoding/   # writes codecs_gen.go

Converting a text file

Jennifer strings are UTF-8 internally, so converting a legacy-encoded file to UTF-8 is just decode-then-write - there’s no separate “encode to UTF-8” step:

use fs;
use encoding;

def raw as bytes init fs.readBytes("legacy.txt");               # Windows-1252 bytes
def text as string init encoding.decode($raw, "windows-1252"); # -> a UTF-8 string
fs.writeString("utf8.txt", $text);                             # written as UTF-8

The reverse (UTF-8 to a single-byte encoding) is encoding.encode(text, codec), which errors if a character has no byte in the target codec.

Length and memory

encode / decode / toText / fromText are one-shot: each takes the whole input and builds the whole output in memory, so both are held at once. There is no fixed cap - a Jennifer string / bytes is a Go string / []byte, bounded only by available memory. For a file too large to hold twice over, note that the single-byte codecs decode each byte independently, so decoding can be split at any byte boundary: read the file in fixed-size byte chunks, decode each, and append to the output. (Encoding the other way must instead split on rune boundaries, since a UTF-8 character can span several bytes.)

Errors

• Wrong arity: encoding.encode expects 2 arguments (string, codec), got 1.
• Wrong scalar type: encoding.encode: first argument must be string, got bytes.
• Unknown codec (the message lists every registered name): encoding.encode: unknown codec "klingon"; known: "ascii", "iso-8859-1", ....
• Unrepresentable rune on encode: encoding.encode (ascii): rune U+00E9 at byte position 3 is outside ASCII (0x00..0x7F).
• Undefined byte on decode: encoding.decode (windows-1252): byte 0x81 at position 0 has no mapping in windows-1252.
• Invalid UTF-8 to lenRunes: encoding.lenRunes: input is not valid UTF-8.

All errors are positioned at the call site.

See also

• convert.md - the UTF-8 pair (bytesFromString / stringFromBytes).
• compress.md - size reduction (distinct from representation).
• hash.md, crc.md - digests whose output you’ll often encoding.toText($digest, "hex").