idna: internationalized domain names for C++

This program converts two names to their ASCII form, and an A-label back to its Unicode form, with the default options:

It prints:

Bücher.Example is mapped first, its capitals lowered, so it is valid, and its first label, which is not ASCII, becomes an A-label. a·b.example is refused: U+00B7 MIDDLE DOT is valid in IDNA2008 only between two l, a CONTEXTO rule that UTS 46 alone does not have. to_unicode decodes the A-label back, and reports no error.

<webcpp/idna.hpp> includes every public header; the names are in the namespace webcpp::idna, and Punycode’s in webcpp::idna::punycode:

Each code point is looked up in UTS 46’s mapping table: a code point that is valid stays, one that is mapped is replaced by its mapping (A by a, fi by fi, U+3002 IDEOGRAPHIC FULL STOP by .), one that is ignored is removed (U+00AD SOFT HYPHEN), a deviation stays unless the processing is transitional (ß, ς, and the two joiners), and one that is disallowed stays, to be refused when its label is checked. The result is normalized to NFC and split at U+002E. A label that starts with xn-- is decoded from Punycode and checked as the U-label it holds; any other is checked as it is. The checks, in their order, are those of The rules.

idna does this one label at a time, in a working storage of one label’s code points, so a name of any length needs the same small storage (Limits).

to_ascii(output, domain, opts) writes the ASCII form of domain into the caller’s span and returns the number of chars written: each label that is not ASCII as xn-- and its Punycode, the others as they are, joined by .. A name that fails any check has no ASCII form: the result is the first error found, in the order of the labels, then bidi for the Bidi rule across the labels, then domain_too_long, and output_too_small last, so a span too small for a name that is too long anyway reports the name. With the DNS lengths verified, the default, the ASCII form is at most idna::max_domain_length, 253 chars, so a std::array<char, idna::max_domain_length> always holds it.

to_ascii(domain, opts, resource) returns the same ASCII form as a std::pmr::string. With the DNS lengths verified, it reports the same first error as a span form large enough to hold the result, for every name, and it is never output_too_small: a label that outgrows the span form’s working storage is too long for the DNS anyway, and both forms refuse it there (The working storage).

to_unicode(output, domain, opts) writes the Unicode form, in UTF-8, and returns an idna::unicode_result: how many chars it wrote, and the first error. As UTS 46 asks, the conversion goes on after an error, so that a name can be shown even when it is not valid: each label is written as the processing leaves it, mapped and normalized, an A-label decoded when its Punycode decodes and kept as it is otherwise. A label that fails its checks is written so too, as the toUnicode column of IdnaTestV2 records it, and no U+FFFD is put in place of what failed: A_B.Example is written a_b.example, with not_ldh. With the DNS lengths verified, a final empty label, the DNS root of example., is no error in the Unicode form, as UTS 46’s X4_2 allows; any other empty label is empty_label. When the span is full, the writing stops at a code point boundary and the error is output_too_small, which takes the place of any other: the caller needs a larger span before anything else. Input that is not well-formed UTF-8 writes nothing, and is invalid_utf8.

to_unicode(domain, opts, resource) returns an idna::unicode_string, the Unicode form as a std::pmr::string and the first error. It is never output_too_small, and the two forms differ only past the working storage, which the Unicode form, checking no label’s maximum length, meets with the default options: the span form stops at a label beyond it, and the pmr form converts the label (The working storage).

Every conversion and every check takes an idna::options, the flags of UTS 46 section 4 and one of idna’s own, idna2008. The defaults are the strict profile: every check on, nontransitional processing, the DNS lengths verified, and IDNA2008’s validity on top. idna::options is an aggregate, so a program names only what changes: idna::options{.idna2008 = false} is UTS 46 alone, and idna::options{.verify_dns_length = false} lets labels of any length through.

A label is checked in this order, and the first rule it breaks is its error; to_ascii and to_unicode apply the same rules to each label after the mapping, with the NFC check only for a decoded A-label:

Across the labels of a name, under check_bidi: when one label holds an R, AL or AN code point, the name is a Bidi domain name, and then every label must satisfy RFC 5893 section 2, bidi. 0a.א fails: its first label starts with a digit, which rule 1 refuses in a Bidi domain name. Under verify_dns_length, the lengths of the ASCII form: each label’s with the label, an empty one as empty_label and, after its rules, one of more than 63 octets as label_too_long; the whole name’s last, after the Bidi rule, domain_too_long. A label that outgrows the working storage of 256 code points, which no label within the DNS lengths does, is refused as soon as it does, before its rules: label_too_long under verify_dns_length, storage_exhausted without it (Errors).

idna derives IDNA2008’s property of a code point from UTS 46’s table: PVALID, CONTEXTJ or CONTEXTO exactly when its status is valid or deviation and it bears no NV8 or XV8 mark. The generator checks this against IANA’s IDNA2008 tables each time it runs (Unicode and the tables).

Under verify_dns_length, the default, a label is 1 to idna::max_label_length, 63, octets in its ASCII form, and a name 1 to idna::max_domain_length, 253, without a final root label, which is refused: RFC 1034 section 3.1’s limits, as UTS 46 section 4.2 states them. A U-label is measured by its A-label: 55 a and a ü make an A-label of 63 octets, and 56 make one of 64.

The conversions and the checks hold one label at a time, in a working storage of 256 code points inside the call, about 1.3 KiB with the label’s ASCII form, and a second one while a label is checked for NFC the slow way. Every label the DNS lengths accept fits: an A-label decodes to at most 59 code points, and a code point decomposes to at most 4, U+1FAF among them, the longest canonical decomposition of Unicode 17.0.0, which the generator checks.

A label beyond 256 code points once decomposed is too long for the DNS, and what meets it depends on the length check:

The storage fills while a label is gathered, before any of the label’s checks, so a label beyond it reports that first: U+FFFF followed by 300 a is label_too_long, not disallowed. In the span form of to_unicode, the processing stops there, keeping an error recorded before it, and its output ends with the last label written before the long one: a name of the labels a_b, 300 a and c gives a_b, and not_ldh.

Punycode’s work grows with the square of a label’s length: RFC 3492’s encoder passes over the label once for each distinct code point, and its decoder inserts each code point among those before it. The conversions bound it at 256 code points, the working storage’s, and Punycode past the bound is punycode_overflow whatever ignore_invalid_punycode says, since a limit of the implementation is no invalid Punycode. to_ascii encodes, so a label of more than 256 code points that is not ASCII is punycode_overflow there. Both conversions decode, so an A-label whose Punycode decodes to more than 256 code points is punycode_overflow in either, and kept as it is in the Unicode form. to_unicode encodes nothing: a long label that is not an A-label is written as it is. Only a pmr form meets the bound, to_ascii without the DNS lengths, and to_unicode, with any options, for an A-label, since a span form refuses such a label as beyond its storage first. At the bound, the worst label costs about 4 times one as long that Punycode handles in linear time, measured at -O2.

punycode::encode and punycode::decode are the primitive, and have no such bound: their pmr forms take input of any length, in time that grows with its square, and their span forms stop when the caller’s span is full, in time that grows at most with the input’s length times the span’s size. A program bounds the length of input it does not trust before it hands it to a pmr form.

The deepest call of idna, a check of a label, or a conversion of an A-label, whose U-label holds more than 32 combining marks in a row, which the slow NFC check sorts by counting, ran in about 11 KiB of stack, and the conversions and checks of an ordinary name, such as xn--bcher-kva.example, in about 8 KiB, a program’s start and its printf included, against the 64 KiB a WebAssembly program gets by default. These are measured figures, with no margin promised: a program built with wasi-sdk 34 in a debug build (-O0 -g) for wasm32-wasip2 and wasm32-wasip3, run by wasmtime 47 with a stack made smaller, 256 bytes at a time, until it trapped. Another compiler or version may need more. idna does not recurse.

tools/generate_tables.py, in idna’s repository, writes the tables, and the vectors of the conformance tests, IdnaTestV2.txt and NormalizationTest.txt as C++ data under test/generated/ (Testing). Its inputs are pinned in tools/unicode.json, each by its URL under https://www.unicode.org/Public/17.0.0/ and its SHA-256, with IANA’s IDNA2008 tables, idna-tables-12.0.0, read only by its checks. A file its cache lacks is downloaded, every file is hashed, and a digest that differs stops it, naming the file and both digests, before it writes anything.

Before it writes, it checks what idna assumes of the data, and stops naming the assumption that fails: every status of the mapping table is one of the five it knows; IANA’s IDNA2008 property of every code point IANA assigns one agrees with the derivation from UTS 46’s table, U+002E FULL STOP aside, which IANA disallows and UTS 46 keeps as the separator no label holds; the CONTEXTJ and CONTEXTO code points are those of RFC 5892 appendix A; no full canonical decomposition is longer than 4 code points; no Hangul syllable has a decomposition in UnicodeData.txt, since idna computes those; and every property value a file holds is one it knows, either one the tables keep or one it reads as other. It reads the test files as strictly: every line of IdnaTestV2.txt has its seven fields, its status codes take the forms the file’s header names, and its one escape is \uXXXX; every line of NormalizationTest.txt has its five columns of code points; and each entry of test/known_failures.json names a line of IdnaTestV2.txt by its number and its source, once, with a reason, for the pinned version of Unicode. Two runs write the same bytes. Its cache is bin/unicode/17.0.0/ of the superproject, which it fills from unicode.org and iana.org when a file is missing. The data are Unicode’s, under the Unicode License v3, whose notice is in the appendix "Unicode data".

The own lane tables runs the generator again, into the build directory, and compares each of its outputs with the committed file, byte for byte, naming each that differs; update-generated writes them in place, and is the one writer of the generated files. The ordinary lanes never run the generator, which may need the network; the CI runs the lane on its own.

Only <webcpp/idna/detail/properties.hpp> includes the tables, and of the public headers only uts46.hpp and validity.hpp reach it, with <webcpp/idna.hpp>, which includes them: punycode.hpp, errors.hpp and options.hpp cost nothing of them. The tables hold 147,324 bytes of data, and a program links only those it reads. A translation unit that includes <webcpp/idna.hpp> and calls to_ascii, compiled at -O2, takes about 0.7 s of processor time and makes an object of 188 KB natively with Apple Clang 21, and about 0.8 s and 197 KB for wasm32-wasip2 with wasi-sdk 34, against 0.14 s and 0.07 s, and under 1 KB, for the same program without idna. Each time is the least of fifteen compilations, user and system time, on an Apple M5 Max shared with other work.

A move is one change of its own, which updates every place that holds the version, in this order:

idna’s tests are lightweight_test programs, one per subject, under test/, each declared in test/Jamfile with webcpp’s webcpp.run. Every vector of the conformance files is compiled into its test, so no test reads a file, and every test runs on every target.

Beside them, test/Jamfile compiles each of the 15 headers alone, and config.hpp three times: as the build asks (config), with WEBCPP_NO_EXCEPTIONS defined for every webcpp library (config_all_off), and with WEBCPP_IDNA_NO_EXCEPTIONS defined by the developer (config_developer_defined). example/Jamfile builds every program this page shows, runs it, and compares what it printed with the .expected beside it.

IdnaTestV2.txt is UTS 46’s own conformance file. idna passes every line of it, which test/known_failures.json records as an empty list: a line listed there must fail in the pass under UTS 46 alone, which the strict pass then leaves out, and every other line must pass, so the list never lies. The one kind of entry it may hold is a line that fails because of a difference between two versions of Unicode, the file’s and the tables’, with a reason that names both. The lines with a lone surrogate, which no UTF-8 string holds, are refused as invalid_utf8, as the file allows. JSON Schema’s own tests of hostname and idn-hostname run in webcpp::lexical, against the validators it builds on idna.

test/Jamfile and example/Jamfile declare webcpp.targets native wasip2 wasip3 emscripten, and b2 builds each program for those alone: natively with any toolset, for wasm32-wasip2 and wasm32-wasip3 with the toolsets clang-wasip2 and clang-wasip3 run by wasmtime, and with Emscripten run by Node.

<webcpp/idna/config.hpp>, which every public header includes first, defines WEBCPP_IDNA_NO_EXCEPTIONS when webcpp has no exceptions (WEBCPP_HAS_EXCEPTIONS is not defined: the compiler has none, or the program defined WEBCPP_NO_EXCEPTIONS), and a program may define it for idna alone, as Boost.Asio’s BOOST_ASIO_NO_EXCEPTIONS. idna raises no exception of its own (guarantee 2), so the macro changes nothing in it: it exists so that a program can set it for every webcpp library alike. The lint compiles every header, test and example a second time without exceptions.

From the root of a webcpp checkout, natively:

builds the 34 programs of test/Jamfile, the tests above, the two run-fail builds of known_failures, each header alone and config.hpp three times, and runs every test among them; and it builds the 4 examples, runs each and compares its output. For wasm32-wasip2, 34 programs and 4 examples, run by wasmtime; toolset=clang-wasip3 does the same for wasm32-wasip3:

With Emscripten, whose programs b2 runs with Node itself, 34 programs and 4 examples:

The own lane tables runs the generator again and compares its outputs with the committed files (The generator):

The test matrix of every lane is at https://webcpporg.github.io/report/.

Punycode (RFC 3492), the encoding of an A-label’s code points after its "xn--" prefix, which UTS 46 and IDNA2008 apply to one label at a time.

decode overloads

Decodes Punycode into a string of code points of the given memory resource.

Decodes Punycode without its "xn--" prefix into code points, into a caller’s buffer (RFC 3492 section 6.2).

encode overloads

Encodes code points as Punycode into a string of the given memory resource.

Encodes code points as Punycode, into a caller’s buffer, without the "xn--" prefix (RFC 3492 section 6.3).

What an operation that can fail returns: a T, or the error_code of the failure.

The flags of UTS 46 section 4, and IDNA2008’s stricter validity.

CheckBidi: every label of a Bidi domain name satisfies RFC 5893 section 2.

CheckHyphens: no hyphens in the third and fourth positions, none at either end (UTS 46 V2, V3); when false, only the "xn--" prefix is refused (V4).

CheckJoiners: the joiners U+200C and U+200D satisfy RFC 5892 A.1 and A.2.

IDNA2008’s validity on top of UTS 46: a code point UTS 46 marks NV8 or XV8 is refused, and the CONTEXTO rules of RFC 5892 A.3 to A.9 apply.

IgnoreInvalidPunycode: an "xn--" label that does not decode is kept as it is, without an error.

Transitional_Processing, which UTS 46 deprecates: the deviations are mapped.

UseSTD3ASCIIRules: the only ASCII code points are a-z, 0-9 and the hyphen (UTS 46 U1).

VerifyDnsLength: labels of 1 to 63 octets, a name of 1 to 253, and no empty root label.

What to_unicode wrote into a caller’s buffer, and the first error UTS 46 recorded.

Empty when the processing succeeded; else its first error, or error::output_too_small when the buffer could not hold the whole result.

The number of chars written.

The string to_unicode produced into a memory resource, and the first error UTS 46 recorded.

Empty when the processing succeeded; else its first error.

The Unicode form, in UTF-8.

The errors idna reports, as boost::system::error_code values of the category error_category.

The input is not well-formed UTF-8 (RFC 3629).

A code point whose UTS 46 status is neither valid nor, in nontransitional processing, deviation (UTS 46 V7), or a U+002E FULL STOP inside a label (UTS 46 V5).

A code point UTS 46 marks NV8 or XV8: DISALLOWED in IDNA2008 (options::idna2008).

An ASCII code point outside a-z, 0-9 and the hyphen (UTS 46 U1, options::use_std3_ascii_rules).

A label not in Normalization Form C (UTS 46 V1, RFC 5891 section 5.4).

Hyphens in both the third and the fourth positions (UTS 46 V2, options::check_hyphens).

A hyphen at the start or the end of a label (UTS 46 V3, options::check_hyphens).

A label that starts with "xn--" while options::check_hyphens is off (UTS 46 V4).

A label that starts with a combining mark (UTS 46 V6, RFC 5891 section 5.4).

A joiner outside the context RFC 5892 A.1 and A.2 allow (IdnaTestV2 C1, C2, options::check_joiners).

A code point outside the context RFC 5892 A.3 to A.9 allow (options::idna2008).

A Bidi domain name with a label that breaks RFC 5893 section 2 (IdnaTestV2 B1 to B6, options::check_bidi).

Punycode that does not decode, an A-label that decodes to nothing or to ASCII alone, or one not in its canonical form (RFC 3492, UTS 46 P4, RFC 5891 section 5.4).

Punycode whose arithmetic overflows 32 bits (RFC 3492 section 6.4, UTS 46 A3), or, in the pmr forms of to_ascii and to_unicode, a label whose Punycode would hold more than 256 code points.

An empty label (UTS 46 A4_2 and X4_2, options::verify_dns_length).

A label longer than 63 octets in its ASCII form (UTS 46 A4_2, options::verify_dns_length).

A domain name longer than 253 octets in its ASCII form (UTS 46 A4_1, options::verify_dns_length).

The caller’s span cannot hold the result.

A label beyond the inline working storage, with no memory resource to hold it.

Checks a domain name, in UTF-8, as it is given: every label as check_label checks it, the Bidi rule across the labels, and the DNS lengths.

Checks one label, in UTF-8, as it is given: a U-label, an A-label or an NR-LDH label.

The category of every error.

Whether check_domain finds no error in domain.

Whether check_label finds no error in label.

Makes the error_code of an error, as its implicit conversion does.

to_ascii overloads

Converts a domain name to its ASCII form into a string of the given memory resource, which also holds a label beyond the inline working storage.

Converts a domain name to its ASCII form (UTS 46 section 4.2), into a caller’s buffer.

to_unicode overloads

Converts a domain name to its Unicode form into a string of the given memory resource, as far as it can.

Converts a domain name to its Unicode form (UTS 46 section 4.3), into a caller’s buffer, as far as it can.

The longest domain name in its ASCII form, in octets, without a trailing root label.

The longest label of a domain name in its ASCII form, in octets (RFC 1034 section 3.1).

The version of Unicode whose data the tables were generated from.

Defined when idna is built without exceptions: when webcpp has none (the compiler has none, or the developer defined WEBCPP_NO_EXCEPTIONS for every webcpp library), or when the developer defines it alone, to disable idna’s in a build that has them.