lexical: the string formats of JSON Schema for C++

A program that checks a string of a few formats:

It prints:

2021-02-29 is no date, 2021 being no leap year; //example.com/p is a URI reference and no URI, which needs a scheme.

<webcpp/lexical.hpp> includes every format but regex, whose header <webcpp/lexical/regex.hpp> brings the Unicode tables of ECMA-262’s classes; the names are in the namespaces webcpp::lexical and webcpp::lexical::grammar. lexical is header-only C++20 and needs Boost 1.92 or newer: Boost.URL, whose compiled part webcpp builds from Boost’s sources for every target, and Boost.System; and webcpp’s configuration, config, and webcpp::idna, for internationalized names. A b2 project uses it through /webcpp/lexical//lexical, which brings all of them:

It runs natively, on wasm32-wasip2 and wasm32-wasip3, and on emscripten, with exceptions or without.

is_hostname checks RFC 1123 2.1’s host names: labels of 1 to 63 letters, digits and hyphens, no hyphen first or last, joined by single dots, the name at most 253 octets, no empty label and so no trailing dot. A label that starts with xn--, in any case, is an A-label, an internationalized label in its ASCII form: lexical lowercases it, as UTS 46 maps an ASCII label, and asks webcpp::idna whether IDNA2008 accepts it, decoded, canonical and with the Bidi rule of the label alone, the verdict is_idn_hostname gives that label. Another label with -- in its third and fourth places, an R-LDH label such as ab--cd, is a valid host name, as RFC 1123 says, and not an internationalized one (RFC 5890 2.3.1).

lexical judges each A-label alone, a decision of its own, so RFC 5893’s Bidi rule across the labels of a name, section 2, is checked only by is_idn_hostname: 0a.xn--4db is a host name, while its U-label form, 0a. and a Hebrew letter, is no internationalized host name. The two formats differ here by that decision, as they do on R-LDH labels.

is_idn_hostname checks a host name that may hold any script, by UTS 46’s processing and IDNA2008’s validity, through idna’s to_ascii with its default options (idna’s strict profile). The name is mapped as UTS 46 says, so fullwidth digits, an ignored U+200B, a name not in NFC and the ideographic full stop U+3002 are valid where IDNA2008 accepts the result; then every label must be valid in IDNA2008, CONTEXTO and the Bidi rule across the labels included, and the name fit DNS’s lengths in its ASCII form: labels of at most 63 octets, the name at most 253, and no trailing dot. An R-LDH label such as ab--cd is a host name and not an internationalized one, and the Bidi rule spans an internationalized name’s labels where is_hostname judges each A-label alone. The two formats side by side:

A refused name’s reason is idna’s: the rule answers the idna::error that to_ascii returned, contexto for a middle dot outside l·l, bidi for 0a. and a Hebrew letter.

is_email checks RFC 5321 4.1.2’s Mailbox, the address of an SMTP envelope: a local part of atoms joined by dots or a quoted string, at most 64 octets, then @ and a host name, or an address literal, [192.0.2.1] or [IPv6:2001:db8::1], the tag in any case. RFC 5321’s other literals need a tag registered with IANA, and IPv6 is the only one, so [RFC-5322-domain-literal] is invalid. RFC 5322’s comments and display names are not part of a Mailbox. The whole Mailbox is at most 254 octets: RFC 5321 4.5.3.1.3 limits a path to 256 octets, its angle brackets included.

A literal’s addresses are judged by Boost.URL’s rules, RFC 3986’s IPv4 address and RFC 4291’s IPv6 address, rather than by RFC 5321 4.1.3’s own Snum and IPv6-comp: an IPv4 part with a leading zero, [001.2.3.4], which Snum allows, is invalid, and an IPv6 address whose :: stands for a single group, [IPv6:1:2:3:4:5:6::8], which IPv6-comp refuses, is valid.

Its address literal is a composition of Boost.URL’s combinators, tuple_rule, variant_rule, delim_rule and squelch, over Boost.URL’s ipv4_address_rule and ipv6_address_rule. The domain is not a variant_rule alternative: the rule looks for [ and then asks the literal’s rule or hostname_rule, so that the reason a host name was refused, idna’s for an A-label, reaches the caller; Boost.URL’s variant_rule answers mismatch when every alternative fails. A grammar of a program’s own composes lexical’s rules the same way (Composing rules).

is_idn_email checks RFC 6531’s internationalized Mailbox: the local part may hold any well-formed non-ASCII UTF-8, C1 controls and noncharacters included, and is not normalized; the domain is an internationalized host name or an address literal. A quoted pair stays ASCII. Its lengths are those of the Mailbox as written, in UTF-8: the local part at most 64 octets and the whole at most 254, the form an SMTPUTF8 envelope carries (RFC 6531 extends RFC 5321’s Mailbox without changing its limits, which RFC 5321 counts in octets), whatever the domain’s ASCII form measures; the domain also fits DNS’s lengths in its ASCII form, as is_idn_hostname asks.

lexical is faithful to RFC 3986, the reference JSON Schema names for these formats, where Boost.URL 1.92 is more liberal. lexical refuses four things Boost.URL accepts: a relative reference whose path after an authority is neither empty nor led by /; [ or ] in a query; # in a fragment; and a zone identifier in an IP literal. Its rules also refuse a reference that Boost.URL’s would stop at an authority before a path segment, as at the second @ of http://a@b@example.com/. The tests pin each of these verdicts.

lexical keeps one verdict of Boost.URL that RFC 3986 does not give, since it never loosens Boost.URL: an IPvFuture led by an uppercase V, http://[V1.x]/, is refused, though RFC 5234 makes ABNF’s quoted strings case-insensitive and RFC 3986’s "v" matches it; http://[v1.x]/ is valid.

These are all the differences found on 2026-10-09, when the verdicts of Boost.URL 1.92 and of lexical were compared with a transcription of RFC 3986’s ABNF, on inputs that put every ASCII character but NUL and line feed into each part of a reference, and on references generated at random.

A pattern is valid when ECMA-262 accepts it as a Pattern in u mode, early errors included: when new RegExp(pattern, "u") does not throw a SyntaxError. lexical’s validator reads the pattern as that grammar reads it, and refuses what the grammar refuses there: Annex B’s escapes and classes, \a, \- outside a class, \8, an octal escape; Perl’s \A, \Z and \z; a group of another dialect, (?P<n>...), (?#...), a leading (?i); a property ECMA-262 does not name, as \p{letter} or \p{Script=Foo}; a backreference to a group the pattern does not hold; two groups of one name that a match could both hold. A pattern that is not well-formed UTF-8 is refused, and a NUL is a literal like any other code point.

Valid means that the syntax and the early errors accept the pattern, and nothing more. is_regex compiles no pattern and runs none: it says nothing of how long a matcher takes on a valid pattern, nor of the limits an engine sets on the program it builds from one. The property names \p{...} accepts are Unicode 17.0.0’s, read from the generated tables (The Unicode tables): no ICU and no std::locale.

ECMA-262 allows a lookbehind of any width, (?<=\$\d*), and so does is_regex: it never measures a lookbehind, since it compiles nothing.

is_regex has no rule: a pattern has no grammar of its own inside a longer input, so is_regex judges a whole string.

Two bounds keep a hostile pattern, which JSON Schema meets in an instance, from taking the stack or the time of a program. Past one of them, is_regex answers false without judging the pattern, though ECMA-262 may allow it:

is_regex allocates nothing, on any target. It takes no memory resource: its reader works on the pattern’s own bytes and keeps its state on the stack, to the depth max_regex_nesting allows. So <webcpp/lexical/regex.hpp> keeps lexical’s rule that a validator allocates nothing, as the allocation table says (Allocation).

A test of lexical, regex_v8, compares is_regex with V8 on 24,297 patterns, of which 16,475 are valid by V8’s verdict. The patterns are drawn at random, from seeds 2 to 9 of Python’s random.Random, from a grammar of atoms, groups of every kind, backreferences, assertions, quantifiers and defects; every name \p{...} accepts, under each of its forms, and the same names misspelt; every pattern the JSON-Schema-Test-Suite’s schemas hold, in pattern and patternProperties, and every string of its tests of the format regex; and the cases the grammar does not draw: ECMAScript 2025’s modifiers and duplicate group names, lookbehinds of every width, escapes of every kind, and classes and quantifiers at their edges. V8’s verdict is new RegExp(pattern, "u") throwing a SyntaxError or not, written by the oracle of test/oracle, which runs the pinned Node, 26.7.0, whose V8 is 14.6 and whose Unicode is 17.0. On each of these patterns the verdict of is_regex is V8’s, on every target (Testing).

To check the committed verdicts against the pinned Node, and is_regex against them:

After a change to the patterns, update-expected writes the verdicts again:

tools/generate_tables.py, in lexical’s repository, writes <webcpp/lexical/detail/regex_tables.hpp> from Unicode’s data files of version 17.0.0, pinned in tools/unicode.json, each by its URL under https://www.unicode.org/Public/17.0.0/ and its SHA-256: a file its cache lacks is downloaded, every file is hashed, and a digest that differs stops it before it writes anything. Before it writes, it checks what the validator and the traits assume of the data: General_Category has 30 values that are no group; every value a file holds is one PropertyValueAliases.txt names; each of ECMA-262’s binary properties holds a code point; every name is ASCII and names one value; the classes fit the traits’ mask; and no simple case folding orbit holds more than four code points. A script that no code point has, Katakana_Or_Hiragana, gets no name, as ECMA-262’s table of the values of Script leaves it out. tools/generate_patterns.py writes the patterns regex_v8 reads, the same way.

The own lane generated runs both generators again, into the build directory, and compares each output with the committed file, byte for byte; update-generated writes them in place, the one writer of the generated files. The ordinary lanes never run them, since a generator may need the network. The data are Unicode’s, under the Unicode License v3, whose notice is in the appendix "Unicode data".

What a rule returns: the value it matched, or the error that refused its input.

The text a rule of lexical matched, a view of its input.

The rule of RFC 3339 5.6’s full-date: a year, a month and a day that exists.

The rule yields the text it matched.

Matches a full-date at it.

The rule of RFC 3339 5.6’s date-time: a full-date, T in either case, a full-time.

The rule yields the text it matched.

Matches a date-time at it.

The rule of RFC 3339 appendix A’s duration, exactly: P, then a date part, a time part, or weeks alone.

The rule yields the text it matched.

Matches a duration at it.

The rule of RFC 5321 4.1.2’s Mailbox: a local part, the at sign, and a domain or an address literal.

The rule yields the text it matched.

Matches a Mailbox at it.

The rule of a host name: RFC 1123 2.1’s labels, with each A-label judged as idn-hostname judges it.

The rule yields the text it matched.

Matches a host name at it.

The rule of RFC 6531 3.3’s internationalized Mailbox: email_rule with UTF-8 in the local part and an internationalized domain.

The rule yields the text it matched.

Matches an internationalized Mailbox at it.

The rule of an internationalized host name: UTS 46’s processing and IDNA2008’s validity, through webcpp::idna::to_ascii.

The rule yields the text it matched.

Matches an internationalized host name at it.

The rule of one label of an internationalized host name, a terminal of lexical’s own that asks webcpp::idna for its verdict.

The rule yields the text it matched.

Matches one internationalized label at it.

The rule of RFC 3987 2.2’s IRI-reference: an IRI or a relative reference, checked and mapped as iri_rule_t does, then parsed by uri_reference_rule.

The rule yields the text it matched.

Matches an IRI-reference at it.

The resource of a mapped URI longer than 1,024 octets; null for the default resource.

The rule of RFC 3987 2.2’s IRI: an IRI checked code point by code point, mapped to a URI, and parsed by uri_rule.

The rule yields the text it matched.

Matches an IRI at it.

The resource of a mapped URI longer than 1,024 octets; null for the default resource.

The rule of RFC 6901 section 3’s json-pointer: zero or more reference tokens, each led by /.

The rule yields the text it matched.

Matches a json-pointer at it.

The rule of the relative-json-pointer of draft-bhutton-relative-json-pointer-00 section 3 and draft-handrews-relative-json-pointer-01 section 3: a non-negative integer, an optional index manipulation, then # or a json-pointer.

The rule yields the text it matched.

Matches a relative-json-pointer at it.

Whether an index manipulation may follow the integer.

The rule of RFC 3339 5.6’s full-time: a time of day and its offset from UTC.

The rule yields the text it matched.

Matches a full-time at it.

Whether second 60 may be valid.

The rule of RFC 3986 4.1’s URI-reference: Boost.URL’s uri_reference_rule, then a guard.

The rule yields Boost.URL’s view of the reference it matched.

Matches a URI-reference at it.

The rule of RFC 3986 3’s URI: Boost.URL’s uri_rule, then a guard.

The rule yields Boost.URL’s view of the URI it matched.

Matches a URI at it.

The rule of RFC 6570 section 2’s URI-Template, with erratum 6937: literals and expressions.

The rule yields the text it matched.

Matches a URI-Template at it.

The rule of RFC 9562 section 4’s UUID: 32 hexadecimal digits in groups of 8, 4, 4, 4 and 12, joined by hyphens.

The rule yields the text it matched.

Matches a UUID at it.

The rule of RFC 3339’s full-date.

The rule of RFC 3339’s date-time.

The rule of RFC 3339 appendix A’s duration.

The rule of RFC 5321’s Mailbox.

The rule of a host name.

The rule of RFC 6531’s Mailbox.

The rule of an internationalized host name.

The rule of one internationalized label.

The rule of RFC 3987’s IRI-reference, with the default resource.

The rule of RFC 3987’s IRI, with the default resource; iri_rule_t{&resource} names another.

The rule of RFC 6901’s json-pointer.

The rule of the relative JSON pointer, index manipulation allowed; relative_json_pointer_rule_t{index_manipulation::refused} refuses it.

The rule of RFC 3339’s full-time, leap seconds allowed; time_rule_t{leap_seconds::refused} refuses them.

The rule of RFC 3986’s URI-reference, guarded.

The rule of RFC 3986’s URI, guarded.

The rule of RFC 6570’s URI-Template.

The rule of RFC 9562’s UUID.

Whether a relative JSON Pointer may move its array index with a sign and an integer.

An index manipulation is valid, as in draft-bhutton-00, which 2020-12 and v1 cite.

An index manipulation is refused, as in draft-handrews-01, which 7 and 2019-09 cite.

Whether a time may hold second 60, a leap second.

Second 60 is valid when the time moved to UTC is 23:59, as RFC 3339 5.7 allows.

Second 60 is never valid, as JSON Schema v1’s time asks.

Whether s is an RFC 3339 full-date, JSON Schema’s date.

Whether s is an RFC 3339 date-time, JSON Schema’s date-time.

Whether s is an RFC 3339 appendix A duration, JSON Schema’s duration.

Whether s is an RFC 5321 Mailbox, JSON Schema’s email.

Whether s is a host name, JSON Schema’s hostname.

Whether s is an RFC 6531 internationalized Mailbox, JSON Schema’s idn-email.

Whether s is an internationalized host name, JSON Schema’s idn-hostname.

Whether s is an IPv4 address in dotted-quad form, JSON Schema’s ipv4.

Whether s is an IPv6 address, JSON Schema’s ipv6.

is_iri overloads

Whether s is an RFC 3987 IRI, JSON Schema’s iri.

Whether s is an RFC 3987 IRI, a mapped URI beyond 1,024 octets taken from resource.

is_iri_reference overloads

Whether s is an RFC 3987 IRI-reference, JSON Schema’s iri-reference.

Whether s is an RFC 3987 IRI-reference, a mapped URI beyond 1,024 octets taken from resource.

Whether s is an RFC 6901 JSON Pointer, JSON Schema’s json-pointer.

Whether pattern is a regular expression, JSON Schema’s regex: an ECMA-262 Pattern in u mode, with ECMAScript 2025’s modifiers and duplicate group names.

Whether s is a relative JSON Pointer, JSON Schema’s relative-json-pointer.