GeezerVison

Problem

A numeric character reference to a code point in the surrogate range is kept as a lone surrogate:

Parser.builder().build().parse("&#xD800;");   // <p>\uD800</p>

The spec, Entity and numeric character references: "Invalid Unicode code points will be replaced by the REPLACEMENT CHARACTER (U+FFFD)". A code point in the surrogate range (U+D800–U+DFFF) is not a valid Unicode code point. The spec.txt used by the tests only exercises �, so the spec tests do not cover this.

Both reference implementations replace the range: cmark in houdini_unescape_ent (src/houdini_html_u.c: codepoint == 0 || (codepoint >= 0xD800 && codepoint < 0xE000) || codepoint >= 0x110000 → 0xFFFD), and commonmark.js 0.31.2 renders <p> + U+FFFD + </p> for the same input.

Because a lone surrogate is not valid UTF-16, the corruption is visible downstream as well: writing the rendered HTML as UTF-8 replaces it with ?.

Fix

Html5Entities.entityToString only special-cased 0 and relied on Character.toChars throwing for values above U+10FFFF, but Character.toChars(0xD800) succeeds. The surrogate range is now rejected in the same way as 0, using Character.MIN_SURROGATE/MAX_SURROGATE, so valid supplementary code points are not affected.

entityToString is also used when unescaping link destinations and link reference definitions, so those are covered by the same change.

Tests

Added SpecialInputTest.surrogateCharacterReferenceShouldBeReplaced, next to the existing nullCharacterEntityShouldBeReplaced:

• �, �, � and before�after → U+FFFD
• 😀 (U+1F600) and  unchanged

Verification:

• Base commit without the fix: Tests run: 1, Failures: 1
• With the fix: SpecialInputTest → Tests run: 33, Failures: 0
• mvn -B package on all modules with JDK 17: BUILD SUCCESS, including the spec tests (1959 tests, 0 failures)
• mvn spotless:check: clean