GeezerVison

Purpose

Parse PostgreSQL string constants with Unicode escapes (U&'...', optionally followed by UESCAPE 'c') with Dialect.POSTGRESQL. Today:

• SELECT U&'\0041BC \00e9' silently becomes U & '\0041BC \00e9': the column U, the & operator and a string. PostgreSQL reads the string ABC é.
• COMMENT ON TABLE t IS U&'\0041BC \00e9' fails because the comment must be a literal. Column defaults such as DEFAULT U&'\0041' get the same bitwise reading.
• U&'d!0061t!+000061' UESCAPE '!' fails in every position.
• There is no way to get the value the server stores.

Current design (As-is)

• S_CHAR_LITERAL knows the prefixes U, E, N, R, B, RB and _utf8, but not U&. U&'x' is lexed as the identifier U, & and 'x'.
• StringValue.Escaping (Resolve backslash escapes in StringValue.getNotExcapedValue() #2791) covers standard strings, MySQL backslash escapes and PostgreSQL E'' strings.
• UESCAPE is not a token.

Target design (To-be)

• Lexing
  • S_CHAR_LITERAL gets a U&'...' alternative. It reads doubled quotes, and backslashes do not escape quotes, as in PostgreSQL.
  • An optional UESCAPE 'c' clause after spaces, tabs or newlines belongs to the same token. Every place that takes a string literal therefore accepts the clause, and no keyword is added.
  • The token is kept only with Dialect.POSTGRESQL. Otherwise the token action shortens it to the identifier U, so U&'x' keeps its current reading. U & 'x', U & 1 and a & b are unchanged in every dialect.
  • With the PostgreSQL dialect, the lexer rejects literals that PostgreSQL rejects: an escape without its hex digits, code point 0 or one above U+10FFFF, an unpaired surrogate, and an escape character that is a hex digit, +, a quote, whitespace or not ASCII. The input is rewound before the error, so lookahead helpers that catch lexer errors cannot skip the literal.
• StringValue
  • New Escaping.POSTGRESQL_UNICODE_ESCAPE, and getUnicodeEscapeCharacter()/setUnicodeEscapeCharacter()/withUnicodeEscapeCharacter(). The value is null when there is no UESCAPE clause.
  • The constructor reads U&'body' [UESCAPE 'c']. The prefix is U& and getValue() is the body as written.
  • getNotExcapedValue() resolves \XXXX, \+XXXXXX, surrogate pairs written as two escapes, doubled escape characters and doubled quotes. It runs in linear time. For a constructed value that PostgreSQL would reject, it throws IllegalArgumentException.
  • toString() and ExpressionDeParser render UESCAPE 'c' after the literal.
• Scope: other U& positions and literal handling are unchanged:
  • A newline-separated continuation of a U& literal is concatenated as before and keeps the Unicode escaping, as in PostgreSQL.
  • UESCAPE after a comment or after a continuation line is not supported.
  • U&"..." quoted identifiers are not covered.
• usage.rst gets a "PostgreSQL Unicode escape strings" section next to "String literal values".

Validation

• New tests:
  • decoded values, for example U&'\0041BC \00e9' → ABC é, U&'d!0061t!+000061' UESCAPE '!' → data, U&'\+01F600' and U&'\D83D\DE00' → 😀, U&'a\\b' → a\b, U&'it''s' → it's;
  • with the PostgreSQL dialect, parse → render → reparse of COMMENT ON, CREATE TABLE ... DEFAULT, ALTER COLUMN ... SET DEFAULT, INSERT ... VALUES, and WHERE/LIKE expressions;
  • 17 invalid literals, rejected in a select list, in function arguments and in COMMENT ON;
  • the default configuration, MySQL, Oracle, SQL Server and ANSI SQL keep the bitwise reading. So does U & 'x' with PostgreSQL. Strings starting with & are unchanged;
  • constructed values, ExpressionDeParser output, and linear-time decoding of 1,000,000 escapes.
• Java 17 offline Gradle spotlessApply check: 10,361 tests, 10,336 passed, 25 skipped, zero failures/errors. The grammar choice conflict check passes.
• Compared with master (bdf8ac31) on the DDL/DCL survey corpora (MySQL 8.4 and PostgreSQL 18, with and without a dialect), every result is unchanged. The PostgreSQL comment corpora have 3 U& comments, which used to fail. They now parse, with values ABC é, ABC é and data.
• MySQL 8.4 and PostgreSQL 18.3: every corpus statement this branch parses ran as written on one server and as rendered by toString() on a second, identical server, in the same order. All 205 MySQL and 327 PostgreSQL statements give the same result (success, or the same error).
• PostgreSQL 18.3 ran SELECT on 39 U& literals covering the escape forms, UESCAPE, surrogate pairs and invalid literals. For the 22 the server accepts, getNotExcapedValue() is byte-identical to the value the server returns. The parser rejects the other 17, which the server also rejects.
• The same check on the comment corpora (statements that create, alter and read MySQL and PostgreSQL comments): all 49 MySQL and 66 PostgreSQL statements this branch parses give the same results, including COMMENT ON TABLE … IS U&'…', which used to fail. Afterwards, every stored comment (MySQL table, column, index, partition, routine and event comments; PostgreSQL pg_description) is byte-identical on both servers.

Integration validation

Applied this PR together with the other comment PRs (#2821, #2822, #2824, #2823), the DDL/DCL PRs #2806–#2819 and #2778 on master (bdf8ac31), with the stacked PRs on top of their bases (#2814 → #2821, #2816 → #2822, #2806 → #2824 and #2811 → #2812 → #2823).

• Each branch merges cleanly into master. Between the branches, conflicts are limited to adjacent additions (README/index rows, usage sections, imports, test methods, visitor and feature lists, an enum line and an alias guard).
• The integrated build passes Java 17 offline Gradle spotlessApply check: 11,877 tests, 11,852 passed, 25 skipped, zero failures/errors. The grammar choice conflict check passes.
• On the corpus of 635 MySQL 8.4 and PostgreSQL 18 DDL/DCL statements, the integrated build parses 265 of 271 MySQL statements (master: 206) and 363 of 364 PostgreSQL statements (master: 328), the same as Share MySQL account names across CREATE USER and DEFINER #2806–Render single spaces in ALTER visibility, FORCE and row level security actions #2819 together. There are no regressions.
• On the comment corpora and an extra set of comment edge cases, the integrated build returns a typed statement for all 80 MySQL statements (master: 67) and for 118 of 119 PostgreSQL statements (master: 76). The exception is a CREATE AGGREGATE setup statement. Every table, column, index, partition, routine, event, user and COMMENT ON comment literal is a StringValue, except COMMENT = _utf8mb4'x', which MySQL rejects.
• On paired MySQL 8.4 and PostgreSQL 18.3 servers, all 263 MySQL and 362 PostgreSQL DDL/DCL statements and all 56 MySQL and 90 PostgreSQL comment statements rendered by the integrated build give the same results as the originals. The stored comments are identical on both servers.