GeezerVison

ParseFile returns the tree with the input's comments (#12)

* ParseFile returns the tree with the input's comments

A formatter cannot put comments back where they were written without
being told where that was, and the grammar is the wrong place to ask:
libpg_query's patch 04 has the scanner emit comments as tokens, and
base_yylex drops them on the way to the parser. Scan exposes that channel,
so Scan plus Parse already answers the question — by lexing the input
twice and marshalling the entire token stream to protobuf to recover a
handful of comments. On a six-statement query file that is 138us and 689
allocations against ParseFile's 70us and 444.

parser.ParseFile keeps the comments from the pass the parse already makes
and returns them beside the tree, as the ScanToken values Scan would have
reported. It is oliphant's second deliberate addition to pg_query_go's
surface, alongside ParseToTree, and like it lives in the parser subpackage
so the root package stays a mirror function-for-function.

The scanner collects at the one site every token is minted, rather than at
the filter's drop site. A successful parse does route every comment
through that drop site, but not every path there goes through
Filter.Next — the UIDENT/UESCAPE resolution pulls tokens off to the
side — and the mint site cannot be bypassed at all. Collection is opt-in
via NewKeepingComments, and the parse path is unchanged when it is off:
interleaved runs of the existing parse benchmarks show no difference.

The corpus is what pins the result. Across all 46,756 cases that parse,
ParseFile's tree is proto.Equal to ParseToTree's, and its comments are
exactly the SQL_COMMENT and C_COMMENT tokens Scan reports for the same
input — 11,880 cases carry at least one, so the oracle-derived scan
goldens stand behind the comment spans too. Focused tests cover the
positions a query file puts comments in (above a statement, trailing one
on its terminator's line, inside one, between two, and after the last),
nested block comments arriving as the single token PostgreSQL scans them
as, and a failed parse returning no half-scanned list.

Unlike Parse, ParseFile does not reject strings that are not valid UTF-8:
that constraint is protobuf's, not the grammar's, and nothing here encodes
the tree.

Co-Authored-By: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01MB4HGvrQk93hmnpSPm21N8

* ParseFile answers the UTF-8 rejection as Parse does

ParseFile skipped the proto3 UTF-8 rejection Parse keeps, on the grounds
that the constraint is the wire format's rather than the grammar's. Two
entry points disagreeing about the same input is worse than the quirk, so
it now answers as Parse does.

The check cannot be on the input. An invalid byte fails only where it
lands in a string field, and a comment is not one, so "SELECT 1 -- \xff"
marshals and Parse returns its tree — and a comment is precisely what
ParseFile must not reject one for. So it decides on the tree, marshalling
it only when utf8.ValidString(input) is already false, which is the same
guard Parse takes its own fallback on and costs nothing on valid input.

TestParseFileInvalidUTF8 pins both sides against Parse: an invalid byte in
a string field or an identifier fails with the same error from both, and
one confined to a line or block comment parses for both, with the comment
kept.

Co-Authored-By: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01MB4HGvrQk93hmnpSPm21N8

---------

Co-authored-by: Claude <[email protected]>

Advance the pin to libpg_query 18.0.0 (PostgreSQL 18.4) (#10)