GeezerVison

Summary

Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer.

Reuse the existing collected warning batch, in source order, when it contains at least one tokenizer-origin escape and no parser-origin escape. Numeric warnings may occur in the same batch. Keep tokenizer-error recovery disabled: an escaped-brace warning promoted to an error must stop before a later numeric warning.

This restores missing warnings for ordinary compile("t'\\{1}'", ..., "eval") and fixes ordering/error selection when an f/t-string escaped-brace warning precedes a numeric warning. Parser-only input keeps the existing #9296 path.

No warning messages, exception metadata/args, raw-string handling, or callback conversion are changed.

Deliberately excluded

• Mixed parser/tokenizer escape events. Template body decoding and nested ordinary strings have different timing; sorting every warning by source position would be incorrect.
• Invalid source, including invalid eval input that is valid as a module.
• single, interactive/prompt, AST-only and BlockExpr paths.
• Lexical-error recovery such as b'\z' + 0o8 under the error filter.

Existing differences in those cases remain. Tokenizer error offsets/text/args also remain outside this patch.

Validation

Tested base: 0b0a3607648b5b01b3adfc1c06f3bec1d87f8a13. Candidate: 11c81757ef68e77c7aecddf104937de4beaf9926.

Compared actual sealed RustPython binaries with final CPython 3.15.0 (d220409a13b77de7a6c30abe16ea90db01528edc). The historical baseline is eb79fef279d646aedf5fca6598d330b54a401138; its warning implementation matches this PR's main base. Source-sensitive native crates were rebuilt for the candidate rather than reusing another worktree's VM artifact.

• Focused 18 observations: warning/error-selection matches improved from 8/18 to 13/18; complete raw matches, including exception fields/args/repr, improved from 1/18 to 4/18. Exactly five outputs changed, all as intended.
• Existing 24 observations: selection matches improved from 18/24 to 20/24; complete raw matches improved from 10/24 to 11/24. Only the two f-string tokenizer-before-numeric cases changed.
• No raw or selection regression in either set. Mixed-origin, invalid-source, Single, AST-only and invalid-octal controls retained their baseline outputs.
• Three existing canonical test_fstring methods passed: test_backslashes_in_string_part, test_fstring_backslash_before_double_bracket, and test_fstring_backslash_before_double_bracket_warns_once.
• CLI build with capi; scoped VM threading, nonthreading, no-default-feature and test-target checks; strict all-targets VM Clippy passed. These are scoped checks, not a full test-suite claim.

Concurrent changes

Source-only composition was reviewed with #9304 and #9314; a combined runtime build was not performed. When resolving #9304, preserve the explicit tokenizer_recovery: !tokenizer_only rather than its earlier unconditional true. The f/t-body diagnostic change adds a field to the escape event, so the origin-only patterns need .. when those changes are combined.

AI assistance

OpenAI Codex assisted with the implementation, source investigation, validation and PR description.

Summary by CodeRabbit

• Bug Fixes
  • Improved warning reporting in tokenizer-only execution modes. Escape-related warnings originating during tokenization are now reported as a collected batch without invoking the parser-related fallback or tokenizer recovery.
  • Existing fallback behavior is unchanged for invalid syntax and for warnings originating from both tokenization and parsing. This preserves the established handling of those cases while making tokenizer-only warning reports more consistent.

Conflict resolution validation (2026-10-23)

Merged main afe5d9030a2c2722642699b00c71f24301805826 into this branch without rewriting its history. Preserved #9304's Single input boundary and recovery behavior, while retaining tokenizer-only Exec/Eval eligibility and tokenizer_recovery: !tokenizer_only. The PR diff remains one Rust file, +17/-4; no library or test changes were added.

Validated tree 1062d4076b918cc9e238ec1fce7ac88d48b036cc, committed unchanged as 78a96fde5a904ba48b7c45d45324ebd0c8554423:

• Rust 1.99.1 VM library check, strict VM library Clippy, and default debug CLI build passed. Normal commit hooks passed without changing the tested tree.
• All three selected, unmodified canonical CPython test_fstring methods passed: test_backslashes_in_string_part, test_fstring_backslash_before_double_bracket, and test_fstring_backslash_before_double_bracket_warns_once.
• A fresh finite set of 32 compile observations compared the built candidate with final CPython 3.15.0 (d220409a13b77de7a6c30abe16ea90db01528edc). Warning/error selection matched in 29/32, including all 12 eligible tokenizer-only observations. Complete raw outputs matched in 12/32. The three selection differences were the explicitly excluded invalid-eval/module input under both filters and invalid-octal input under the error filter. Both Single boundary controls matched selection under both filters.

These are scoped checks, not a full-suite result. No selected-main baseline binary was rebuilt, so the counts are candidate-versus-CPython parity, not measured before/after improvements. Existing exception metadata/args differences remain outside this patch. The canonical tests used their unmodified source file with the candidate's real standard-library/support modules; no fakes or test-marker changes were used.

Source-only mergeability was checked against newer main 5dcef1aa6d53c72315a44296146c768120841953; the merge was clean. No combined runtime build with that newer main was performed.