GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Fix source decoding diagnostics and normalize decoded newlines by youknowdot · Pull Request #9429 · RustPython/RustPython

Conversation

youknowdot commented Oct 28, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Summary

Preserve the coding-cookie line for BOM conflicts, recognize CR line boundaries, and continue scanning after invalid coding candidates.
Convert codec initialization failures to SyntaxError while preserving the underlying reason, following CPython's tokenizer initialization error handling.
Normalize successfully decoded source before parsing so literal values, warning ranges, and diagnostic text use the same universal-newline input.
Add the unchanged CPython 3.15 test_source_encoding suite using scripts/update_lib migrate from d220409a13b77de7a6c30abe16ea90db01528edc.

The native changes are confined to the source decoder and its encoding-location record. They do not replace the compiler/parser diagnostic or type-comment work in #8954.

Validation

Focused CLI/VM cargo check and strict Clippy (-D warnings) passed.
Repository hooks passed, including cspell; 228 updater unit tests passed.
All 93 ordinary canonical cases passed, without new skips or assertion changes.
The separate canonical Linux CPU validator passed all 489 fixed subprocess cases within its 300-second wall-time limit.
A bounded CPython/RustPython comparison agrees on CR/CRLF literal values, AST locations, and syntax-error location attributes for compile(str) and compile(bytes).

Scope and remaining differences

This is not a claim of complete tokenizer compatibility. Successful decoder consumers are normalized together; direct string paths that bypass the decoder are not changed. CPython's pre-codec newline conversion and file-stream-relative codec error offsets remain separate work. Generic SyntaxError.args location tuples also remain different even where the location attributes match. No dependency pins or Ruff patches are changed.

AI assistance

OpenAI Codex assisted with the implementation, CPython source comparison, tests, and this pull request description; the model version is unavailable. Full workspace and C-API tests were not run locally; hosted CI remains pending.

Summary by CodeRabbit

Bug Fixes
- Improved recognition of source-file encoding declarations, including declarations on either of the first two lines and files with a UTF-8 byte-order mark.
- Source files using CRLF or CR line endings are now normalized consistently, helping prevent parsing issues.
- Encoding-related errors now provide more useful location and filename information.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          Fix source decoding diagnostics and normalize decoded newlines

1f77499

Track coding-cookie locations, preserve codec initialization errors as SyntaxError, and normalize decoded source before parsing. Add unchanged CPython 3.15 source-encoding coverage via update_lib.

Assisted-by: OpenAI Codex: model version unavailable

coderabbitai Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: RustPython/RustPython
Branch: main
File: .coderabbit.yml
Commit: 3b0dd78
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: 5e6316ec-2984-4f73-b746-7d301e0b8f67

Commits

Reviewing files that changed from the base of the PR and between 3b0dd78 and 1f77499.

Files ignored due to path filters: 1

Lib/test/test_source_encoding.py is excluded by !Lib/**

Files selected for processing: 1

crates/vm/src/vm/compile.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

Source encoding detection now returns cookie metadata, including its line and byte range. The compiler uses that metadata for decoding and error reporting. Successfully decoded source normalizes CRLF and bare CR to LF.

Changes: Source encoding

Layer / File(s)	Summary
Encoding cookie detection `crates/vm/src/vm/compile.rs`	Detection scans up to two physical lines, handles an optional UTF-8 BOM and CR/LF line endings, and returns the encoding name with the cookie’s line and byte range.
Decoding and error reporting `crates/vm/src/vm/compile.rs`	`program_text` passes the detected encoding name to decoding. BOM and codec errors use location data, and successfully decoded source normalizes CRLF and bare CR to LF.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Possibly related issues

RustPython issue 6839: This PR performs one item of the tracked CPython 3.15 standard-library migration work by adding the test_source_encoding suite.

Suggested reviewers: youknowone

Merge Risk: Minimal · up to 1f774

The decoding and newline changes appear mergeable with normal checks. No new merge-blocking defect was identified.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main changes: improved source-decoding diagnostics and normalized decoded newlines.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 1 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

youknowone approved these changes

Oct 28, 2026

View reviewed changes

youknowone marked this pull request as ready for review

October 28, 2026 13:00

chatgpt-codex-connector Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-28T13:02:05.535922Z We couldn’t complete this review. Comment `@codex review` to try again.	`1f77499`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

github-actions Bot commented Oct 28, 2026

Copy link

Copy Markdown

Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] test: cpython/Lib/test/test_startfile.py

dependencies:

dependent tests: (no tests depend on startfile)

[x] lib: cpython/Lib/zipimport.py
[x] test: cpython/Lib/test/test_zipimport.py (TODO: 2)
[ ] test: cpython/Lib/test/test_zipimport_support.py (TODO: 2)

dependencies:

zipimport

dependent tests: (10 tests)

zipimport: test_cmd_line_script test_importlib test_pkgutil test_zipimport test_zipimport_support
- pkgutil: test_pydoc test_pyrepl test_runpy
  - pydoc: test_enum
  - site: test_site

[x] lib: cpython/Lib/pdb.py
[ ] test: cpython/Lib/test/test_pdb.py (TODO: 30)

dependencies:

dependent tests: (2 tests)

pdb: test_pdb test_remote_pdb

[x] test: cpython/Lib/test/test_source_encoding.py

dependencies:

dependent tests: (no tests depend on source_encoding)

[ ] test: cpython/Lib/test/test_setcomps.py

dependencies:

dependent tests: (no tests depend on setcomps)

[x] test: cpython/Lib/test/test_dict.py (TODO: 1)
[ ] test: cpython/Lib/test/test_dictcomps.py
[x] test: cpython/Lib/test/test_dictviews.py (TODO: 1)
[x] test: cpython/Lib/test/test_userdict.py
[x] test: cpython/Lib/test/mapping_tests.py

dependencies:

dependent tests: (no tests depend on dict)

[ ] test: cpython/Lib/test/test_list.py
[ ] test: cpython/Lib/test/test_listcomps.py
[ ] test: cpython/Lib/test/test_userlist.py

dependencies:

dependent tests: (no tests depend on list)

[x] test: cpython/Lib/test/test_generators.py
[ ] test: cpython/Lib/test/test_genexps.py (TODO: 2)
[x] test: cpython/Lib/test/test_generator_stop.py
[x] test: cpython/Lib/test/test_yield_from.py

dependencies:

dependent tests: (no tests depend on generator)

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

youknowone approved these changes

Oct 28, 2026

View reviewed changes

youknowone merged commit 2e84a07 into RustPython:main

Oct 28, 2026

21 checks passed

Uh oh!

There was an error while loading. Please reload this page.

taxor03 mentioned this pull request

Nov 9, 2026

exec(), eval() and -c keep a lone CR in string literals when the source is a str #9761

Open

Copilot AI mentioned this pull request

Nov 9, 2026

Normalize lone CR to LF in str source for exec(), eval() and -c #9762

Closed

Reviewers

youknowone

youknowone approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

youknowdot commented Oct 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Scope and remaining differences

AI assistance

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Oct 28, 2026

📦 Library Dependencies

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

youknowdot commented Oct 28, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

coderabbitai Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

chatgpt-codex-connector Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.