GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-145234: Fix SystemError in parser when \r is introduced after code… by gourijain029-del · Pull Request #145276 · python/cpython

Conversation

@gourijain029-del

@gourijain029-del gourijain029-del commented Feb 26, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Description
This PR fixes a SystemError: Parser/string_parser.c:286: bad argument to internal function that occurred when a Python file used an encoding (like UTF-7) that introduced \r characters after decoding.

Root Cause
The crash was caused by a synchronization failure between the tokenizer, the lexer, and the string parser:

Tokenizer: When the file tokenizer recoded a line (e.g., from UTF-7 to UTF-8), it was not normalizing newlines. If the codec introduced a \r, it remained in the buffer.
Lexer: The lexer skipped \r characters but did not correctly trigger "beginning-of-line" (atbol) logic. This meant that if a \r followed a comment (#...), the lexer would remain in a state where it thought it was still on the same line, causing it to merge the comment and the subsequent string literal into a single, invalid token.
String Parser: When

_PyPegen_parse_string
received this broken token (which didn't start with a quote character), it raised a SystemError.
Changes

Parser/lexer/lexer.c
: Updated the lexer to treat a standalone \r as a full newline. It now correctly sets atbol = 1 and resets the current token start, preventing the "merging" of tokens across lines.

Parser/tokenizer/file_tokenizer.c
:
Updated

tok_readline_recode
to explicitly call

_PyTokenizer_translate_newlines
on the UTF-8 decoded buffer.
Optimized

tok_underflow_file
to immediately discard and re-decode the buffer as soon as a coding spec is identified, preventing raw bytes from leaking into the parser.

Lib/test/test_parser_utf7_r.py
: Added a new regression test that uses a UTF-7 encoded \r to reproduce the original crash.

  • Issue: \rs introduced after codec decoding cause SystemError #145234

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@gourijain029-del
pythongh-145234: Fix SystemError in parser when \r is introduced afte…
ccf6915
…r codec decoding
@gourijain029-del
gourijain029-del requested review from lysnikolaou and pablogsal as code owners February 26, 2026 21:02
@bedevere-app bedevere-app Bot added the awaiting review label Feb 26, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Feb 26, 2026
\rs introduced after codec decoding cause SystemError #145234
Closed
@bedevere-app

bedevere-app Bot commented Feb 26, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Feb 26, 2026

Copy link
Copy Markdown
Member

Closing as per https://devguide.python.org/getting-started/generative-ai/

Please don't randomly open AI generated PRs

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal pablogsal closed this Feb 26, 2026
@gourijain029-del
gourijain029-del deleted the fix-parser-r-systemerror branch February 28, 2026 04:21
@pablogsal pablogsal mentioned this pull request Mar 1, 2026
gh-144957: Add test for lazy imports with __getattr__ #145330
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@gourijain029-del @pablogsal
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.