GeezerVison is showing github.com — `patchcheck.py` raises `SystemError` on `main` · Issue #105820 · python/cpython
patchcheck.py raises SystemError on main #105820
DescriptionBug report
Following #105801, running python Tools/patchcheck/patchcheck.py now results in SystemError being raised.
To reproduce
- Make any change to a
.py file, e.g.:
--- a/Lib/test/test_fstring.py
+++ b/Lib/test/test_fstring.py
@@ -19,6 +19,7 @@
a_global = 'global variable'
+
- Run
python Tools/patchcheck/patchcheck.py:
C:\Users\alexw\coding\cpython>python Tools/patchcheck/patchcheck.py
Running Release|x64 interpreter...
Getting base branch for PR ... upstream/main
Getting the list of files that have been added/changed ... 1 file
Fixing Python file whitespace ... Traceback (most recent call last):
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 325, in <module>
main()
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 293, in main
normalize_whitespace(python_files)
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 35, in call_fxn
result = fxn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 188, in normalize_whitespace
and reindent.check(os.path.join(SRCDIR, path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\reindent.py", line 138, in check
if r.run():
^^^^^^^
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\reindent.py", line 203, in run
for _token in tokens:
File "C:\Users\alexw\coding\cpython\Lib\tokenize.py", line 537, in _generate_tokens_from_c_tokenizer
for info in it:
SystemError: Negative size passed to PyUnicode_New
This failure was visible on the Azure Pipelines job on the PR FYI: https://dev.azure.com/Python/cpython/_build/results?buildId=130389&view=logs&j=256d7e09-002a-52d7-8661-29ee3960640e&t=3d7276d3-4e8d-5309-55ad-fb0b172d9925
Your environment
Python 3.13.0a0 (heads/main:c5111aec2b, Jun 15 2023, 10:53:37) [MSC v.1932 64 bit (AMD64)] on win32
Linked PRs
Activitypablogsal commented on Jun 15, 2023 AlexWaygood commented on Jun 15, 2023 Unlike the "hypothesis checks on Ubuntu" job, though, Azure Pipelines is incredibly flaky, it fails all the time for random reasons. So it's not realistic to make that one "required" in CI. pablogsal commented on Jun 15, 2023
So it's not realistic to make that one "required" in CI.
Then we should move the parts we care about to some other check and just remove IMHO because nobody looks at it when merging because is constantly failing. AlexWaygood commented on Jun 15, 2023
nobody looks at it when merging
"nobody" isn't accurate — I generally check failures to see if they're relevant before merging a PR. But I agree with:
Then we should move the parts we care about to some other check and just remove
And we've actually started doing that already; the new "lint" job in CI is basically meant to be a replacement for patchcheck.py, for example. Personally I don't care about patchcheck.py; I only opened this issue because I worried this might be indicative of other breakages :) pablogsal commented on Jun 15, 2023
I only opened this issue because I worried this might be indicative of other breakages
Thanks a lot @AlexWaygood, this is certainly the case and we are investigating. Thanks for all the gray work 🖤 hugovk commented on Jun 15, 2023 Similarly, see #105823 to remove some duplicated docs building from Azure Pipelines. lysnikolaou commented on Jun 15, 2023 I don't really know why we're seeng this just now, but git bisecting shows that this is the first bad commit:
9216e69a87d16d871625721ed5a8aa302511f367 is the first bad commit
commit 9216e69a87d16d871625721ed5a8aa302511f367
Author: Pablo Galindo Salgado <[email protected]>
Date: Tue May 30 22:43:34 2023 +0100
gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070)
The reproducer I used was (the good case):
cpython (2ea34cf) (BISECTING) via C v14.0.3-clang via 🐍 pyenv 3.11.3
❯ cat tmp/t.py
self.assertEqual(f'''{
3
=}''', '\n3\n=3')%
cpython (2ea34cf) (BISECTING) via C v14.0.3-clang via 🐍 pyenv 3.11.3
❯ ./python.exe -m tokenize tmp/t.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,4: NAME 'self'
1,4-1,5: OP '.'
1,5-1,16: NAME 'assertEqual'
1,16-1,17: OP '('
1,17-1,21: FSTRING_START "f'''"
1,21-1,22: OP '{'
1,22-1,23: NL '\n'
2,0-2,1: NUMBER '3'
2,1-2,2: NL '\n'
3,0-3,1: OP '='
3,1-3,2: OP '}'
3,2-3,5: FSTRING_END "'''"
3,5-3,6: OP ','
3,7-3,16: STRING "'\\n3\\n=3'"
3,16-3,17: OP ')'
3,17-3,18: NEWLINE '\n'
4,0-4,0: ENDMARKER '' I'm working on a fix. Reacted by Alex Waygood, Pablo Galindo Salgado and Hugo van Kemenade added 2 commits that reference this issue on Jun 15, 2023gh-105820: Fix tok_mode expression buffer in file & readline tokenizer ( lysnikolaou commented on Jun 15, 2023
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|