GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||
GeezerVison is showing github.com — gh-105820: Fix tok_mode expression buffer in file & readline tokenizer by lysnikolaou · Pull Request #105828 · python/cpython Conversation
|
|
@lysnikolaou Hummm, can you quickly cover why moving this chunk solves the problem? I think I am missing some key insight here |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
Sure. There's two different cases to consider here:
cpython on main via C v14.0.3-clang via 🐍 pyenv 3.11.3 ❯ cat tmp/t.py print(f'''{ 3 =}''')% cpython on main via C v14.0.3-clang via 🐍 pyenv 3.11.3 ❯ ./python.exe -m tokenize tmp/t.py unexpected error: Negative size passed to PyUnicode_New Traceback (most recent call last): File "/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py", line 198, in _run_module_as_main return _run_code(code, main_globals, None, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py", line 88, in _run_code exec(code, run_globals) File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 547, in <module> main() File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 489, in main tokens = list(tokenize(f.readline)) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 447, in tokenize yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True) File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 537, in _generate_tokens_from_c_tokenizer for info in it: SystemError: Negative size passed to PyUnicode_New
cpython on main via C v14.0.3-clang via 🐍 pyenv 3.11.3 took 15s ❯ cat tmp/t.py print(f'''{ 3 =}''')% cpython on main via C v14.0.3-clang via 🐍 pyenv 3.11.3 ❯ ./python.exe tmp/t.py 3 3 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Do you think I should split this PR in two and open a second issue for the second case above? |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
2efb0a4 to
a83de94
Compare
Nah, I think is fine, but we may want to add two NEWS entries |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thanks for the explanation @lysnikolaou. Fantastic work as always! 💪
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
@lysnikolaou If you want add the second NEWS entry and then land it. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Thanks @lysnikolaou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…kenizer (pythonGH-105828) (cherry picked from commit d382ad4) Co-authored-by: Lysandros Nikolaou <[email protected]>
|
GH-105832 is a backport of this pull request to the 3.12 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…okenizer (GH-105828) (#105832) (cherry picked from commit d382ad4) Co-authored-by: Lysandros Nikolaou <[email protected]>
* main: (57 commits) pythongh-105831: Fix NEWS blurb from pythongh-105828 (python#105833) pythongh-105820: Fix tok_mode expression buffer in file & readline tokenizer (python#105828) pythongh-105751, test_ctypes: Remove disabled tests (python#105826) pythongh-105821: Use a raw f-string in test_httpservers.py (python#105822) pythongh-105751: Remove platform usage in test_ctypes (python#105819) pythongh-105751: Reenable disable test_ctypes tests (python#105818) pythongh-105751: Remove dead code in test_ctypes (python#105817) More reorganisation of the typing docs (python#105787) Improve docs for `typing.dataclass_transform` (python#105792) pythonGH-89812: Churn `pathlib.Path` test methods (python#105807) pythongh-105800: Issue SyntaxWarning in f-strings for invalid escape sequences (python#105801) pythongh-105751: Cleanup test_ctypes imports (python#105803) pythongh-105481: add HAS_JUMP flag to opcode metadata (python#105791) pythongh-105751: test_ctypes avoids the operator module (pythonGH-105797) pythongh-105751: test_ctypes: Remove @need_symbol decorator (pythonGH-105798) pythongh-104909: Implement conditional stack effects for macros (python#105748) pythongh-75905: Remove test_xmlrpc_net: skipped since 2017 (python#105796) pythongh-105481: Fix types and a bug for pseudos (python#105788) Update DSL docs for cases generator (python#105753) pythonGH-77273: Better bytecodes for f-strings (pythonGH-6132) ...
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.