GeezerVison is showing github.com — gh-159739: Preserve CJK codec state across chunk boundaries and callbacks by jelle-openai · Pull Request #160400 · python/cpython
Conversation
jelle-openai
commented
Oct 22, 2026
•
edited by bedevere-app
Bot
Loading
Uh oh!
There was an error while loading. Please reload this page.
The HZ encoder can consume a tilde after writing only its first escape byte. A split ISO-2022 escape can exceed the incremental decoder’s pending buffer, and a reset can leave the previous character set selected. Reserve space for both HZ bytes before consuming the tilde, allow the full pending escape, and restore the initial ISO-2022 state on reset. Keep the native codec table and the active encoder error policy alive so replacing their Python owners cannot leave a codec using freed memory.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
… callbacks
The HZ encoder can consume a tilde after writing only its first escape byte. A split ISO-2022 escape can exceed the incremental decoder’s pending buffer, and a reset can leave the previous character set selected. Reserve space for both HZ bytes before consuming the tilde, allow the full pending escape, and restore the initial ISO-2022 state on reset. Keep the native codec table and the active encoder error policy alive so replacing their Python owners cannot leave a codec using freed memory.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Sorry, @jelle-openai and @JelleZijlstra, I could not cleanly backport this to 3.15 due to a conflict.
Please backport manually with cherry_picker, see the devguide for more information.
cherry_picker a96fbdca7fe1216f9c6079a570327a283d844aee 3.15
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Sorry, @jelle-openai and @JelleZijlstra, I could not cleanly backport this to 3.14 due to a conflict.
Please backport manually with cherry_picker, see the devguide for more information.
cherry_picker a96fbdca7fe1216f9c6079a570327a283d844aee 3.14
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
gpshead
added a commit
that referenced
this pull request
Nov 1, 2026
…d callbacks (GH-160400) (#161070)
The HZ encoder can consume a tilde after writing only its first escape byte. A split ISO-2022 escape can exceed the incremental decoder’s pending buffer, and a reset can leave the previous character set selected. Reserve space for both HZ bytes before consuming the tilde, allow the full pending escape, and restore the initial ISO-2022 state on reset. Keep the native codec table and the active encoder error policy alive so replacing their Python owners cannot leave a codec using freed memory.
(cherry picked from commit a96fbdc)
Co-authored-by: Jelle Zijlstra <[email protected]>
gpshead
added a commit
that referenced
this pull request
Nov 1, 2026
…d callbacks (GH-160400) (#161069)
The HZ encoder can consume a tilde after writing only its first escape byte. A split ISO-2022 escape can exceed the incremental decoder’s pending buffer, and a reset can leave the previous character set selected. Reserve space for both HZ bytes before consuming the tilde, allow the full pending escape, and restore the initial ISO-2022 state on reset. Keep the native codec table and the active encoder error policy alive so replacing their Python owners cannot leave a codec using freed memory.
(cherry picked from commit a96fbdc)
Co-authored-by: Jelle Zijlstra <[email protected]>
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot AMD64 Windows11 Non-Debug 3.14 (tier-1) has failed when building commit d67f2a9.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1684/builds/1089) and take a look at the build logs.
- Check if the failure is related to this commit (d67f2a9) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/#/builders/1684/builds/1089
Failed tests:
Failed subtests:
- testBoth - test.test_zipimport.UncompressedZipImportTestCase.testBoth
- testGetCompiledSource - test.test_zipimport.CompressedZipImportTestCase.testGetCompiledSource
- testGetCompiledSource - test.test_zipimport.UncompressedZipImportTestCase.testGetCompiledSource
- testBoth - test.test_zipimport.CompressedZipImportTestCase.testBoth
Summary of the results of the build (if available):
==
Click to see traceback logs
Traceback (most recent call last):
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 791, in testGetCompiledSource
self.doTest(pyc_ext, files, TESTMOD, call=self.assertModuleSource)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 161, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 175, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[49 chars]\test_python_4104�\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[49 chars]\test_python_4104�\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_4104�\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_4104�\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (test.test_zipimport.UncompressedZipImportTestCase.testTraceback) ... ok
Traceback (most recent call last):
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 222, in testBoth
self.doTest(pyc_ext, files, TESTMOD)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 161, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 175, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[49 chars]\test_python_4104�\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[49 chars]\test_python_4104�\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_4104�\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_4104�\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (most recent call last):
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 222, in testBoth
self.doTest(pyc_ext, files, TESTMOD)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 161, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 175, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[49 chars]\test_python_2168�\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[49 chars]\test_python_2168�\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_2168�\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_2168�\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (most recent call last):
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 791, in testGetCompiledSource
self.doTest(pyc_ext, files, TESTMOD, call=self.assertModuleSource)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 161, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.14.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 175, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[49 chars]\test_python_2168�\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[49 chars]\test_python_2168�\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_2168�\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.14.ware-win11.nondebug\build\build\test_python_2168�\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (test.test_zipimport.CompressedZipImportTestCase.testTraceback) ... ok
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot AMD64 Windows11 Non-Debug 3.15 (tier-1) has failed when building commit 06367c6.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/2014/builds/444) and take a look at the build logs.
- Check if the failure is related to this commit (06367c6) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/#/builders/2014/builds/444
Failed tests:
Failed subtests:
- testGetCompiledSource - test.test_zipimport.DeflateCompressedZipImportTestCase.testGetCompiledSource
- testBoth - test.test_zipimport.UncompressedZipImportTestCase.testBoth
- testGetCompiledSource - test.test_zipimport.UncompressedZipImportTestCase.testGetCompiledSource
- testGetCompiledSource - test.test_zipimport.ZStdCompressedZipImportTestCase.testGetCompiledSource
- test_syntax_error_in_source_with_valid_pyc - test.test_zipimport.UncompressedZipImportTestCase.test_syntax_error_in_source_with_valid_pyc
- test_syntax_error_in_source_with_valid_pyc - test.test_zipimport.ZStdCompressedZipImportTestCase.test_syntax_error_in_source_with_valid_pyc
- testBoth - test.test_zipimport.DeflateCompressedZipImportTestCase.testBoth
- testBoth - test.test_zipimport.ZStdCompressedZipImportTestCase.testBoth
- test_syntax_error_in_source_with_valid_pyc - test.test_zipimport.DeflateCompressedZipImportTestCase.test_syntax_error_in_source_with_valid_pyc
- test_wmi_query_threads - test.test_wmi.WmiTests.test_wmi_query_threads
Summary of the results of the build (if available):
==
Click to see traceback logs
Traceback (test.test_zipimport.UncompressedZipImportTestCase.testTraceback) ... ok
Traceback (most recent call last):
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_wmi.py", line 89, in test_wmi_query_threads
self.assertRegex(t.result(), "ProcessId=")
~~~~~~~~^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\concurrent\futures\_base.py", line 454, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\concurrent\futures\_base.py", line 396, in __get_result
raise self._exception
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\concurrent\futures\thread.py", line 86, in run
result = ctx.run(self.task)
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\concurrent\futures\thread.py", line 73, in run
return fn(*args, **kwargs)
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_wmi.py", line 17, in wmi_exec_query
return _wmi.exec_query(query)
~~~~~~~~~~~~~~~^^^^^^^
OSError: [WinError -2147024890] The handle is invalid
Traceback (most recent call last):
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 286, in test_syntax_error_in_source_with_valid_pyc
self.assertEqual(spec.origin, expected_path)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[40 chars]d\\build\\test_python_13172æ\\junk95142.zip\\ziptestmodule.py' != 'B:\\[40 chars]d\\build\\test_python_13172æ\\junk95142.zip\\ziptestmodule.pyc'
- B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_13172æ\junk95142.zip\ziptestmodule.py
+ B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_13172æ\junk95142.zip\ziptestmodule.pyc
? +
Traceback (most recent call last):
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 315, in testBoth
self.doTest(pyc_ext, files, TESTMOD)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 160, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 174, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[50 chars]test_python_13172æ\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[50 chars]test_python_13172æ\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_13172æ\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_13172æ\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (most recent call last):
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 877, in testGetCompiledSource
self.doTest(pyc_ext, files, TESTMOD, call=self.assertModuleSource)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 160, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 174, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[50 chars]test_python_13172æ\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[50 chars]test_python_13172æ\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_13172æ\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_13172æ\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (test.test_zipimport.ZStdCompressedZipImportTestCase.testTraceback) ... ok
Traceback (most recent call last):
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 877, in testGetCompiledSource
self.doTest(pyc_ext, files, TESTMOD, call=self.assertModuleSource)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 160, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 174, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[49 chars]\test_python_9868æ\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[49 chars]\test_python_9868æ\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_9868æ\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_9868æ\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (most recent call last):
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 286, in test_syntax_error_in_source_with_valid_pyc
self.assertEqual(spec.origin, expected_path)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[39 chars]ld\\build\\test_python_9868æ\\junk95142.zip\\ziptestmodule.py' != 'B:\\[39 chars]ld\\build\\test_python_9868æ\\junk95142.zip\\ziptestmodule.pyc'
- B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_9868æ\junk95142.zip\ziptestmodule.py
+ B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_9868æ\junk95142.zip\ziptestmodule.pyc
? +
Traceback (most recent call last):
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 315, in testBoth
self.doTest(pyc_ext, files, TESTMOD)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 160, in doTest
self.doTestWithPreBuiltZip(expected_ext, *modules, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\uildarea\3.15.ware-win11.nondebug\build\Lib\test\test_zipimport.py", line 174, in doTestWithPreBuiltZip
self.assertEqual(file, os.path.join(zip_path,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*modules) + expected_ext)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'B:\\[49 chars]\test_python_9868æ\\junk95142.zip\\pre\\fix\\ziptestmodule.py' != 'B:\\[49 chars]\test_python_9868æ\\junk95142.zip\\pre\\fix\\ziptestmodule.pyc'
- B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_9868æ\junk95142.zip\pre\fix\ziptestmodule.py
+ B:\uildarea\3.15.ware-win11.nondebug\build\build\test_python_9868æ\junk95142.zip\pre\fix\ziptestmodule.pyc
? +
Traceback (test.test_zipimport.DeflateCompressedZipImportTestCase.testTraceback) ... ok
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
corona10
Awaiting requested review from corona10
corona10 is a code owner
Uh oh!
There was an error while loading. Please reload this page.
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|