Description

Aasyaco

opened on Mar 20, 2026

Issue body actions

Windows CI failure: test_itertools corrupts Unicode temp directory name under mbcs encoding, breaks subsequent tests

Summary

On Windows (official Azure CI), the CPython 3.15 test suite produces cascading failures originating from test_itertools. The test creates a temporary directory whose name contains a non-ASCII character. Because the Windows default filesystem encoding is mbcs, the Unicode byte sequence is misinterpreted and the directory name becomes corrupted on disk. The directory is never cleaned up, which causes all subsequent tests that create temporary directories to fail with FileNotFoundError.

---

CPython Version

Python 3.15.0a7 (main branch)

Operating System / Platform

Windows (Azure Pipelines Microsoft-hosted runner)

Filesystem Encoding

>>> import sys
>>> sys.getfilesystemencoding()
'mbcs'

---

CI Pipeline Context

Build step:

PCbuild\build.bat -e $(buildOpt)

Test step:

PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 \
  --junit-xml="$(Build.BinariesDirectory)\test-results.xml" \
  --tempdir="$(Build.BinariesDirectory)\test"

Azure DevOps build log:
https://dev.azure.com/Python/cpython/_build/results?buildId=168864&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765&t=ddcdae4e-111a-5c2a-2289-6b784c553924

---

Steps to Reproduce

1. Check out the CPython main branch.
2. Build on Windows:

   PCbuild\build.bat -e <buildOpt>
   

3. Run the test suite:

   PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 ^
       -j0 --tempdir="%TEMP%"
   

4. Observe a directory with a corrupted name (e.g. packageæ/) left behind in the temp directory.
5. Observe that subsequent tests fail with FileNotFoundError.

---

Expected Behavior

• All tests pass cleanly on Windows during CI builds.
• No test leaves behind artifacts that interfere with other tests.
• Test isolation is preserved — temporary directories are created and removed correctly.
• Unicode filenames are handled consistently regardless of platform filesystem encoding.

---

Actual Behavior

• test_itertools creates a directory whose name contains a non-ASCII character (e.g. æ).
• Under the Windows default codepage (mbcs), the Unicode name is misinterpreted: æ (U+00E6) is stored as æ.
• The corrupted directory is not removed after test_itertools completes.
• Subsequent tests (e.g. test_class, test_pdb) fail when attempting to create temporary directories because the corrupted entry causes os_helper.temp_dir to raise FileNotFoundError.
• The runner reports a warning for test_itertools and marks unrelated tests as failed, significantly increasing triage cost.

---

Relevant Log Excerpt

From Azure build 168864:

Warning -- files was modified by test_itertools
After: ... 'packageæ/'

test_class failed (env changed)
FileNotFoundError: [WinError 3] unable to create temporary directory:
'D:\a\1\s\build\test_python_9140æ'

---

Root Cause (Analysis)

Two independent problems combine to produce this failure:

1. Encoding mismatch in test_itertools

The test constructs or receives a directory name containing a non-ASCII character. On Windows, sys.getfilesystemencoding() returns 'mbcs' rather than 'utf-8'. The UTF-8 byte sequence for æ (0xC3 0xA6) is reinterpreted under the ANSI code page as two separate characters, producing the visible corruption æ. This is a violation of cross-platform Unicode path handling requirements.

2. Missing cleanup in test_itertools

The test creates the directory but does not register it for cleanup (e.g. via addCleanup, self.addCleanup, or a context manager such as tempfile.TemporaryDirectory). The corrupted directory persists in the shared --tempdir location for the duration of the entire test run, causing every subsequent call to os_helper.temp_dir to fail with WinError 3.

---

Impact

Area	Description	Severity
CI pipeline stability	All Windows builds fail after test_itertools runs	High
Test isolation	Subsequent tests inherit corrupted filesystem state	High
Unicode correctness	Non-ASCII filenames silently corrupted on Windows	Medium
Triage cost	Unrelated failures obscure the root cause	Medium
Local Windows dev	Developers running the full suite locally may be affected	Low

---

Files Involved

• Lib/test/test_itertools.py — primary location of the encoding issue and missing cleanup
• Lib/test/support/os_helper.py — secondary impact point (temp_dir context manager)

---

Linked Resources

• Azure DevOps build 168864: https://dev.azure.com/Python/cpython/_build/results?buildId=168864
• PEP 529 — Change Windows filesystem encoding to UTF-8: https://peps.python.org/pep-0529/
• sys.getfilesystemencoding() docs: https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding
• os.fsencode() docs: https://docs.python.org/3/library/os.html#os.fsencode

Linked PRs

• gh-146202: Create tmp_dir in regrtest worker #146347
• [3.14] gh-146202: Create tmp_dir in regrtest worker (GH-146347) #146349
• [3.13] gh-146202: Create tmp_dir in regrtest worker (GH-146347) #146350

Activity

1. aisk

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   testsTests in the Lib/test dirTests in the Lib/test dir

   OS-windows

   on Mar 20, 2026
2. vstinner commented on Mar 20, 2026

   

   vstinner

   on Mar 20, 2026

   Member

   More actions

   Azure DevOps build log:
   https://dev.azure.com/Python/cpython/_build/results?buildId=168864&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765&t=ddcdae4e-111a-5c2a-2289-6b784c553924

   Extract of logs:

   PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 --junit-xml="D:\a\1\b\test-results.xml" --tempdir="D:\a\1\b\test"
   
   D:\a\1\s>"D:\a\1\s\PCbuild\amd64\python.exe"  -m test --fast-ci -uall -u-cpu -rwW --slowest --timeout 1200 -j0 --junit-xml D:\a\1\b\test-results.xml --tempdir D:\a\1\b\test 
   + 'D:\a\1\s\PCbuild\amd64\python.exe' -u -W error -bb -E -m test --fast-ci -uall -u-cpu -rwW --slowest --timeout 1200 -j0 --junit-xml 'D:\a\1\b\test-results.xml' --tempdir 'D:\a\1\b\test' --dont-add-python-opts
   == CPython 3.15.0a7+ (remotes/origin/52c01864c4778a351e5aa3584e86ed6fd212a5a4:52c01864c, Mar 19 2026, ) [MSC v.1944 64 bit (AMD64)]
   == Windows-2022Server-10.0.20348-SP0 little-endian
   == Python build: release with_assert
   == cwd: D:\a\1\b\test\test_python_worker_2896æ
   == CPU count: 2
   == encodings: locale=cp1252 FS=utf-8
   == resources: all,-cpu
   
   Using random seed: 1027629067
   0:00:00 Run 501 tests in parallel using 4 worker processes (timeout: 20 min, worker timeout: 25 min)
   0:00:00 [  1/501] test.test_io.test_file passed
   0:00:00 [  2/501] test_select passed
   0:00:01 [  3/501] test_type_comments passed
   0:00:02 [  4/501/1] test_range failed (env changed)
   tests may fail, unable to create temporary directory 'D:\\a\\1\\s\\build\\test_python_4940æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
   Traceback (most recent call last):
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 524, in temp_dir
       os.mkdir(path)
       ~~~~~~~~^^^^^^
   FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
   Stack (most recent call last):
     File "<frozen runpy>", line 196, in _run_module_as_main
     File "<frozen runpy>", line 87, in _run_code
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
       main()
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
       with os_helper.temp_cwd(work_dir, quiet=True):
     File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
       return next(self.gen)
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 597, in temp_cwd
       with temp_dir(path=name, quiet=quiet) as temp_path:
   tests may fail, unable to change the current working directory to 'D:\\a\\1\\s\\build\\test_python_4940æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
   Traceback (most recent call last):
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 563, in change_cwd
       os.chdir(os.path.realpath(path))
       ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
   FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
   Stack (most recent call last):
     File "<frozen runpy>", line 196, in _run_module_as_main
     File "<frozen runpy>", line 87, in _run_code
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
       main()
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
       with os_helper.temp_cwd(work_dir, quiet=True):
     File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
       return next(self.gen)
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 598, in temp_cwd
       with change_cwd(temp_path, quiet=quiet) as cwd_dir:
   test_attributes (test.test_range.RangeTest.test_attributes) ... ok
   test_comparison (test.test_range.RangeTest.test_comparison) ... ok
   test_contains (test.test_range.RangeTest.test_contains) ... ok
   test_count (test.test_range.RangeTest.test_count) ... ok
   test_empty (test.test_range.RangeTest.test_empty) ... ok
   test_exhausted_iterator_pickling (test.test_range.RangeTest.test_exhausted_iterator_pickling) ... ok
   test_index (test.test_range.RangeTest.test_index) ... ok
   test_invalid_invocation (test.test_range.RangeTest.test_invalid_invocation) ... ok
   test_issue11845 (test.test_range.RangeTest.test_issue11845) ... ok
   test_iterator_invalid_setstate (test.test_range.RangeTest.test_iterator_invalid_setstate) ... ok
   test_iterator_pickling (test.test_range.RangeTest.test_iterator_pickling) ... ok
   test_iterator_pickling_overflowing_index (test.test_range.RangeTest.test_iterator_pickling_overflowing_index) ... ok
   test_iterator_setstate (test.test_range.RangeTest.test_iterator_setstate) ... ok
   test_iterator_unpickle_compat (test.test_range.RangeTest.test_iterator_unpickle_compat) ... ok
   test_large_exhausted_iterator_pickling (test.test_range.RangeTest.test_large_exhausted_iterator_pickling) ... ok
   test_large_operands (test.test_range.RangeTest.test_large_operands) ... ok
   test_large_range (test.test_range.RangeTest.test_large_range) ... ok
   test_odd_bug (test.test_range.RangeTest.test_odd_bug) ... ok
   test_pickling (test.test_range.RangeTest.test_pickling) ... ok
   test_range (test.test_range.RangeTest.test_range) ... ok
   test_range_constructor_error_messages (test.test_range.RangeTest.test_range_constructor_error_messages) ... ok
   test_range_iterators (test.test_range.RangeTest.test_range_iterators) ... ok
   test_range_iterators_invocation (test.test_range.RangeTest.test_range_iterators_invocation) ... ok
   test_repr (test.test_range.RangeTest.test_repr) ... ok
   test_reverse_iteration (test.test_range.RangeTest.test_reverse_iteration) ... ok
   test_slice (test.test_range.RangeTest.test_slice) ... ok
   test_strided_limits (test.test_range.RangeTest.test_strided_limits) ... ok
   test_types (test.test_range.RangeTest.test_types) ... ok
   test_user_index_method (test.test_range.RangeTest.test_user_index_method) ... ok
   
   ----------------------------------------------------------------------
   Ran 29 tests in 1.076s
   
   OK
   Warning -- files was modified by test_range
   Warning --   Before: ['.azure-pipelines/', '.coveragerc', '.devcontainer/', '.editorconfig', '.git/', '.gitattributes', '.github/', '.gitignore', '.mailmap', '.pre-commit-config.yaml', '.readthedocs.yml', '.ruff.toml', '.well-known/', 'Android/', 'Apple/', 'Doc/', 'Grammar/', 'Include/', 'InternalDocs/', 'LICENSE', 'Lib/', 'Mac/', 'Makefile.pre.in', 'Misc/', 'Modules/', 'Objects/', 'PC/', 'PCbuild/', 'Parser/', 'Platforms/', 'Programs/', 'Python/', 'README.rst', 'Tools/', 'aclocal.m4', 'config.guess', 'config.sub', 'configure', 'configure.ac', 'iOS/', 'install-sh', 'pyconfig.h.in', 'python.bat']
   Warning --   After:  ['.azure-pipelines/', '.coveragerc', '.devcontainer/', '.editorconfig', '.git/', '.gitattributes', '.github/', '.gitignore', '.mailmap', '.pre-commit-config.yaml', '.readthedocs.yml', '.ruff.toml', '.well-known/', 'Android/', 'Apple/', 'Doc/', 'Grammar/', 'Include/', 'InternalDocs/', 'LICENSE', 'Lib/', 'Mac/', 'Makefile.pre.in', 'Misc/', 'Modules/', 'Objects/', 'PC/', 'PCbuild/', 'Parser/', 'Platforms/', 'Programs/', 'Python/', 'README.rst', 'Tools/', 'aclocal.m4', 'config.guess', 'config.sub', 'configure', 'configure.ac', 'iOS/', 'install-sh', 'junk95142.zip', 'pyconfig.h.in', 'python.bat', 'ziptestmodule']
   
3. vstinner commented on Mar 20, 2026

   

   vstinner

   on Mar 20, 2026

   Member

   More actions

   On Windows, sys.getfilesystemencoding() returns 'mbcs' rather than 'utf-8'.

   That's wrong: sys.getfilesystemencoding() returns utf-8 on Windows. You can see FS=utf-8 in logs above (line == encodings: locale=cp1252 FS=utf-8).

   You can also see sys.filesystem_encoding: utf-8/surrogatepass in the previous build step, in "Display build info".

   Steps to Reproduce:
   (...)
   PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 --tempdir="%TEMP%"

   I built Python in debug mode and I ran:

   PCbuild\rt.bat -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j1 test_range --tempdir=%TEMP%

   I'm unable to reproduce the error.
4. Aasyaco commented on Mar 20, 2026

   

   Aasyaco

   on Mar 20, 2026

   Author

   More actions

   I am sorry. The real error message is

   tests may fail, unable to create temporary directory 'D:\\a\\1\\s\\build\\test_python_9140æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
   Traceback (most recent call last):
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 524, in temp_dir
       os.mkdir(path)
       ~~~~~~~~^^^^^^
   FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
   Stack (most recent call last):
     File "<frozen runpy>", line 196, in _run_module_as_main
     File "<frozen runpy>", line 87, in _run_code
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
       main()
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
       with os_helper.temp_cwd(work_dir, quiet=True):
     File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
       return next(self.gen)
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 597, in temp_cwd
       with temp_dir(path=name, quiet=quiet) as temp_path:
   tests may fail, unable to change the current working directory to 'D:\\a\\1\\s\\build\\test_python_9140æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
   Traceback (most recent call last):
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 563, in change_cwd
       os.chdir(os.path.realpath(path))
       ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
   FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
   Stack (most recent call last):
     File "<frozen runpy>", line 196, in _run_module_as_main
     File "<frozen runpy>", line 87, in _run_code
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
       main()
     File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
       with os_helper.temp_cwd(work_dir, quiet=True):
     File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
       return next(self.gen)
     File "D:\a\1\s\Lib\test\support\os_helper.py", line 598, in temp_cwd
       with change_cwd(temp_path, quiet=quiet) as cwd_dir:
   

   This is only a example.
5. serhiy-storchaka commented on Mar 20, 2026

   

   serhiy-storchaka

   on Mar 20, 2026

   Member

   More actions

   It seems that some of functions still use non-Unicode Windows API.

   We use non-ASCII temporary directory name specially to catch such bugs.
6. vstinner commented on Mar 20, 2026

   

   vstinner

   on Mar 20, 2026

   Member

   More actions

   It seems that some of functions still use non-Unicode Windows API.

   At least, os.chdir() is implemented with SetCurrentDirectoryW() which takes a wide string (LPCWSTR path).

   This is only a example.

   @Aasyaco: Can you reproduce the error outside the Azure Pipeline?
7. Aasyaco commented on Mar 21, 2026

   

   Aasyaco

   on Mar 21, 2026

   Author

   More actions

   @vstinner Thank you for the patience and correction.

   The traceback shows the corrupted path test_python_9140æ is
   already invalid before reaching os_helper.temp_cwd() — the
   corruption happens upstream in libregrtest/worker.py line 133,
   where work_dir is constructed. The os_helper code is just the crash site, not the source.

   I also want to correct my earlier analysis: the root cause is not
   test_itertools or mbcs filesystem encoding. I apologise for the inaccurate attribution.
8. vstinner

   added a commit that references this issue on Mar 23, 2026

   pythongh-146202: Create tmp_dir in regrtest worker

   d0830f8
9. vstinner commented on Mar 23, 2026

   

   vstinner

   on Mar 23, 2026

   Member

   More actions

   tests may fail, unable to create temporary directory 'D:\a\1\s\build\test_python_4940æ': [WinError 3] The system cannot find the path specified: 'D:\a\1\s\build\test_python_4940æ'

   This error means that the D:\a\1\s\build\ directory (the build directory) doesn't exist. When the --tempdir option is used, the main process doesn't create the "build" directory. It seems like sometimes, the build directory is created. Sometimes not, causing tons errors related to temp_cwd() and change_cwd().

   I wrote #146347 to fix regrtest.

   See also this old issue about the same bug in 2023: #110644.

   Note: This error is unrelated to the filesystem encoding.
10. vstinner

    changed the title [-]test_itertools: non-ASCII temp dir name corrupted under Windows mbcs codepage and not removed, causing FileNotFoundError in subsequent tests[/-] [+]Tests: error "tests may fail, unable to create temporary directory (...): [WinError 3] The system cannot find the path specified" on Windows[/+] on Mar 23, 2026
11. vstinner commented on Mar 23, 2026

    

    vstinner

    on Mar 23, 2026

    Member

    More actions

    Most Azure Pipeline jobs are green because tests are re-run, and they pass when they are re-run.

    But many tests are re-run: https://dev.azure.com/Python/cpython/_build/results?buildId=168944&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765&t=ddcdae4e-111a-5c2a-2289-6b784c553924

    43 re-run tests:
        test.test_asyncio.test_base_events test.test_asyncio.test_locks
        test.test_asyncio.test_server test.test_asyncio.test_staggered
        test.test_concurrent_futures.test_thread_pool
        test.test_multiprocessing_spawn.test_processes
        test.test_os.test_os test.test_pydoc.test_pydoc test___all__
        test__interpreters test_argparse test_base64 test_binop
        test_codeccallbacks test_compileall test_ctypes test_dbm_dumb
        test_eof test_finalization test_graphlib test_imaplib
        test_itertools test_keywordonlyarg test_lazy_import test_list
        test_mmap test_module test_ntpath test_pathlib test_profiling
        test_pyexpat test_regrtest test_runpy test_sched test_ssl test_sys
        test_tabnanny test_traceback test_ucn test_venv test_wave
        test_zipfile test_zipimport
    
    455 tests OK.
    
12. vstinner

    added a commit that references this issue on Mar 23, 2026

    gh-146202: Create tmp_dir in regrtest worker (#146347)

    bcff99c
13. miss-islington

    added 2 commits that reference this issue on Mar 23, 2026

    pythongh-146202: Create tmp_dir in regrtest worker (pythonGH-146347)

    f14bf53

    pythongh-146202: Create tmp_dir in regrtest worker (pythonGH-146347)

    7f8d07e
14. vstinner

    added 2 commits that reference this issue on Mar 24, 2026

    [3.13] gh-146202: Create tmp_dir in regrtest worker (GH-146347) (#146350

    0d82ce8

    [3.14] gh-146202: Create tmp_dir in regrtest worker (GH-146347) (#146349

    ad273f7
15. vstinner commented on Mar 24, 2026

    

    vstinner

    on Mar 24, 2026

    Member

    More actions

    The issue is fixed in the latest Azure Pipeline job: https://dev.azure.com/Python/cpython/_build/results?buildId=168959&view=results

    455 tests OK.

    There are no longer re-run tests.
16. vstinner

    closed this as completedon Mar 24, 2026
17. CuriousLearner

    added a commit that references this issue on Mar 24, 2026

    Merge branch 'fix-pythongh-138577' of github.com:CuriousLearner/cpyth…

    78b2c6a
18. Aasyaco commented on Mar 24, 2026

    

    Aasyaco

    on Mar 24, 2026

    Author

    More actions

    @vstinner thank you so much.

    Reacted by Victor Stinner
19. clin1234

    added a commit that references this issue on Apr 16, 2026

    pythongh-146202: Create tmp_dir in regrtest worker (python#146347)

    87af7bc
20. ljfp

    added a commit that references this issue on Apr 25, 2026

    pythongh-146202: Create tmp_dir in regrtest worker (python#146347)

    62b1563

Issue actions