Description

pgy

opened on Jul 3, 2022

Issue body actions

Bug report

I am building python in a directory that has non-ascii name on linux and get the following error:

./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Exception ignored error evaluating path:
Traceback (most recent call last):
  File "<frozen getpath>", line 349, in <module>
ModuleNotFoundError: No module named 'encodings'
Fatal Python error: error evaluating path
Python runtime state: core initialized

Current thread 0x00007f16d6c10740 (most recent call first):
  <no Python frame>
make: *** [Makefile:1218: Python/frozen_modules/abc.h] Error 1

The problem seems to be that the limited environment used to execute the precompiled Modules/getpath.py does not have the encodings module.

I tracked the exception to a PyImport_ImportModule("encodings"); call in Python/codecs.c which is a consequence of a PyCodec_LookupError that happens in ucs2lib_utf8_encoder which I think gets called in getpath_dirname.

The original string is /home/pgy/letöltések/cpython/_bootstrap_python, and the unicode object ucs2lib_utf8_encoder gets as an argument is:

(rr) p PyObject_Print(unicode, stderr, 0)
'/home/pgy/let\udcc3\udcb6lt\udcc3\udca9sek/cpython/_bootstrap_python'

Your environment

• CPython versions tested on: 7db1d2e
• Operating system and architecture: Linux hostname 5.15.44-1-lts #1 SMP Mon, 30 May 2022 13:45:47 +0000 x86_64 GNU/Linux

Activity

1. pgy

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jul 3, 2022
2. kumaraditya303

   self-assigned this

   on Sep 17, 2022
3. kumaraditya303

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   3.11only security fixesonly security fixes

   3.12only security fixesonly security fixes

   on Sep 17, 2022
4. vstinner commented on Sep 29, 2022

   

   vstinner

   on Sep 29, 2022

   Member

   More actions

   I created PR #97645 to fix the root issue.

   I can reproduce the issue:

   • Clone Python Git repository in a non-ASCII directory: /home/vstinner/python/mainé (é is non-ASCII)
   • Build Python: ./configure --with-pydebug && make
   • make fails with: ModuleNotFoundError: No module named 'encodings'

   My locale encoding is UTF-8, my LC_CTYPE locale is fr_FR.UTF-8.

   The problem is that the getpath_dirname() function of Modules/getpath.c encodes the Unicode path to UTF-8/strict using s format and PyArg_ParseTuple(), whereas a path can contain surrogate characters.

   The getpath_dirname() and getpath_basename() convert their Unicode input string to bytes just to be able to use strrchr(path, SEP) since SEP is a bytes string. There is no need to convert these strings to bytes, whereas the conversion is the root issue here.
5. vstinner

   added a commit that references this issue on Sep 30, 2022

   gh-94526: getpath_dirname() no longer encodes the path (#97645)

   9f2f1dd
6. miss-islington

   added a commit that references this issue on Sep 30, 2022

   pythongh-94526: getpath_dirname() no longer encodes the path (pythonG…

   0aa7e74
7. kumaraditya303

   removed their assignment

   on Sep 30, 2022
8. vstinner commented on Sep 30, 2022

   

   vstinner

   on Sep 30, 2022

   Member

   More actions

   Fixed by PR #97645. Backport to 3.11: PR #97677.
9. vstinner

   closed this as completedon Sep 30, 2022
10. miss-islington

    added a commit that references this issue on Sep 30, 2022

    gh-94526: getpath_dirname() no longer encodes the path (GH-97645)

    6537bc9
11. serhiy-storchaka

    added a commit that references this issue on Oct 2, 2022

    pythongh-94526: getpath_dirname() no longer encodes the path (python#…

    cc1e8e0

Issue actions