GeezerVison is showing github.com — On Windows, shutil.which incorrectly resolves to non-executable extensionless files (python >= 3.12.0b1) · Issue #109590 · python/cpython
On Windows, shutil.which incorrectly resolves to non-executable extensionless files (python >= 3.12.0b1) #109590DescriptionBug report
Bug description:
This change (935aa45, PR #103179) changed the behavior of shutil.which so that it checks for the extensionless filename first, before checking extensions from PATHEXT. Previously, the extensionless name was not checked at all.
This is causing problems because, at least in my case, there is both an extensionless and extensioned version of the program, and the extensionless version is a bash script (which apparently can't be executed directly on Windows.)
Full Description
On a GitHub Windows shared runner, node.js is installed.
Run Get-ChildItem -Path "C:\Program Files\nodejs"
Directory: C:\Program Files\nodejs
Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 9/10/2023 11:53 PM node_modules
[... elided ...]
-a--- 8/8/2023 11:20 PM 71207576 node.exe
-a--- 6/23/2023 6:02 AM 702 nodevars.bat
-a--- 8/8/2023 10:22 PM 1365 npm
-a--- 6/23/2023 6:01 AM 483 npm.cmd
-a--- 8/8/2023 10:22 PM 1567 npx
-a--- 6/23/2023 6:01 AM 539 npx.cmd
Here, npm is a bash script, while npm.cmd is a cmd.exe script.
In python 3.11 (and before), shutil.which("npm") returns "C:\\Program Files\\nodejs\\npm.CMD". That can be executed successfully by subprocess.run.
In python 3.12rc3, shutil.which("npm") returns "C:\\Program Files\\nodejs\\npm", which, being a bash script, appears not to be by executable by subprocess.
subprocess.run((shutil.which("npm"), "install"))fails with OSError: [WinError 193] %1 is not a valid Win32 application on python 3.12rc3.
Notes
On the system in question Get-Command npm returns C:\Program Files\nodejs\npm.cmd. So PowerShell gets it right.
(PATHEXT is .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL.)
I'm not absolutely convinced that this is a bug. But again, I'm also not sure how to work around it without essentially reimplementing shutil.which.
CPython versions tested on:
3.12
Operating systems tested on:
Windows
Linked PRs
Activitydairiki commented on Sep 20, 2023 More research as to the history of the change:
The original name, without added extension from PATHEXT, was added to the search list in #103179 to address #75586:
On windows, shutil.which does not match the semantics of built-in command lookup. If you pass the name of a script like foo.py and the PATHEXT variable doesn't include .py it will search for foo.py.exe, foo.py.bat, foo.py.cmd, etc. but not foo.py, which should be the first name checked.
But, later in #75586, there is this:
Note that CMD will only try to execute an extensionless file if "." is in PATHEXT. (In the Windows file namespace, "name" and "name." are equivalent.) [...]
Assuming that is true, then the original name should only be prepended to the search list if it has an extension. Extensionless names should not be considered when searching for an executable. changed the title [-]Changes in the handling of PATHEXT by shutil.which are causing issues on Windows[/-] [+]On Windows, shutil.which incorrectly resolves to extensionless files (python > 3.12.0b1)[/+] on Sep 20, 2023changed the title [-]On Windows, shutil.which incorrectly resolves to extensionless files (python > 3.12.0b1)[/-] [+]On Windows, shutil.which incorrectly resolves to extensionless files (python >= 3.12.0b1)[/+] on Sep 20, 2023changed the title [-]On Windows, shutil.which incorrectly resolves to extensionless files (python >= 3.12.0b1)[/-] [+]On Windows, shutil.which incorrectly resolves to non-executable extensionless files (python >= 3.12.0b1)[/+] on Sep 21, 2023csm10495 commented on Sep 24, 2023 @eryksun can you give some thoughts here?
I'm a bit nervous about this type of case hitting others in 3.12 and wondering your thoughts.
We could add logic to shutil.which to do:
if there is a directory (/ or \\) in the cmd OR there is a dot in it (denoting an extension), check the given cmd first.
OR
IF '.' is in PATHEXT, also check cmd first if it doesn't have a dot.
I think this gets it in-line with what CMD would do, but I'm not 100% on it. csm10495 commented on Sep 24, 2023 Some testing with CMD:
C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\<TAB>
_testembed.exe python.exe pyw.exe venvwlauncher.exe
py.exe pythonw.exe venvlauncher.exe
C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\python
Python 3.13.0a0 (heads/customization_modules_test:cbad68d4d2, Sep 23 2023, 21:14:32) [MSC v.1936 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
C:\Users\csm10495\Desktop\cpython>echo "TEST" > PCbuild\amd64\python
C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\python
Python 3.13.0a0 (heads/customization_modules_test:cbad68d4d2, Sep 23 2023, 21:14:32) [MSC v.1936 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\python.
'PCbuild\amd64\python.' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\csm10495\Desktop\cpython>type PCbuild\amd64\python
"TEST"
C:\Users\csm10495\Desktop\cpython>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
So i guess if you give a directory component, it still consults PATHEXT first. csm10495 commented on Sep 24, 2023 I implemented the '.' in pathext to check in this draft pr: #109797
.. but i couldn't replicate the described behavior in CMD in Win 10.0.19045.3448:
I cleared C:\Program Files\nodejs\npm to be an empty file. If it was called it shouldn't really do anything.. but it kept hitting npm.cmd instead.
C:\Windows\System32>where npm
C:\Program Files\nodejs\npm
C:\Program Files\nodejs\npm.cmd
C:\Windows\System32>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
C:\Windows\System32>type "C:\Program Files\nodejs\npm"
C:\Windows\System32>npm --version
8.5.2
C:\Windows\System32>set PATHEXT=%PATHEXT%;.;
C:\Windows\System32>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW;.;
C:\Windows\System32>where npm
C:\Program Files\nodejs\npm
C:\Program Files\nodejs\npm.cmd
C:\Windows\System32>type "C:\Program Files\nodejs\npm"
C:\Windows\System32>npm --version
8.5.2
C:\Windows\System32>set PATHEXT=.;%PATHEXT%;.;
C:\Windows\System32>npm --version
8.5.2
C:\Windows\System32>echo %PATHEXT%
.;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW;.;;.;
I'll wait for @eryksun 's thoughts. csm10495 commented on Sep 24, 2023 One last thing before bed :)
Maybe it does work but only if the rest of PATHEXT can't resolve something first (as in: it ignores the order in PATHEXT. Even if the dot is first, it will do all the other extensions before checking for an extensionless file).
I renamed npm.cmd to .npm.cmd, then:
C:\Windows\System32>npm
'npm' is not recognized as an internal or external command,
operable program or batch file.
C:\Windows\System32>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
C:\Windows\System32>set PATHEXT=%PATHEXT%;.
C:\Windows\System32>npm
<opened the npm file which led to the 'how should windows open this dialog'>
C:\Windows\System32>dir "C:\Program Files\nodejs"
Volume in drive C has no label.
Volume Serial Number is CAC3-BECF
Directory of C:\Program Files\nodejs
09/23/2023 11:16 PM <DIR> .
09/23/2023 11:16 PM <DIR> ..
02/02/2022 06:00 AM 483 .npm.cmd
01/10/2022 04:22 PM 335 corepack
01/10/2022 04:22 PM 218 corepack.cmd
02/02/2022 06:00 AM 3,032 install_tools.bat
03/10/2022 06:31 PM 62,530,184 node.exe
10/14/2021 12:30 AM 702 nodevars.bat
03/15/2022 04:07 PM <DIR> node_modules
09/23/2023 11:09 PM 11 npm
02/02/2022 06:00 AM 1,552 npx
02/02/2022 06:00 AM 539 npx.cmd
9 File(s) 62,537,056 bytes
3 Dir(s) 390,529,925,120 bytes free
C:\Windows\System32>
If that's the desired behavior, it wouldn't be too hard to update the PR. csm10495 commented on Sep 26, 2023 zooba commented on Sep 26, 2023
Assuming that is true, then the original name should only be prepended to the search list if it has an extension. Extensionless names should not be considered when searching for an executable.
This sounds reasonable at face value.
CMD is not the hallmark of consistent behaviour. I wouldn't want to set it as our standard. If you want CMD behaviour, pass shell=True (no, don't really).
The closer we get to the underlying API behaviour, the happier I'll be - in this case, it's SearchPathW. csm10495 commented on Sep 27, 2023 So unfortunately (or fortunately), SearchPathW doesn't match CMD behavior here and actually matches Python 3.12's current behavior:
Via pywin32:
In [1]: from win32api import *
In [2]: SearchPath(None, 'npm')
Out[2]: ('C:\\Program Files\\nodejs\\npm', 24)
In [3]: import os
In [4]: os.environ.get('pathext')
Out[4]: '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW'
So it seems fine giving back a the file without an extension. So I guess we match right now.
I can't seem to find any definitive documentation on the expectation of PATHEXT and extensionless. So I guess at the moment if we're matching SearchPath.. we match. zooba commented on Sep 28, 2023 Having seen the bugs that are getting in vs. being deferred, I don't think we want this one in. The risk far outweighs the value right now.
If we'd found it before RC1, then definitely. Right now, the important thing to do is to make sure that anyone who fixes this in 3.12.0 is still going to work in 3.12.1 (which is why we check if there's already a PATHEXT extension on the requested name). added a commit that references this issue on Oct 2, 2023gh-109590: Update shutil.which on Windows to prefer a PATHEXT extensi… hugovk commented on Nov 9, 2023 Closing as the PRs have been merged. Thanks!
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|