GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-31961: Fix support of path-like executables in subprocess. by serhiy-storchaka · Pull Request #5914 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Feb 26, 2018 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

https://bugs.python.org/issue31961

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
bpo-31961: Fix support of path-like executables in subprocess.
9e87239
@serhiy-storchaka serhiy-storchaka added the needs backport to 3.7 label Feb 26, 2018
@serhiy-storchaka
serhiy-storchaka requested a review from gpshead as a code owner February 26, 2018 22:21
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Feb 26, 2018
@bedevere-bot bedevere-bot added the awaiting merge label Feb 26, 2018
@serhiy-storchaka serhiy-storchaka added the skip news label Feb 26, 2018
@serhiy-storchaka
More changes:
8cce344
* Simplify the code on Posix
* Fix the code on Windows.
* Use absolute paths in tests.
@serhiy-storchaka serhiy-storchaka mentioned this pull request Feb 27, 2018
Revert "bpo-31961: subprocess now accepts path-like args (GH-4329)" #5912
Merged
eryksun
eryksun reviewed Feb 27, 2018
View reviewed changes
Comment thread Lib/subprocess.py
else:
args = list(args)
args[0] = os.fsdecode(args[0])
args = list2cmdline(args)

@eryksun eryksun Feb 27, 2018

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be addressed in list2cmdline to be consistent with how POSIX converts every element of args via PyUnicode_FSConverter. For example:

if isinstance(args, bytes):
    args = os.fsdecode(args)
elif isinstance(args, os.PathLike):
    if shell:
        raise ValueError('PathLike args is not allowed when '
                         'shell is true.')
    else:
        args = [args]

if not isinstance(args, str):
    args = list2cmdline(args)

if executable is not None:
    executable = os.fsdecode(executable)

Then in list2cmdline, decode the items via os.fsdecode :

for arg in seq:
    if not isinstance(arg, str):
        arg = os.fsdecode(arg)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@gpshead gpshead May 18, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does look like it would be nicer, allowing more mixing and matching of types.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka May 25, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eryksun 's comment is related to the old version of the PR. I addressed it, except that that I decided to reject bytes args on Windows with shell=True as it is ambiguous and is not actually needed. It is enough to have a difference in interpreting str args.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
serhiy-storchaka added 3 commits February 28, 2018 16:05
@serhiy-storchaka
Revert "Revert "bpo-31961: subprocess now accepts path-like args (pyt…
0b4e876
…honGH-4329)" (python#5912)"

This reverts commit be50a7b.
@serhiy-storchaka
Merge branch 'revert-revert-31961' into subprocess-pathlike
e72cf67
@serhiy-storchaka
Restore NEWS.
ccd3464
eryksun
eryksun reviewed Feb 28, 2018
View reviewed changes
Comment thread Lib/subprocess.py

if executable is not None:
executable = os.fsdecode(executable)

@eryksun eryksun Feb 28, 2018

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think bytes and pathlike should be supported for every item of args. The POSIX implementation doesn't limit this to args[0] (though not documented), and I think that's proper. A script shouldn't have to special-case calling os.fsdecode for a pathlike argument. That's inefficient for POSIX, for which os.fsencode would be better. While os.fspath would be efficient, it's insufficient because it might return bytes on Windows.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Mar 27, 2018

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The POSIX implementation accepts also bytes as args and as any item of args. If allow path-like arguments, we should consider also adding support of bytes arguments and program name.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
serhiy-storchaka added 4 commits March 2, 2018 11:56
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
291b0c6
@serhiy-storchaka
Minimize docs changes.
cee318b
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
43eef6c
@serhiy-storchaka
Retarget to 3.8 and add a news entry.
5c0dd88
@serhiy-storchaka serhiy-storchaka added type-feature A feature request or enhancement and removed needs backport to 3.7 labels Mar 27, 2018
serhiy-storchaka added 5 commits July 8, 2018 12:59
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
ab8ebfe
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
80f0af3
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
9a0e746
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
148b13d
@serhiy-storchaka
Add support for bytes and path-like args.
6a8d394
@serhiy-storchaka

serhiy-storchaka commented Nov 9, 2018

Copy link
Copy Markdown
Member Author

You convinced me. I made bytes and pathlike be supported for every item of args. But bytes as a single argument is not supported because this case is ambiguous. It is not clear how to interpret it: as a full command line (as single str value) or as a program file name (as single path-like value).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka serhiy-storchaka mentioned this pull request Nov 9, 2018
bpo-34699: allow path-like objects in program arguments in Windows #9336
Closed
serhiy-storchaka added 2 commits November 18, 2018 19:38
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
10fba36
@serhiy-storchaka
Improve error messages for invalid args.
7d51e79
@serhiy-storchaka

serhiy-storchaka commented Jan 12, 2019

Copy link
Copy Markdown
Member Author

@gpshead, could you please take a look?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@csabella
csabella removed the request for review from gpshead May 18, 2019 00:09
@csabella
csabella requested a review from gpshead May 18, 2019 00:09
@gpshead
Fix the NEWS wording to be correct English.
1073a0f
gpshead
gpshead reviewed May 18, 2019
View reviewed changes
Comment thread Lib/subprocess.py
if not isinstance(args, str):
if isinstance(args, str):
pass
elif isinstance(args, bytes):

@gpshead gpshead May 18, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The elif chain could be simplified to have less repeated code:

elif isinstance(args, (bytes, os.PathLike)):
    if shell:
        raise TypeError(f'{type(args)} args not allowed when shell=True on Windows')`
    args = list2cmdline([args])

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

tirkarthi reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@serhiy-storchaka serhiy-storchaka May 25, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error messages should be different. Path-like args is not allowed on all platforms, bytes args is not allowed only on Windows.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZackerySpytz reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Lib/subprocess.py
args = [args]
elif isinstance(args, os.PathLike):
if shell:
raise TypeError('path-like args is not allowed when '

@gpshead gpshead May 18, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for consistency of error messages with my suggestion above I think TypeError(f'{type(args)} args is not allowed when ' ...). It is good for that to render as <class os.PathLike> as that is less confusing to the error message reader than "path-like" which they might be more likely to mis-interpret in English as suggesting anything that looks like a path within a string.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_subprocess.py
self.assertEqual(cp.returncode, 33)

def test_run_with_pathlike_path(self):
# bpo-31961: test run(pathlike_object)

@gpshead gpshead May 18, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like to make the first line descriptive comment a docstring. it shows up nicely in verbose testing mode.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/library/subprocess.rst
*cwd* parameter accepts a :term:`path-like object`.
*cwd* parameter accepts a :term:`path-like object` on POSIX.

.. versionchanged:: 3.7

@gpshead gpshead May 18, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI - these 3.6 and 3.7 versionchanged updates should be made to the docs today to clarify the existing behavior on their own (and backported to at least the 3.7 docs)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Lib/subprocess.py
else:
args = list(args)
args[0] = os.fsdecode(args[0])
args = list2cmdline(args)

@gpshead gpshead May 18, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does look like it would be nicer, allowing more mixing and matching of types.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
serhiy-storchaka added 2 commits May 25, 2019 22:53
@serhiy-storchaka
Merge branch 'master' into subprocess-pathlike
fe37262
@serhiy-storchaka
Merge branch 'subprocess-pathlike' of github.com:serhiy-storchaka/cpy…
c6b0ade
…thon into subprocess-pathlike
asvetlov
asvetlov approved these changes May 28, 2019
View reviewed changes

@asvetlov asvetlov left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@gpshead merging it into Python 3.8 would be the really nice improvement of subprocess support on Windows

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@lilydjwg lilydjwg mentioned this pull request May 28, 2019
bpo-35246: fix support for path-like args in asyncio subprocess #13628
Merged
@asvetlov

asvetlov commented May 28, 2019

Copy link
Copy Markdown
Contributor

#13628 depends on this PR for Windows tests green (we can skip Windows, sure. But I honestly prefer to fix upstream :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka
serhiy-storchaka merged commit 9e3c452 into python:master May 28, 2019
@bedevere-bot bedevere-bot removed the awaiting merge label May 28, 2019
@serhiy-storchaka
serhiy-storchaka deleted the subprocess-pathlike branch May 28, 2019 19:49
@serhiy-storchaka

serhiy-storchaka commented May 28, 2019

Copy link
Copy Markdown
Member Author

I think that I addressed all review comments, but if I missed something it can be fixed in the other PR (even after beta1).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@asvetlov

asvetlov commented May 28, 2019

Copy link
Copy Markdown
Contributor

Thanks, @serhiy-storchaka !

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@serhiy-storchaka @DinoV
bpo-31961: Fix support of path-like executables in subprocess. (pytho…
0ba5eb7
…nGH-5914)
@ambv ambv mentioned this pull request Feb 9, 2023
[3.7] gh-101283: Improved fallback logic for subprocess with shell=True on Windows (GH-101286) #101713
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead left review comments
@eryksun eryksun eryksun left review comments
@asvetlov asvetlov asvetlov approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@serhiy-storchaka @asvetlov @gpshead @eryksun @the-knights-who-say-ni @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.