GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Adjust shutil.move stubs to allow bytes/Pathlike[bytes] args by jpy-git · Pull Request #6832 · python/typeshed

Conversation

@jpy-git

@jpy-git jpy-git commented Jan 5, 2022

Copy link
Copy Markdown
Contributor

Closes #6831. Thorough example included in the issue.

TLDR: shutil.move can take bytes arguments

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@jpy-git
Adjust shutil.move stubs to allow bytes/Pathlike[bytes] args
c4fe8dc
jpy-git
jpy-git commented Jan 5, 2022
View reviewed changes
Comment thread stdlib/shutil.pyi Outdated
else:
# See https://bugs.python.org/issue32689
def move(src: str, dst: StrPath, copy_function: _CopyFn = ...) -> _PathReturn: ...
def move(src: str | bytes, dst: StrOrBytesPath, copy_function: _CopyFn = ...) -> _PathReturn: ...

@jpy-git jpy-git Jan 5, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI the linked bug applies to Pathlike[bytes] as well, but I've verified it works fine with bytes for Python < 3.9.

>>> import shutil
>>> shutil.move(b"test1.txt", b"test2.txt")
'test2.txt'

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@github-actions

This comment has been minimized.

Sign in to view
@Akuli

Akuli commented Jan 5, 2022

Copy link
Copy Markdown
Collaborator

I don't think mixing str and bytes paths works. The source code has this:

real_dst = os.path.join(dst, _basename(src))

which fails if it tries to join strings and bytes.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jpy-git

jpy-git commented Jan 5, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

I don't think mixing str and bytes paths works. The source code has this:

real_dst = os.path.join(dst, _basename(src))

which fails if it tries to join strings and bytes.

This def works for me:

>>> import shutil
>>> shutil.move("test.txt", b"test2.txt")
b'test2.txt'
>>> shutil.move(b"test2.txt", "test.txt")
'test.txt'
>>> shutil.move("some_folder", "some_folder2")
'some_folder2'
>>> shutil.move(b"some_folder2", b"some_folder")
b'some_folder'
>>> shutil.move(b"some_folder", "some_folder2")
'some_folder2'
>>> shutil.move("some_folder2", b"some_folder")
b'some_folder'

EDIT: I noticed that bit in the source code only impacts directories, so I've added them above. They still seem to work though.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Akuli

Akuli commented Jan 5, 2022

Copy link
Copy Markdown
Collaborator

The code I mentioned runs when dst is a directory.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jpy-git

jpy-git commented Jan 5, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

The code I mentioned runs when dst is a directory.

Ahh it's only when dst is an existing directory.

In that case yes I get errors:

>>> shutil.move("some_folder", b"some_existing_folder")
...
TypeError: Can't mix strings and bytes in path components
>>> shutil.move(b"some_folder", "some_exising_folder")
...
File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 771, in _basename
    return os.path.basename(path.rstrip(sep))
TypeError: a bytes-like object is required, not 'str'

Feels like a bug given that it's only in this section of the code...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jpy-git

jpy-git commented Jan 5, 2022

Copy link
Copy Markdown
Contributor Author

The second of those issues seems like a bigger issue since this would also fail:

>>> shutil.move(b"some_folder", b"some_exising_folder")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 810, in move
    real_dst = os.path.join(dst, _basename(src))
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 771, in _basename
    return os.path.basename(path.rstrip(sep))
TypeError: a bytes-like object is required, not 'str'
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jpy-git

jpy-git commented Jan 5, 2022

Copy link
Copy Markdown
Contributor Author

With those test cases, I think we should close this PR since bytes will always break the existing folder case.

Do you happen to know where the best place flag this issue is for cpython?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hauntsaninja

hauntsaninja commented Jan 5, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

Edit: disregard my comment, I just pattern matched to a thing I'd seen before (shutil has historically not supported path objects well, in general worth double checking these kinds of changes against older Pythons)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jpy-git

jpy-git commented Jan 6, 2022

Copy link
Copy Markdown
Contributor Author

No worries, it looks like the types shutil functions accept is a bit inconsistent in general (although looks like they're improving) 😄

I'm working through the other shutil functions atm as I'm finding a lot of them work fine with bytes. Once I've gone through all the others we should hopefully have quite a strong precedent for getting move updated to accept bytes arguments in the existing dst case.

In the meantime, maybe it's worth me removing my changes and just adding a comment above the move stub mentioning the reasoning for not accepting bytes argument (or a link to this PR) so people are aware of our reasoning for not changing it (given that from my examples its easy to be misled by this function)?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra

JelleZijlstra commented Jan 7, 2022

Copy link
Copy Markdown
Member

In the meantime, maybe it's worth me removing my changes and just adding a comment above the move stub mentioning the reasoning for not accepting bytes argument (or a link to this PR) so people are aware of our reasoning for not changing it (given that from my examples its easy to be misled by this function)?

That seems right to me.

jpy-git reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jpy-git
Revert type changes and add explanatory comment
cdc0a15
@jpy-git

jpy-git commented Jan 7, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

I've reverted the typing changes and added a comment linking to this PR to highlight for future contributors 👍

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions

github-actions Bot commented Jan 7, 2022

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Akuli
Akuli approved these changes Jan 8, 2022
View reviewed changes

@Akuli Akuli left a comment

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Akuli
Akuli merged commit 0e5a00b into python:master Jan 8, 2022
@Akuli

Akuli commented Jan 8, 2022

Copy link
Copy Markdown
Collaborator

Do you happen to know where the best place flag this issue is for cpython?

bugs.python.org. My experience with reporting and fixing bugs in cpython hasn't been great, but you might have better luck.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra JelleZijlstra mentioned this pull request Jan 15, 2022
Fix shutil.make_archive stub #6868
Closed
eaftan added a commit to eaftan/python-dotenv that referenced this pull request Jan 10, 2023
@eaftan
Allow only string paths, not byte paths
183e59e
These paths can flow into `shutil.move`, which does not accept byte
paths or (int) file descriptors.  See python/typeshed#6832
theskumar pushed a commit to theskumar/python-dotenv that referenced this pull request Jan 11, 2023
@eaftan
Fix type hint for dotenv_path var, add StrPath alias (#432)
09cf4ab
* Fix type hint for load_dotenv

Fixes #431

* Quote type hints to avoid runtime errors in earlier Python versions

* Revise type of dotenv_path parameter

Based on PR feedback and typeshed's type hint for the built-in open()
function:
https://github.com/python/typeshed/blob/e2d67bf7034f68c07bd35150247e58e0817725d9/stdlib/builtins.pyi#L1421

* Allow only string paths, not byte paths

These paths can flow into `shutil.move`, which does not accept byte
paths or (int) file descriptors.  See python/typeshed#6832

* Create a type alias for the paths this library accepts

And use it consistently in main.py.
@JelleZijlstra JelleZijlstra mentioned this pull request Oct 5, 2023
Should shutil functions support bytes paths? python/cpython#90883
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Akuli Akuli Akuli approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

shutil.move stub does not account for bytes paths

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@jpy-git @Akuli @hauntsaninja @JelleZijlstra
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.