GeezerVison is showing github.com — If the working directory and the venv are on different disks, venv doesn't work in bash on Windows · Issue #103088 · python/cpython
If the working directory and the venv are on different disks, venv doesn't work in bash on Windows #103088DescriptionEnvironment
- OS: Windows 10
- Architecture: x86_64
- Shell: bash 4.4.23, part of the git 2.37.0 install
- Python version: 3.11.1
- Output of
path/to/python -m pip list:
Package Version
---------- -------
pip 23.0.1
setuptools 65.5.0 Issue
When I create a venv on D: and use it from C:, executables like pip or python can't be found:
$ pwd
/d
$ /path/to/python -m venv myenv
$ . myenv/Scripts/activate
$ pip --version
pip 22.3.1 from D:\myenv\Lib\site-packages\pip (python 3.11)
$ cd /c
$ pip --version
bash: \myenv/Scripts/pip: No such file or directory Linked PRs
Activityeryksun commented on Mar 28, 2023 A POSIX shell delimits PATH with ":". Thus "D:\myenv" represents two separate pathnames "D" and "\myenv". If a path begins with "\", MSYS apparently resolves it against the drive path of the current working directory (e.g. "/c" or "/d").
You can work around this issue by editing "{env_dir}/Scripts/activate" to replace the value of VIRTUAL_ENV with the equivalent MSYS path (e.g. "D:\myenv" -> "/d/myenv").
Maybe the "activate" script could detect MSYS or Cygwin and automatically translate the value of VIRTUAL_ENV via export VIRTUAL_ENV=$(cygpath -u "$VIRTUAL_ENV"). Zabolekar commented on Mar 29, 2023 ContributorAuthor More actions Thanks for the answer. My current workaround was to move venvs and code to the same disk.
Now that you mention Cygwin I've tested the script with Cygwin's version of bash as well. The script fails because it contains \r. After applying dos2unix, it works but has the same issue as MSYS bash.
Let's say I want to fix the script for everyone and not just for me. What would be the process, and what bash variants other than MSYS and Cygwin should I be aware of? eryksun commented on Mar 29, 2023
Let's say I want to fix the script for everyone and not just for me. What would be the process, and what bash variants other than MSYS and Cygwin should I be aware of?
Cygwin and MSYS2 are the common POSIX environments on Windows, other than WSL. Create a pull request that modifies the template script "Lib/venv/scripts/common/activate".
|
VIRTUAL_ENV="__VENV_DIR__" |
|
export VIRTUAL_ENV |
Apparently MSYS and Cygwin can be detected on bash using the OSTYPE environment variable. The following change seems to do the job. I haven't checked whether it causes a problem for an environment that's created by the MSYS2 or Cygwin versions of Python, but I don't think it should cause a problem. cygpath -u should only modify a Windows path such as "D:\myenv", not a POSIX path such as "/d/myenv" or "/cygdrive/d/myenv".
if [ "$OSTYPE" = "cygwin" ] || [ "$OSTYPE" = "msys" ] ; then
export VIRTUAL_ENV=$(cygpath -u "__VENV_DIR__")
else
export VIRTUAL_ENV="__VENV_DIR__"
fiThe venv module replaces the __VENV_DIR__ template parameter with the path of the created virtual environment. Zabolekar commented on Apr 6, 2023 ContributorAuthor More actions Thanks for your help. I didn't even know MSYS had cygpath, too. I've tested creating venvs in cmd and in MSYS2, then activating them in cmd, MSYS2, and cygwin, and looking at pip --version and python --version. Everything works as expected. Here is a pull request.
Two things I've noticed while doing this, not sure whether those are bugs or not:
- the script has CR LF, so Cygwin only understands it after I run dos2unix on it
- https://devguide.python.org/ advises to build on Windows with
-d by default (set the configuration to Debug). So the resulting venvs don't contain python.exe, only python_d.exe, and calling pip directly doesn't work ("Fatal error in launcher"). If Python has been built without -d, everything works as expected.
zooba commented on Apr 11, 2023
- the script has CR LF, so Cygwin only understands it after I run dos2unix on it
It may depend on your Git settings, if you've cloned and built from source. Or maybe this file gets modified when venv copies it? @vsajip any thoughts?
- resulting venvs don't contain python.exe, only python_d.exe, and calling pip directly doesn't work
Guessing this is a script launcher limitation, or maybe they aren't being generated properly in this scenario. In any case, it's external - I believe pip still gets their launchers from distlib? Zabolekar commented on Apr 11, 2023 ContributorAuthor More actions
It may depend on your Git settings, if you've cloned and built from source
It doesn't only affect those who build from source: Python 3.11.1 that I installed from python.org has CR LF in Lib/venv/scripts/common/activate, too.
It doesn't feel right to just clone everything with Unix line endings on Windows. I'm not sure whether all relevant programs can handle it correctly. On my computer, it seems to suffice if I add Lib/venv/scripts/common/activate text eol=lf to .gitattributes, then delete the file and do git checkout-index --force --all to recreate it. But I don't know how to fix it for the installer as well. zooba commented on Apr 11, 2023 If that's added to .gitattributes then it should work fine for the installer. We don't do anything to that file.
I'd be more concerned about venv creation reading, modifying and writing the file with different line endings. If it doesn't do that, then modifying the attributes should be enough. vsajip commented on Apr 12, 2023
Or maybe this file gets modified when venv copies it?
The activate script in my repo on Linux has LF terminators, and the copying code reads and writes the files as binary (there are some variable substitutions, but none of those should introduce CRLF line endings). added a commit that references this issue on Apr 12, 2023gh-103088: Fix virtual environment activate script not working in Cyg… zooba commented on Apr 12, 2023 I've merged both fixes - any thoughts on backports? I think 3.11 is going to last long enough to be worth it, and while it won't help anyone who doesn't recreate their venvs, we'll have plenty of new users in the future. zooba commented on Apr 12, 2023 Well, buildbot failures mean we won't be backporting soon. Looks like:
- some buildbots ignore
.gitattributes (or maybe just need a clean checkout?)
- we need to change the second parameter to
assertFalse to use line.decode() - it doesn't like bytes
The second issue only appears because the test is failing, so worth fixing that one first. zooba commented on Apr 13, 2023 Hopefully this (#103524) approach to cleaning up all the buildbots will work. Would be nice to have something a bit narrower/faster, but this will do. db3l commented on Apr 13, 2023 Isn't that just using the PR branch directory on the buildbots and not any of the actual branch trees?
Sort of a kludge, but perhaps actually touching the activate file in some innocuous way to ensure it will have to be updated simultaneous with or subsequent to the .gitattributes change?
Or, any thought to just not going down this road, and expecting Cygwin users to use the igncr shell option? It's been a while since my Cygwin days but having to deal with CRLF scripts wasn't that uncommon. Separately, as it is, this change only fixes use under Cygwin with sh/bash (e.g., the csh script is still CRLF). zooba commented on Apr 13, 2023 Ah yeah, it will only touch the PR branches.
Touching the file and merging it is probably the best option then. We've already merged the .gitattributes change, and I think it's fine to leave the line endings alone. Windows is basically okay with LF endings virtually everywhere these days. If there's another file that ought to have LF endings preserved, then we should update that one at the same time (i.e. before the RM starts looking at buildbots for the next release). db3l commented on Apr 13, 2023 I just happened to notice the activate.csh and activate.fish versions in the posix folder, which are also CRLF on Windows. It seemed plausible that changes for activate would hold for those too. Then again, the venn diagram overlap of Cygwin users using those shells with Python testing may be null... Reacted by Stanislav Syekirin zooba commented on Apr 13, 2023 I don't think those even get copied for a venv on Windows, even in Cygwin. We moved activate to common for Cygwin (specifically Git Bash) users. But it wouldn't be unreasonable to mark the whole posix folder there as explicitly LF, just in case. Zabolekar commented on Apr 14, 2023 ContributorAuthor More actions
Cygwin (specifically Git Bash)
But git bash is not Cygwin. GalaxySnail commented on Apr 14, 2023 zooba commented on Apr 17, 2023 It's where the source code lives, ultimately. So most shell bugs/limitations that affect Cygwin also end up affecting msys2 and Git Bash, which makes them far more significant than "just" Cygwin.
I'll update the attributes and touch the files. zooba commented on Apr 18, 2023 I'm going to consider this closed, unless someone really wants it backported to 3.11
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|