GeezerVison is showing github.com — Default to --user · Issue #1668 · pypa/pip
DescriptionWhen pip is installed on a system that has an OS Python install there is currently a problem where pip install foo will throw an error because it doesn't have file permissions. This causes people to instead run sudo pip install foo which globally installs to the system Python. This creates an issue where people are using pip to manage system level packages when they should likely be using the system package manager.
So my intention is that pip should default to --user however there are a few sticking points with this:
- How does this interact with Windows? Does this make sense there?
- How does this interact with altinstall'd Pythons? Specially such as are installed with tools like https://github.com/yyuu/pyenv
- What do we do for when people do invoke pip as root? Installing into
/root/.local/ does not seem very useful.
- What does this mean for
get-pip and the pypa install instructions?
- ~/.local/bin is not on many people's $PATH, is there anything that can be done about this?
--user installs lack precedence to global easy_install'd packages, which can be quite unexpected.
There are a number of issues that are relevant here: #624 #1443 #1153 #1500 #1051
/cc @ncoghlan Reacted by Cody Scott, abigailbunyan, Yijie Ma, Walter Dolce, Thomas Kluyver, Russell Jones, Zen, David Tucker, Edwin Khoo, Leo Gallucci and 144 more Reacted by Ikanihoda Activityqwcode commented on Mar 21, 2014
What does this mean for the pypa install instructions?
currently the mention of root or Administrator access is in a footnote, so the instructions would fundamentally stay the same (assuming get-pip automatically defaulted to --user as well). the change would be to explain that they will end up with installs that only apply to their current user.
regardless of the --user change, the current instructions need to explain that some distributions like debian, are disabling ensurepip, so they shouldn't go looking for it as a way to get pip. dstufft commented on Mar 21, 2014 Long term I'm pretty sure they are going to keep it enabled, they are just figuring out how to do it. qwcode commented on Mar 21, 2014 although this issue is described as being about where pip should manage packages by default, almost more important (to me at least) is that this is indirectly about where get-pip will place pip by default. qwcode commented on Mar 21, 2014
How does this interact with altinstall'd Pythons?
Specially such as are installed with tools like https://github.com/yyuu/pyenv
the user scheme is .local/lib/pythonX.Y/site-packages, so if you're managing multiple pythons with the same major/minor version, you could end up with a mess I guess.
for tools like that, that manage real pythons, pinning pip to global mode would make sense for those. dstufft commented on Mar 21, 2014 Reacted by Ahmad Yoosofan dstufft commented on Mar 21, 2014 It looks like for Fedora/RedHat PATH=$PATH:$HOME/.local/bin:$HOME/bin is already in their /etc/skel/.bash_profile. This would make stuff installed with --user still show up by default, at least for bash users (default shell). Maybe other distros already have this? Reacted by Ahmad Yoosofan pfmoore commented on Mar 21, 2014 There's nothing specially different about Windows that I know of. But we've only just got the standard Python directory onto people's PATH, I don't expect that getting a per-user scripts directory on is going to be particularly easy - and there may be technical difficulties as well, I'm not sure putting a user-level environment variable like %LOCALAPPDATA% into the system-level %PATH% even works :-(
I'm against rushing this. I would prefer waiting till we've had more experience with user installs and have managed to iron out the issues first. I know that's a chicken and egg problem, but I don't see rushing into a change as helping.
Also, with my Windows hat on, I'd have to say that making it harder for Unix users who haven't yet worked out that careless use of sudo is bad to do stupid things, isn't really that good a justification...
I'm assuming that this would only apply when running pip from a system Python. dstufft commented on Mar 21, 2014 Well the different thing on Windows is there isn't a system provided Python like there is on *nix that also comes with system provided Python packages :)
I'm not particularly wanting to rush this either. I just wanted to start the discussion.
This doesn't really affect people who type sudo pip install as I think installing something as root should still go to the system site packages by default. This would be to guide people towards using --user when running as an unprivileged user. Right now what you get is that pip install as a regular user just bombs out with a permission error. Even if we improve that message to bomb out and suggest --user that's still not the greatest UX.
In talking to one of the Fedora people, I think the way it may make to do this is to implement a permissions check. If I have permission to write to the site-packages directory then I am a privledged user and pip installs to the global Python, If I do not then I am an unprivileged user and it installs to --user. ncoghlan commented on Mar 21, 2014 As far as Windows goes, we usually duck the problem because we install Python to a non-privileged directory by default, so pip doesn't trigger UAC when installing globally. If someone changes that to install into Program Files instead, then UAC will trigger when they run pip (which is also OK). I'm not sure what happens if they select a per-user install in the installer.
As Paul notes, PATH on Windows doesn't include the user scripts directory yet (just the global one), so that's definitely worth taking into account.
I don't see any major barriers on the POSIX side though. How does this UX idea sound: for wheel based installs, check for permissions on the installation target directory, and if the user doesn't have write permissions, put up a prompt asking if they would like to do a --user install instead? A new "--global" or "--system" flag would force the "no" answer.
And then at some point in the future, we could skip the prompt and simply assume --user if the permissions weren't right for a global install. Reacted by James E. A., Marshak Igor and schwarzichet dstufft commented on Mar 21, 2014 A prompt (except in the case of --no-input) probably makes sense for the transition step. I think checking permissions should make things work for Windows too since by default the location is in a spot where people have permissions on and so the only time they'd hit this would be if they are using a systems admin provided Python that explicitly didn't give them permission.
Essentially this makes our failure mode much better. Reacted by Ahmad Yoosofan qwcode commented on Mar 21, 2014
making it harder for Unix users who haven't yet worked out that careless use of sudo
is bad to do stupid things, isn't really that good a justification..
sudo and security is really a secondary issue IMO.
this is primarily about preventing the conflict between OS packaging and pip in the system python.
Currently, linux users are often placed in an impossible situation.
- OS Mandate: 'Use the OS pkg mgr; Don't infect your system with roque pip-installed packages (including a
get-pip'd pip)'
- Reality: "The OS packages (including pip) are too old, and I can't upgrade to the experimental release of the distro, just to get upgrades of some package. I'm going rogue...."
Reacted by James E. A., Dharam, I AM, Glenn Willen and miked0004 qwcode commented on Mar 21, 2014 another baby step is to document (assuming we test it) that you can get-pip.py --user, and have the PUG mention --user as a possibility for the install of virtualenv (and wheel, and twine too, if not in a virtualenv) ncoghlan commented on Mar 21, 2014 FWIW, distro vendors also consider the status quo a problem and are looking at various ways to improve the tools for doing selective upgrades without impacting core OS components. Still worth us tackling from the upstream side though, since those efforts are at various stages of maturity and we want a consistent cross-platform solution for end users. pfmoore commented on Mar 22, 2014 I have no problem with changing things to help co-operate with distros on Unix; if defaulting to --user on Unix and not on Windows is acceptable, that's fine with me. I just don't see switching from a bad experience on Unix to a bad experience on Windows as a good trade-off. And I'm yet to be convinced that --user on Windows is ready for prime time.
Note that triggering UAC when running pip is bad on Windows, because what it actually means is that pip won't run except in an elevated console window. It does not mean that pip users get a nice prompt which they can say "OK" to, unfortunately, that's only how GUI apps work... dstufft commented on Mar 22, 2014 @pfmoore What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory? Aricg commented on Jan 22, 2020 I followed the release notes here.
https://pip.pypa.io/en/stable/news/#id3
"Default to doing a user install (as if --user was passed) when the main site-packages directory is not writeable and user site-packages are enabled. (#1668)"
My builds are now failing with
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
This seems fixed by removing the --user from my pip commands.
Was this expected behavior? The release notes to not seem to reflect this.
link to the script that now fails on pip 20.0.1 (removing the --user fixes the hard fail)
https://github.com/lfit/releng-global-jjb/blob/master/shell/python-tools-install.sh takluyver commented on Jan 22, 2020 If it is correct that user site packages aren't importable from that environment, then that is the expected behaviour, but it's orthogonal to this PR. That check was introduced years ago (#567), but it looks like it was broken with venv until recently (#7155). There's a release note for that:
Correctly handle system site-packages, in virtual environments created with venv (PEP 405).
Aricg commented on Jan 23, 2020 Ah yes, thank you for your comments. I figured this out...
my script was populating .local/bin/
when I ran
and then in a ubuntu login shell (#!/bin/bash -l) .local/bin gets added to path (so calling python3 now calls the .local/bin/python3 executable) Which we don't want.
I never needed to run python3 -m venv ~/.local
or call my scripts from a login shell, and removing both or either of these problems fixes it.
#fresh ubuntu docker
root@7d8107816f64:/# python3 -m pip install --user --upgrade pip
Successfully installed pip-20.0.1
root@7d8107816f64:/# python3 -m pip --version
pip 20.0.1 from /root/.local/lib/python3.6/site-packages/pip (python 3.6)
root@7d8107816f64:/# ls root/.local/bin/pip
pip pip3 pip3.6
#Now we populate ~/.local/bin/
root@7d8107816f64:/# python3 -m venv ~/.local
root@7d8107816f64:/# ls root/.local/bin/
activate activate.fish easy_install-3.6 pip3 python
activate.csh easy_install pip pip3.6 python3
root@7d8107816f64:/# ./root/.local/bin/python --version
Python 3.6.9
root@7d8107816f64:/# ./root/.local/bin/python -m pip install --user --upgrade pip
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
So if we ever run the executable .local/bin/python we can not pip install with --user takluyver commented on Jan 23, 2020 Oh, I'd missed that you were doing python3 -m venv ~/.local . That's likely to confuse tools like pip, because ~/.local is the prefix for --user installs (on Linux). venvs and --user are two different ways you can install packages without making systemwide changes: you should use one or the other. Creating a venv in ~/.local makes some weird hybrid of the two. tgs commented on Mar 18, 2020 The new behavior is presumably convenient for many, but 'user' installs have caused me some grief as a web developer - I often need to prepare virtualenvs that will be deployed to our production machines, and if any package is installed in my user library, then by default pip will refuse to install it to the virtualenv I'm preparing, and will still exit with success. The notification that it decided not to install the package is buried inside 30 or 40 pages of output from the script I use to prepare the virtualenv. The end effect is that once the virtualenv is deployed, the packages from my home dir are no longer available, so some random packages are missing from my production web app, and it falls over dead.
I'm willing to admit that my use case is not the typical one, so maybe this was still a good idea from a general usability point of view.
For others who have had similar problems, there are a few ways around it:
- Add
--force-reinstall to all your pip install command lines in scripts etc
- Delete
~/.local/lib/python* and edit ~/.pip/pip.conf to add user = no in the global section:
- Do something more drastic, like change
~/.local/lib to a file rather than a directory 😏
As always, dear packaging maintainers, thank you for all you do - I'm sure it's virtually impossible to change anything without breaking someone's stuff, and the fact that you forge on anyway is actually great, because it lets us make progress. tgs commented on Mar 18, 2020 Ah, user installs can also cause huge problems in continuous integration environments, where lots of different stuff is run as the same user account, but people still expect builds to be isolated from each other. We solved this by making ~/.local read-only, but the solutions above would probably also work. takluyver commented on Mar 19, 2020 When you create a virtualenv with the default options, it's isolated: it can't see your user or system site-packages, and consequently pip should only work with packages installed in the env. This became the default many years ago, because of exactly the kinds of issues you describe.
If you create virtualenvs with the --system-site-packages option (or very old versions of virtualenv where that was the default), they act as an overlay on the packages you'd otherwise have installed, both system-wide and in your home directory. If you want to have system packages visible but not user packages, you can run Python with the -s option or the PYTHONNOUSERSITE environment variable. tgs commented on Mar 19, 2020 Oh! I'm very sorry, you're totally right. My organization's weird system-site-packages policy strikes again. Thanks for pointing out some solutions. Reacted by Thomas Kluyver takluyver commented on Mar 19, 2020 No problem. System site packages was more common a few years ago when we often relied on system package managers for things like numpy, so maybe it's worth revisiting that policy. But there may still be sensible reasons to use it today. pradyunsg commented on Mar 19, 2020 catPill commented on Apr 10, 2020 takluyver commented on Apr 10, 2020 locked as resolved and limited conversation to collaborators on May 20, 2020
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|