GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-75723: Fix double evaluation of .pth and .site files in venvs by warsaw · Pull Request #149583 · python/cpython

Conversation

@warsaw

@warsaw warsaw commented May 8, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This is a more comprehensive fix for Python 3.15 for #75723 reported by @asottile (thanks!). This should not be backported since it's specific to the additions made for PEP 829. I'll evaluate and update !147951 for 3.13 and 3.14 separately.

Huge thanks to @asottile for finding, reporting, and fixing the original bug.

  • Issue: pth files in site-packages of venvs are executed twice #75723

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

tcely reacted with thumbs up emoji asottile, mrswats, ddellspe, and emmatyping reacted with heart emoji
All reactions
  • 👍 1 reaction
  • ❤️ 4 reactions
asottile and others added 11 commits March 31, 2026 16:16
@asottile
Idempotent .pth execution in site.addsitedir
6c342df
@FFY00
Merge branch 'main' into asottile-pythongh-75723
96ed0e8
@asottile
potentially fix monkeypatch leak?
c44e192
@asottile
fix blind copy paste of recommendation
f077abd
@asottile
Update 2026-03-31-16-15-15.gh-issue-75723.BZ4Rsn.rst
aeee2ff
@asottile
Merge remote-tracking branch 'origin/main' into asottile-pythongh-75723
4005472
@asottile
fix implicit merge conflict with 24c4aec
66ebfc9
@warsaw
Merge branch 'main' into asottile-pythongh-75723
63df6bd
@warsaw
Add failing tests for pythongh-75723
d96a414
Based on @asottle branch !asottle-pythongh-75723 but refactored for `main`.
This will need a different backport.
@warsaw
Repair pythongh-75723
61ba389
The original fix is here: python#147951
but I'm refactoring a bit for `main`.
@warsaw
Refactor _make_mod() so we can use it to create package modules too
d1dc0ee
@warsaw warsaw self-assigned this May 8, 2026
@warsaw
warsaw requested a review from FFY00 as a code owner May 8, 2026 20:56
@bedevere-app bedevere-app Bot mentioned this pull request May 8, 2026
pth files in site-packages of venvs are executed twice #75723
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label May 8, 2026
@warsaw
Add myself to CODEOWNERS for the site module
d825329
@warsaw
warsaw requested review from AA-Turner, ezio-melotti, hugovk and webknjaz as code owners May 8, 2026 20:58
warsaw added 3 commits May 8, 2026 13:59
@warsaw
Merge branch 'main' into warsaw/asottle-gh-75723
d123fe1
@warsaw
Merge branch 'main' into warsaw/asottle-gh-75723
95895dd
@warsaw
Merge branch 'main' into warsaw/asottle-gh-75723
b43c3c9
ncoghlan
ncoghlan approved these changes May 10, 2026
View reviewed changes

@ncoghlan ncoghlan left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_site.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels May 10, 2026
@warsaw

warsaw commented May 11, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review @ncoghlan !

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@warsaw
warsaw merged commit 1978785 into python:main May 11, 2026
56 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label May 11, 2026
@warsaw
warsaw deleted the warsaw/asottle-gh-75723 branch May 11, 2026 00:25
@scoder

scoder commented May 12, 2026

Copy link
Copy Markdown
Contributor

We are seeing CI test failures in Cython with the latest CPython main branch since yesterday, and this PR seems the most likely culprit. Some of our tests start Python commands in subprocesses (on Linux/fork) that build code using setuptools, which injects a fake distutils import via a .pth file. The error we see is that the distutils package can no longer be found and imported from the subprocesses, whereas the main process has no problem using it.

From a quick look through this change, it feels like a status leak into the child process, but I couldn't immediately pin point one. Any idea what might go wrong here?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@scoder scoder mentioned this pull request May 12, 2026
Refactor type related error raising into separate utility functions cython/cython#7680
Merged
@warsaw

warsaw commented May 12, 2026

Copy link
Copy Markdown
Member Author

We are seeing CI test failures in Cython with the latest CPython main branch since yesterday, and this PR seems the most likely culprit. Some of our tests start Python commands in subprocesses (on Linux/fork) that build code using setuptools, which injects a fake distutils import via a .pth file. The error we see is that the distutils package can no longer be found and imported from the subprocesses, whereas the main process has no problem using it.

From a quick look through this change, it feels like a status leak into the child process, but I couldn't immediately pin point one. Any idea what might go wrong here?

If you're able, please try with #149659

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@scoder

scoder commented May 12, 2026

Copy link
Copy Markdown
Contributor

If you're able, please try with #149659

I patched the site.py changes of that PR into my local CPython (and created a fresh venv from it) but couldn't find a difference in behaviour. Still failing.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@warsaw

warsaw commented May 12, 2026

Copy link
Copy Markdown
Member Author

I patched the site.py changes of that PR into my local CPython (and created a fresh venv from it) but couldn't find a difference in behaviour. Still failing.

Can you please go over to #149659 and post a reproducer?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@warsaw

warsaw commented May 12, 2026

Copy link
Copy Markdown
Member Author

If you're able, please try with #149659

I patched the site.py changes of that PR into my local CPython (and created a fresh venv from it) but couldn't find a difference in behaviour. Still failing.

Oh, sorry, patching site.py won't fix the problem because that module is frozen. You need to rebuild Python from source after running make regen-all. I don't think there's an easy way to run a built Python with a patched site.py overriding the frozen module.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@warsaw

warsaw commented May 12, 2026

Copy link
Copy Markdown
Member Author

@scoder Run Python with PYTHON_FROZEN_MODULES=off or -X frozen_modules=off

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@scoder

scoder commented May 13, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

I tried with frozen modules off, no difference, then ran make regen-all and rebuilt, no difference.
I'll look into a reproducer and post it in the other ticket. Thanks for the feedback so far.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@warsaw

warsaw commented May 13, 2026

Copy link
Copy Markdown
Member Author

I tried with frozen modules off, no difference, then ran make regen-all and rebuilt, no difference. I'll look into a reproducer and post it in the other ticket. Thanks for the feedback so far.

@scoder if you don't mind, I'm going to go ahead and merge the other PR, and since then both will be closed, if you are able to craft a reproducer with main, please open a new issue. Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@scoder

scoder commented May 14, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

I think the issue is with sys.path when starting a Python subprocess. It already contains all the paths that the parent process added, so the new implementation won't evaluate any .pth files in a subprocess.

This is now #149819

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@scoder scoder mentioned this pull request May 14, 2026
New site.py implementation fails to load .pth files in subprocesses #149819
Closed
@IntentBug IntentBug mentioned this pull request May 15, 2026
gh-149819: Fix .pth files not loaded in Python subprocesses #149888
Closed
IntentBug pushed a commit to IntentBug/cpython that referenced this pull request May 15, 2026
BugBounty Mind
pythongh-149819: Fix .pth files not loaded in Python subprocesses
b7f6ba1
After PR pythongh-149583 (Fix double evaluation of .pth and .site files in
venvs), .pth files are no longer loaded in subprocesses started with
subprocess.run([sys.executable, ...]).  The root cause: main() seeds
known_paths from removeduppaths() with all sys.path entries inherited
from the parent process.  addsitedir() then skips .pth processing for
every directory already in known_paths.

Fix:
- main(): call removeduppaths() for dedup but start known_paths as a
  fresh empty set, so that addsitedir() processes .pth files in every
  site-packages directory regardless of inherited sys.path.
- addsitedir(): move known_paths.add() before the sys.path.append and
  guard the append with 'sitedir not in sys.path' to avoid creating
  duplicate entries when called with a fresh known_paths.

This preserves the pythongh-75723 dedup guarantee while allowing subprocesses
to load .pth files.
@scoder scoder mentioned this pull request May 18, 2026
Fix type inference of subscripted list slice cython/cython#7689
Merged
warsaw added a commit that referenced this pull request May 21, 2026
@warsaw BugBounty Mind @scoder
gh-149819: fix .pth and .start file processing in subprocess when inh…
3c298e2
…eriting PYTHONPATH (#150177)

* gh-149819: Fix .pth files not loaded in Python subprocesses

After PR gh-149583 (Fix double evaluation of .pth and .site files in
venvs), .pth files are no longer loaded in subprocesses started with
subprocess.run([sys.executable, ...]).  The root cause: main() seeds
known_paths from removeduppaths() with all sys.path entries inherited
from the parent process.  addsitedir() then skips .pth processing for
every directory already in known_paths.

Fix:
- main(): call removeduppaths() for dedup but start known_paths as a
  fresh empty set, so that addsitedir() processes .pth files in every
  site-packages directory regardless of inherited sys.path.
- addsitedir(): move known_paths.add() before the sys.path.append and
  guard the append with 'sitedir not in sys.path' to avoid creating
  duplicate entries when called with a fresh known_paths.

This preserves the gh-75723 dedup guarantee while allowing subprocesses
to load .pth files.

* Fill out the tests for GH#149888

* Extend _make_start() and _make_pth() to take an optional `basedir` which is used instead of
 `site.tmpdir` if given.
* Add test_pth_processed_when_sitedir_already_on_path() to test the core GH#149819 bug: .pth files
  in subprocesses aren't handled if PYTHONPATH pointing to the .pth directory is inherited.
* Similarly add test_start_processed_when_sitedir_already_on_path() to verify that .start files in
  the same circumstances are also now processed.

* Update Lib/site.py

Co-authored-by: scoder <[email protected]>

* Oops!  Remove redundant code

---------

Co-authored-by: BugBounty Mind <[email protected]>
Co-authored-by: scoder <[email protected]>
@miss-islington miss-islington mentioned this pull request May 21, 2026
[3.15] gh-149819: fix .pth and .start file processing in subprocess when inheriting PYTHONPATH (GH-150177) #150202
Merged
warsaw added a commit that referenced this pull request May 21, 2026
@miss-islington @warsaw
BugBounty Mind @scoder
[3.15] gh-149819: fix .pth and .start file processing in subprocess w…
233cf75
…hen inheriting PYTHONPATH (GH-150177) (#150202)

gh-149819: fix .pth and .start file processing in subprocess when inheriting PYTHONPATH (GH-150177)

* gh-149819: Fix .pth files not loaded in Python subprocesses

After PR gh-149583 (Fix double evaluation of .pth and .site files in
venvs), .pth files are no longer loaded in subprocesses started with
subprocess.run([sys.executable, ...]).  The root cause: main() seeds
known_paths from removeduppaths() with all sys.path entries inherited
from the parent process.  addsitedir() then skips .pth processing for
every directory already in known_paths.

Fix:
- main(): call removeduppaths() for dedup but start known_paths as a
  fresh empty set, so that addsitedir() processes .pth files in every
  site-packages directory regardless of inherited sys.path.
- addsitedir(): move known_paths.add() before the sys.path.append and
  guard the append with 'sitedir not in sys.path' to avoid creating
  duplicate entries when called with a fresh known_paths.

This preserves the gh-75723 dedup guarantee while allowing subprocesses
to load .pth files.

* Fill out the tests for GH#149888

* Extend _make_start() and _make_pth() to take an optional `basedir` which is used instead of
 `site.tmpdir` if given.
* Add test_pth_processed_when_sitedir_already_on_path() to test the core GH#149819 bug: .pth files
  in subprocesses aren't handled if PYTHONPATH pointing to the .pth directory is inherited.
* Similarly add test_start_processed_when_sitedir_already_on_path() to verify that .start files in
  the same circumstances are also now processed.

* Update Lib/site.py



* Oops!  Remove redundant code

---------
(cherry picked from commit 3c298e2)

Co-authored-by: Barry Warsaw <[email protected]>
Co-authored-by: BugBounty Mind <[email protected]>
Co-authored-by: scoder <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ncoghlan ncoghlan ncoghlan approved these changes
@FFY00 FFY00 Awaiting requested review from FFY00 FFY00 is a code owner
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti ezio-melotti is a code owner
@hugovk hugovk Awaiting requested review from hugovk hugovk is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
@webknjaz webknjaz Awaiting requested review from webknjaz webknjaz is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@warsaw warsaw

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@warsaw @scoder @ncoghlan @asottile @FFY00
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.