GeezerVison is showing github.com — PEP 829: resolve the design of the site dir batch processing API · Issue #150228 · python/cpython
PEP 829: resolve the design of the site dir batch processing API #150228DescriptionFeature or enhancement
Proposal:
(flagging as a feature, since it isn't really a bug report, just revisiting an API design detail for 3.15 where we had to make a design decision with less time for consideration than we might have wished)
The initial PEP 829 implementation relies on hidden global state in site.py to allow deferring site directory processing until several directories have been added:
known_paths = sys.path[:]
for dir_to_add in site_dirs_to_add:
site.addsitedir(dir_to_add, known_paths, defer_processing_start_files=True)
site.process_startup_files() # Defined, but not covered in the API docs yetThe hidden global state is a mixed blessing, as it means we can transparently make any site.addsitedir calls participate in a currently active batch (as was done to resolve #149504), but also makes the management and documentation of that hidden state a bit finicky.
With a slight tweak to the implementation of site._StartupState to also cover the known_paths functionality, the batch processing API could instead look like:
site_batch = site.StartupState(sys.path)
for dir_to_add in site_dirs_to_add:
site.addsitedir(dir_to_add, startup_state=site_batch)
site_batch.process()(passing both known_paths and startup_state would be an error)
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
The previous discussion was in the initial PEP 829 PR, where we settled for the boolean option in the public API because it was the simplest way to solve the problem before the beta 1 implementation deadline.
The fix for #149504 has subsequently done most of the work needed to enable switching the public API over to a state object rather than the boolean flag.
Linked PRs
Activityncoghlan commented on May 22, 2026 ContributorAuthor More actions @warsaw @hugovk I wanted to get this idea written up properly rather than leaving it languishing in the review comments on the initial implementation PR, but with sys.path itself already being inherently global state, I don't actually mind the current boolean flag + hidden state based implementation all that much.
If we decide to keep the existing design (which is certainly the simplest option), then this ticket would just cover adding the docs for site.process_startup_files(), including an example of how to use it for batched site directory processing. warsaw commented on May 22, 2026 I need to think about it a bit more, but on the face of it, I think it's a good idea. I'm also not super happy about the hidden state, so making that explicit is generally better. I'm also not necessarily opposed to trying to work this up for 3.15 if @hugovk approves.
Sadly, a lot of site.py just isn't documented, but that's also not license to break things. Case in point: addsitepackages() attempted removal 😄 Reacted by Alyssa Coghlan hugovk commented on May 22, 2026 Yes, if you can get it in by b2 (2026-06-02), and don't see any big risks when putting it together. ncoghlan commented on May 22, 2026 ContributorAuthor More actions
I'm also not super happy about the hidden state, so making that explicit is generally better.
Given #149504, I'm not sure we can get rid of the hidden state completely, as any revised public API still needs to handle the startup file processing attempting to make further site directory additions.
It's technically a gap in the semantic specification for PEP 829's deferred processing (previously, with the always eager processing, startup code could assume other startup files in the newly added site directory had been processed by the time addsitedir returned) ncoghlan commented on May 22, 2026 ContributorAuthor More actions
previously, with the always eager processing, startup code could assume other startup files in the newly added site directory had been processed by the time addsitedir returned
I realised this was an assumption that needed double checking, and determined that it isn't correct, but the relevant known_paths processing is a bit non-obvious.
The trick is that if known_paths is not set, it is populated from sys.path, and addsitedir adds the directory to sys.path before processing any startup files.
So I think we're fine to say startup file processing appends to the end of any site directory batch currently being processed.
We might need to tweak the definition of known_paths though, as it really doesn't work right if you don't initialise it from sys.path the way site does. ncoghlan commented on May 24, 2026 ContributorAuthor More actions Thinking about the batch processing prompted an experiment that revealed an unintentional backwards incompatible behaviour change.
The eager path processing in Python 3.14 inserts each site dir's path entries immediately after the containing site directory:
$ cat ~/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/lib/python3.14/site-packages/example.pth
/home/acoghlan/devel/_misc/added_from_sys_site
$ cat ~/.local/lib/python3.14/site-packages/example.pth
/home/acoghlan/devel/_misc/added_from_user_site
$ python3.14 -m site
sys.path = [
'/home/acoghlan/devel/_misc',
'/home/acoghlan/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/lib/python314.zip',
'/home/acoghlan/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/lib/python3.14',
'/home/acoghlan/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/lib/python3.14/lib-dynload',
'/home/acoghlan/.local/lib/python3.14/site-packages',
'/home/acoghlan/devel/_misc/added_from_user_site',
'/home/acoghlan/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/lib/python3.14/site-packages',
'/home/acoghlan/devel/_misc/added_from_sys_site',
]
USER_BASE: '/home/acoghlan/.local' (exists)
USER_SITE: '/home/acoghlan/.local/lib/python3.14/site-packages' (exists)
ENABLE_USER_SITE: True
The current Python 3.15 batch processing puts all of the added directories after the last site directory (because the paths passed directly to addsitedir are still processed eagerly, rather than being included in the batch processing):
$ cat ~/.local/share/uv/python/cpython-3.15.0b1-linux-x86_64-gnu/lib/python3.15/site-packages/example.pth
/home/acoghlan/devel/_misc/added_from_sys_site
$ cat ~/.local/lib/python3.15/site-packages/example.pth
/home/acoghlan/devel/_misc/added_from_user_site
$ python3.15 -m site
sys.path = [
'/home/acoghlan/devel/_misc',
'/home/acoghlan/.local/share/uv/python/cpython-3.15.0b1-linux-x86_64-gnu/lib/python315.zip',
'/home/acoghlan/.local/share/uv/python/cpython-3.15.0b1-linux-x86_64-gnu/lib/python3.15',
'/home/acoghlan/.local/share/uv/python/cpython-3.15.0b1-linux-x86_64-gnu/lib/python3.15/lib-dynload',
'/home/acoghlan/.local/lib/python3.15/site-packages',
'/home/acoghlan/.local/share/uv/python/cpython-3.15.0b1-linux-x86_64-gnu/lib/python3.15/site-packages',
'/home/acoghlan/devel/_misc/added_from_user_site',
'/home/acoghlan/devel/_misc/added_from_sys_site',
]
USER_BASE: '/home/acoghlan/.local' (exists)
USER_SITE: '/home/acoghlan/.local/lib/python3.15/site-packages' (exists)
ENABLE_USER_SITE: True
I didn't file a separate bug report for that, as I think it ties into the suggestion to make StartupState also cover the known_paths parameter's functionality: when processing a batch of site directories, the site directories themselves should be considered part of the batch, so they still get added in the same order as in 3.14 and earlier relative to their associated .pth entries.
There will still be one remaining discrepancy, which is that directories added with sys.path.append or site.addsitedir in startup code execution will still be moved to the end rather than being interleaved (since the code execution happens after the application of the static path modifications). Rather than trying to change that outcome, I think we can instead cover that detail with a porting note in the What's New (if the directory import priority change matters, the affected startup code is going to have to be updated to ensure the order in sys.path is adjusted as desired) warsaw commented on May 28, 2026 @ncoghlan Thanks again for the great engagement on this issue and for finding the interleaving regression. I think I've fixed the latter in the linked PR and made the whole API much more approachable and documented, while preserving the legacy behavior, hopefully all in backward compatible (to <= 3.14) ways. I think this also conforms ot @hugovk 's RM request to land before 3.15b2. I feel like with this we're in good shape for 3.15 final, and 🤞 we don't find any other regressions! Reacted by Hugo van Kemenade
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|