GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-101758: Clean Up Uses of Import State by ericsnowcurrently · Pull Request #101919 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented Feb 14, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This change is almost entirely moving code around and hiding import state behind internal API. We introduce no changes to behavior, nor to non-internal API. (Since there was already going to be a lot of churn, I took this as an opportunity to re-organize import.c into topically-grouped sections of code.) The motivation is to simplify a number of upcoming changes.

Specific changes:

  • move existing import-related code to import.c, wherever possible
  • add internal API for interacting with import state (both global and per-interpreter)
  • use only API outside of import.c (to limit churn there when changing the location, etc.)
  • consolidate the import-related state of PyInterpreterState into a single struct field (this changes layout slightly)
  • add macros for import state in import.c (to simplify changing the location)
  • group code in import.c into sections
  • remove _PyState_AddModule()1

I avoided some related changes that would slightly change behavior, like the order of interpreter/runtime init. I may pursue those separately later.

  • Issue: Isolate PyModuleDef to Each Interpreter for Extension/Builtin Modules #101758

Footnotes

  1. I've also removed _PyState_AddModule() from the stable ABI manifest. It was part of "public" API since PEP 3121 was implemented 15 years ago (before a stable ABI existed) but was removed from limited API 6 years ago and then moved to internal API 4 years ago. Presumably it inadvertently slipped into the stable ABI manifest (with PEP 652) due to PC/python3dll.c. ↩

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ericsnowcurrently added 19 commits February 14, 2023 14:31
@ericsnowcurrently
Group code in import.c.
4433124
@ericsnowcurrently
Move PyState_*() to import.c.
7626bab
@ericsnowcurrently
Add _PyImport_GetNextModuleIndex().
5836e93
@ericsnowcurrently
Add _PyImport_SwapPackageContext() and _PyImport_ResolveNameWithPacka…
bc031c1
…geContext().
@ericsnowcurrently
Add _PyImport_GetBuiltinModuleNames().
f67c299
@ericsnowcurrently
Add an "extension modules" section.
9b312e1
@ericsnowcurrently
Move the "extension modules" section down.
24ec4de
@ericsnowcurrently
Add _PyImport_GetDLOpenFlags() and _PyImport_SetDLOpenFlags().
b2043e4
@ericsnowcurrently
Hide sys.modules.
43d8de7
@ericsnowcurrently
_PyInterpreterState_ClearModules() -> _PyImport_ClearModulesByIndex().
b7cabeb
@ericsnowcurrently
Hide interp->import_func.
d4693ea
@ericsnowcurrently
Hide interp->importlib.
7120be2
@ericsnowcurrently
Add _PyImport_InitCore() and _PyImport_InitExternal().
1935ee5
@ericsnowcurrently
Re-order import.c sections.
7e29b7c
@ericsnowcurrently
Add _PyImport_FiniExternal() and _PyImport_FiniCore().
66de41d
@ericsnowcurrently
Factor out init_builtin_modules_table() and fini_builtin_builtins_tab…
384bd69
…le().
@ericsnowcurrently
Add PyInterpreterState.imports.
e4777b7
@ericsnowcurrently
Move the lifecycle sections to the bottom.
48fb47c
@ericsnowcurrently
Add macros for runtime-global import state.
e8930c8
@ericsnowcurrently ericsnowcurrently added the skip news label Feb 14, 2023
@ericsnowcurrently
ericsnowcurrently requested review from brettcannon, encukou, markshannon, ncoghlan and warsaw as code owners February 14, 2023 22:19
@bedevere-bot bedevere-bot added the awaiting core review label Feb 14, 2023
@bedevere-bot bedevere-bot mentioned this pull request Feb 14, 2023
Isolate PyModuleDef to Each Interpreter for Extension/Builtin Modules #101758
Closed
@ericsnowcurrently
ericsnowcurrently removed the request for review from markshannon February 14, 2023 22:19
@ericsnowcurrently

ericsnowcurrently commented Feb 14, 2023

Copy link
Copy Markdown
Member Author

@brettcannon, this is what I was talking about. 😄

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently
minor fixes
79a72c4
@erlend-aasland erlend-aasland mentioned this pull request Feb 14, 2023
gh-101819: Remove _PyWindowsConsoleIO_Type from the Windows DLL #101904
Merged
@ericsnowcurrently
Drop _PyState_AddModule().
0527913
@ericsnowcurrently
ericsnowcurrently requested a review from a team as a code owner February 15, 2023 18:29
@ericsnowcurrently
ericsnowcurrently removed the request for review from a team February 15, 2023 18:30
@ericsnowcurrently
Drop a wrong assert.
a4deb8a
@ericsnowcurrently
ericsnowcurrently merged commit b2fc549 into python:main Feb 15, 2023
@bedevere-bot bedevere-bot removed the awaiting core review label Feb 15, 2023
@ericsnowcurrently
ericsnowcurrently deleted the hide-import-state branch February 15, 2023 22:32
@bedevere-bot

bedevere-bot commented Feb 15, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 LTO 3.x has failed when building commit b2fc549.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/567/builds/3626) and take a look at the build logs.
  4. Check if the failure is related to this commit (b2fc549) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/567/builds/3626

Failed tests:

  • test_asyncio

Failed subtests:

  • test_subprocess_consistent_callbacks - test.test_asyncio.test_subprocess.SubprocessThreadedWatcherTests.test_subprocess_consistent_callbacks

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

415 tests OK.

10 slowest tests:

  • test_tools: 3 min 16 sec
  • test_gdb: 2 min 48 sec
  • test_concurrent_futures: 2 min 31 sec
  • test_multiprocessing_spawn: 1 min 38 sec
  • test_multiprocessing_fork: 1 min 27 sec
  • test_signal: 1 min 14 sec
  • test_multiprocessing_forkserver: 1 min 13 sec
  • test_asyncio: 1 min 4 sec
  • test_logging: 44.6 sec
  • test_math: 41.1 sec

1 test failed:
test_asyncio

18 tests skipped:
test_check_c_globals test_devpoll test_ioctl test_kqueue
test_launcher test_msilib test_nis test_peg_generator
test_perf_profiler test_startfile test_tix test_tkinter test_ttk
test_winconsoleio test_winreg test_winsound test_wmi
test_zipfile64

1 re-run test:
test_asyncio

Total duration: 5 min 7 sec

Click to see traceback logs
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/test/test_asyncio/test_subprocess.py", line 771, in test_subprocess_consistent_callbacks
    self.loop.run_until_complete(main())
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/test/test_asyncio/test_subprocess.py", line 763, in main
    self.assertEqual(events, [
AssertionError: Lists differ: [('pi[69 chars]), 'process_exited', 'pipe_connection_lost', '[17 chars]ost'] != [('pi[69 chars]), 'pipe_connection_lost', 'pipe_connection_lo[17 chars]ted']


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/test/test_asyncio/test_subprocess.py", line 771, in test_subprocess_consistent_callbacks
    self.loop.run_until_complete(main())
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/test/test_asyncio/test_subprocess.py", line 763, in main
    self.assertEqual(events, [
AssertionError: Lists differ: ['process_exited', ('pipe_data_received', 1, b'stdout')] != [('pipe_data_received', 1, b'stdout'), ('p[95 chars]ted']
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Feb 15, 2023

Copy link
Copy Markdown
Member Author

I'm fairly sure that buildbot failure is unrelated to this PR. If the buildbot fails the next commit then I'll look into this failure further.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@markshannon

markshannon commented Feb 16, 2023

Copy link
Copy Markdown
Member

OOI, did you benchmark this? Startup times are sensitive to the changes in the import system.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brettcannon brettcannon Awaiting requested review from brettcannon brettcannon is a code owner
@encukou encukou Awaiting requested review from encukou encukou is a code owner
@ncoghlan ncoghlan Awaiting requested review from ncoghlan ncoghlan is a code owner
@warsaw warsaw Awaiting requested review from warsaw warsaw is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@ericsnowcurrently @bedevere-bot @markshannon
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.