GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-83065: Fix import deadlock by implementing hierarchical module locking by gpshead · Pull Request #137196 · python/cpython

Conversation

@gpshead

@gpshead gpshead commented Jul 29, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

When one thread imports pkg.sub.mod and another thread imports pkg.sub
and pkg/sub/__init__.py itself imports pkg.sub.mod, the two threads can
acquire the per-module locks in opposite orders and trip the import system's
deadlock detector, raising _DeadlockError to user code.

This change makes _find_and_load() acquire the module locks for the full
dotted-name chain (parent before child) when loading a nested module, so
both threads contend on the same first lock and serialise instead of
deadlocking.

When acquiring a parent's lock would itself deadlock with another thread
that is loading that parent (cross-package circular imports), the parent's
lock is skipped and the partially-initialised parent is accepted -- the
same policy _lock_unlock_module() already applies on the existing code
path -- so concurrent circular imports that worked before continue to work.

A second new test covers that cross-package case.

The hot path (the module is already fully loaded) does not reach the new
locking code at all.

Fixes #83065

  • Issue: __import__ is not thread-safe on Python 3 #83065

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bdraco reacted with hooray emoji bdraco reacted with heart emoji bdraco reacted with rocket emoji
All reactions
  • 🎉 1 reaction
  • ❤️ 1 reaction
  • 🚀 1 reaction
@gpshead @claude
pythongh-83065: Fix import deadlock by implementing hierarchical modu…
78d4373
…le locking

Fix a deadlock in the import system that occurs when concurrent threads
import modules at different levels of the same package hierarchy while
`__init__.py` files have circular imports.

The deadlock scenario (correctly analyzed by @emmatyping):
- Thread 1 imports `package.subpackage`, which imports `package.subpackage.module`
  in its `__init__.py`
- Thread 2 imports `package.subpackage.module`, which needs to ensure
  `package.subpackage` is loaded first
- Each thread holds a lock the other needs, causing deadlock

The fix introduces _HierarchicalLockManager that acquires all necessary
module locks in a consistent order (parent before child) for nested modules.
This ensures all threads acquire locks in the same order, preventing
circular wait conditions.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@bedevere-app bedevere-app Bot mentioned this pull request Jul 29, 2025
__import__ is not thread-safe on Python 3 #83065
Closed
gpshead
gpshead commented Jul 29, 2025
View reviewed changes
Comment thread Lib/importlib/_bootstrap.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/importlib/_bootstrap.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@gpshead gpshead self-assigned this Jul 29, 2025
@emmatyping

emmatyping commented Jul 30, 2025

Copy link
Copy Markdown
Member

I don't feel qualified to say this PR is correct, as import logic has a lot of nasty reentrant behavior I haven't dove into, but the PR does look right, based on what I imagined an implementation of a solution might be.

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ilyapoz

ilyapoz commented Jan 7, 2026

Copy link
Copy Markdown

Maybe we could get authors of original implementation to help review this? @pitrou @exarkun ?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions

github-actions Bot commented Apr 27, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 27, 2026
gpshead added 5 commits April 27, 2026 18:33
@gpshead
Merge upstream/main into gps/import-lock-heirarchy-gh-83065
22c3d9e
Resolved conflict in Lib/test/test_importlib/test_threaded_import.py:
both branches added a new test method at the same location. Kept both
test_import_failure_race_condition (from upstream) and
test_hierarchical_import_deadlock (from this PR).
@gpshead
_HierarchicalLockManager: release locks on __enter__ failure; accept …
4784023
…partial parent on cycle

If lock.acquire() raises (e.g. _DeadlockError or KeyboardInterrupt) after
some chain locks are already held, __exit__ is never called and the held
locks leak permanently. Wrap the loop so any exception releases what was
acquired before re-raising.

When acquiring a parent's lock would deadlock with another thread that is
loading that parent, skip the parent and proceed - the same accept-a-
partial-parent policy that _lock_unlock_module() applies. Without this,
concurrent cross-package circular imports that previously succeeded via
that path now raise _DeadlockError to user code. The leaf's lock is still
required; a deadlock there propagates as before.

Also lift _get_module_chain to a module-level function as it does not use
self.
@gpshead
test_threaded_import: tighten hierarchical test; add cross-package ci…
fd9da14
…rcular test

test_hierarchical_import_deadlock: synchronise thread start with a
Barrier; assert errors == [] (the prior isinstance(error, ImportError)
check never matched _DeadlockError, a RuntimeError); record exception
type and message rather than the bare object so failures self-describe.

test_cross_package_circular_import: new. Two packages whose __init__.py
each import a submodule of the other; concurrent imports of further
submodules must not raise _DeadlockError.
@gpshead
Add NEWS entry and What's New for the pythongh-83065 import deadlock fix
12eb777
@gpshead
Drop comments that restate adjacent code
b3d4028
@gpshead gpshead removed the stale Stale PR or inactive for long period of time. label Apr 28, 2026
@gpshead
gpshead marked this pull request as ready for review April 28, 2026 06:16
@gpshead
gpshead requested review from AA-Turner, FFY00, brettcannon, ericsnowcurrently, ncoghlan and warsaw as code owners April 28, 2026 06:16
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 28, 2026
@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 28, 2026
@bedevere-bot

bedevere-bot commented Apr 28, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit b3d4028 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F137196%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 28, 2026
@gpshead
gpshead merged commit 9a57179 into python:main Apr 28, 2026
104 of 113 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting core review label Apr 28, 2026
@YangAn-microsoft YangAn-microsoft mentioned this pull request May 21, 2026
{Core} Restore sequential module loading Azure/azure-cli#33250
Merged
3 tasks
@duckboycool duckboycool mentioned this pull request Oct 12, 2026
Bug: Launcher fails to load worlds on python 3.13.16 ArchipelagoMW/Archipelago#6506
Open
@gpshead gpshead added the needs backport to 3.14 bugs and security fixes label Oct 13, 2026
@miss-islington-app

miss-islington-app Bot commented Oct 13, 2026

Copy link
Copy Markdown

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington-app

miss-islington-app Bot commented Oct 13, 2026

Copy link
Copy Markdown

Sorry, @gpshead, I could not cleanly backport this to 3.14 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker 9a57179d74c1a20e3188779696c60c8dd812e6fb 3.14
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead

gpshead commented Oct 13, 2026

Copy link
Copy Markdown
Member Author

okay so it'll require more effort to backport.. had to try; this is a large change. not sure we'd actually want to put it in a patch release.

emmatyping reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead gpshead mentioned this pull request Oct 20, 2026
3.15 regression: concurrent first imports can fail or hang without any import cycle #159963
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@emmatyping emmatyping emmatyping left review comments
@brettcannon brettcannon Awaiting requested review from brettcannon brettcannon is a code owner
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@ncoghlan ncoghlan Awaiting requested review from ncoghlan ncoghlan is a code owner
@warsaw warsaw Awaiting requested review from warsaw warsaw is a code owner
@FFY00 FFY00 Awaiting requested review from FFY00 FFY00 is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

needs backport to 3.14 bugs and security fixes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

__import__ is not thread-safe on Python 3

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@gpshead @emmatyping @ilyapoz @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.