GeezerVison is showing github.com — __import__ is not thread-safe on Python 3 · Issue #83065 · python/cpython
__import__ is not thread-safe on Python 3 #83065Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-importlibtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionValentynTymofieiev mannequin
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-11-21.19:11:54.536>
labels = ['3.7', '3.8', '3.9']
title = '__import__ is not thread-safe on Python 3'
updated_at = <Date 2021-12-08.22:53:27.704>
user = 'https://bugs.python.org/ValentynTymofieiev' bugs.python.org fields:
activity = <Date 2021-12-08.22:53:27.704>
actor = 'Valentyn Tymofieiev'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2019-11-21.19:11:54.536>
creator = 'Valentyn Tymofieiev'
dependencies = []
files = ['48734', '48735', '48737']
hgrepos = []
issue_num = 38884
keywords = []
message_count = 14.0
messages = ['357198', '357199', '357202', '357214', '357227', '357228', '357230', '357234', '357464', '360068', '360069', '362746', '375958', '408052']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'gregory.p.smith', 'gjb1002', 'ncoghlan', 'eric.snow', 'ryan.petrello', 'Valentyn Tymofieiev', 'Martin Laus', 'repalviglator']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue38884'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9'] Linked PRs
Reacted by Daniel Shokouhi, J. Nick Koston, Antonio G. De Crisci, Jorim Tielemans, yousaf465, TheJulianJES, elmurato, Jan-Philipp Benecke, Ståle Storø Hauknes and Michael Berdyshev Reacted by J. Nick Koston and yousaf465 ActivityValentynTymofieiev commented on Nov 21, 2019 ValentynTymofieievmannequin MannequinAuthor More actions Attached import_module_not_found.py consistently fails for me on Python 3.7.5 and earlier Python 3 versions that I have tried with
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "import_module_not_found.py", line 8, in t2
return __import__("tensorflow.estimator", level=0)
ModuleNotFoundError: No module named 'tensorflow.estimator
Threads in this example finish successfully if executed sequentially.
I have not tried higher versions of Python, but I cannot reproduce this on Python 2.7.
Is this an expected behavior?
Thank you. Reacted by Antonio G. De Crisci and Martijn van der Pol ValentynTymofieiev commented on Nov 21, 2019 ValentynTymofieievmannequin MannequinAuthor More actions Attached import_deadlock.py fails with
...
File "<frozen importlib._bootstrap>", line 980, in _find_and_load
File "<frozen importlib._bootstrap>", line 149, in __enter__
File "<frozen importlib._bootstrap>", line 94, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('tensorflow_transform.tf_metadata.metadata_io')
under the same conditions. ValentynTymofieiev commented on Nov 21, 2019 ValentynTymofieievmannequin MannequinAuthor More actions brettcannon commented on Nov 21, 2019 Can you test this with a package that isn't tensorflow (e.g. something else in the stdlib or something created manually; basically a smaller reproducer)? It's hard to diagnose if this is really Python or not without a smaller reproducer as tensorflow might be doing some things behind the scenes that would break this due to it being an extension module. ValentynTymofieiev commented on Nov 21, 2019 ValentynTymofieievmannequin MannequinAuthor More actions Attaching a minimal repro with simple project hierarchy. ValentynTymofieiev commented on Nov 21, 2019 ValentynTymofieievmannequin MannequinAuthor More actions Repro: unpack bpo-38884.zip
python input_deadlock.py
(fails on Python 3.7, but not on 2.7). ValentynTymofieiev commented on Nov 21, 2019 ValentynTymofieievmannequin MannequinAuthor More actions input_deadlock.py in bpo-38884.zip has a left-over comment:
# Requires pip install tensorflow==2.0.0 tensorflow-transform==0.15.0
Please ignore that. ValentynTymofieiev commented on Nov 21, 2019 ValentynTymofieievmannequin MannequinAuthor More actions The behavior changes between Python 3.2 (no deadlock) and Python 3.3 (deadlock). Deadlock also reproducible on Python 3.8. ValentynTymofieiev commented on Nov 25, 2019 ValentynTymofieievmannequin MannequinAuthor More actions @brett.cannon, looks like I inadvertently dropped you from nosy list, so in case you missed the updates - please take a look at bpo-38884.zip. Thank you. ValentynTymofieiev commented on Jan 15, 2020 ValentynTymofieievmannequin MannequinAuthor More actions Just checking - is bpo-38884.zip an acceptable repro for this issue?
Thanks. brettcannon commented on Jan 15, 2020 @Valentyn just assume it's good enough unless some asks otherwise. People are probably too busy right now to be looking at this and so you might be waiting a while for a more definitive answer. ryanpetrello commented on Feb 26, 2020 emmatyping commented on Jul 29, 2025 The case as I understand it is that T1 imports package.subpackage, which in it's __init__.py imports package.subpackage.module
Simultaneously, T2 is trying to import package.subpackage.module, which should involve importing package.subpackage?
Then you end up with a deadlock where T1 needs to resolve package.subpackage.module to resolve package.subpackage, while T2 needs to resolve package.subpackage to resolve package.subpackage.module, so we end up with a deadlock where they are holding mutually referential locks.
Here's the traceback on main:
Exception in thread Thread-1 (t1):
Traceback (most recent call last):
File "/home/emma/cpython/Lib/threading.py", line 1074, in _bootstrap_inner
self._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/emma/cpython/Lib/threading.py", line 1016, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/emma/test_import_race/import_deadlock.py", line 5, in t1
return __import__("package.subpackage", level=0)
File "/home/emma/test_import_race/package/__init__.py", line 1, in <module>
from package import subpackage
File "/home/emma/test_import_race/package/subpackage/__init__.py", line 1, in <module>
from package.subpackage.module import *
File "<frozen importlib._bootstrap>", line 1368, in _find_and_load
File "<frozen importlib._bootstrap>", line 421, in __enter__
File "<frozen importlib._bootstrap>", line 346, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('package.subpackage.module') at 138200313720016
I'm not really sure what can be done to solve this other than some kind of earlier check when resolving package.subpackage.module if another thread is resolving a parent (in this case package.subpackage) and may need to resolve package.subpackage.module? Then block that thread until the other thread is done. gpshead commented on Jul 29, 2025 The blunt force workaround for import madness of this form regardless of cause has always been: Do your imports from the main thread. Not super friendly, and extra annoying when it's something nobody likes to think about such as unplanned indirect imports such as those from codecs dynamically loading infrequently used codec implementations, possibly due to untrusted data, in a serving stack.
Out of curiosity, I just had Claude Opus 4 take a stab at this as Emma's analysis sounded right. Draft PR from that attached. If that is sensible (I'll give it more of a look later), I think real performance tests are needed. But hierarchical lock acquisition was also going to be my suggestion w/o having re-read importlib guts as well, so it maybe served as a bit of validation of both the issue & analysis and that thought.
Claude code prompts (/model is opus 4):
- load github issue 83065 and analyze the workings of Lib/importlib/ to understand the situation. focus on emmatyping's issue comment, is that accurate given how the importlib code works? what could be done to avoid this?
- Plan the import system improvements
- before benchmarking, look over the diff so far with the eye of a code reviewer and simplify it. also remove the _use_hierarchical_import_locks logic, we'll always want this enabled. -- (this helped the code quality a lot!)
- run the reindent tool -- (it knows how via my CLAUDE.local.md)
- commmit your work, credit Emma for the analysis in the commit message.
- be sure to put markdown backticks around
__init__.py
- push to the gpshead remote and create a Draft PR, prefix the title of the PR with "__import__ is not thread-safe on Python 3 #83065: "
half an hour of API use. this cost $$ money (stated just to let others know the scale). added a commit that references this issue on Apr 28, 2026GH-83065: Fix import deadlock by implementing hierarchical module loc… gpshead commented on Apr 28, 2026 thanks all for earlier reviews! i found some remaining nits and fixed them, polished up, documented and merged. if we uncover problems with this during 3.15 betas - reverting to the prior status quo can happen. so many things in the world have worked around this long-standing flaw in odd manners. it'll be nice to stop encountering those. bdraco commented on Apr 28, 2026 Thanks for fixing this. I have personally made a donation to PSF in your honor. Reacted by Gregory P. Smith, Emma Smith and Jacob Coffee gpshead commented on May 8, 2026 steve-s commented on Sep 2, 2026 FTR there seem to be also a rare race in stdlib with collections/collections.abc fixed by the linked PR, and causing ModuleNotFoundError: No module named 'collections.abc'
Thread A:
import collections
-> _find_and_load("collections")
-> execute collections/__init__.py
-> stop between these lines:
```
# collections/__init__.py:29
import _collections_abc
import sys
# INTERRUPTED HERE
_sys.modules['collections.abc'] = _collections_abc
abc = _collections_abc
```
-> "collections" is already in sys.modules
-> although spec._initializing is still True
Thread B:
import collections.abc
-> _find_and_load("collections.abc")
-> parent "collections" already exists in sys.modules
-> do not wait for the parent package to finish (no check of _initializing on the spec)
-> search collections.__path__ for an abc submodule
-> no collections/abc.py so no finder result
-> ModuleNotFoundError at _bootstrap.py:1329
self-contained reproducer fails for me locally on 3.13 and passes on 3.15 dev build gpshead commented on Oct 13, 2026 FYI - The fix we landed in 3.15 is not complete and I & claude have discovered problems with it. I'll open a new issue with details and a proposed fix soon. Reacted by TheJulianJES Reacted by Emma Smith added 2 commits that reference this issue on Oct 25, 2026gh-159963: Don't hold parent package import locks while a submodule e…
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-importlibtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error StatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|