GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing github.com — 3.15 regression: concurrent first imports can fail or hang without any import cycle · Issue #159963 · python/cpython Closed Closed Copy link Labels stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-importlibtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error Description... because parents' module locks stay held while a submodule's body runs Bug report(bear with me this is verbose, yes mostly model written, i tried to make sure the important info was up top - and i have a proposed fix coming) -gps DescriptionSince gh-83065 (GH-137196, in 3.15.0b1 and later), This causes three failures that 3.14 and earlier don't have. None needs an import cycle, and in one thread the same imports work in either order:
Short reproducer (failure 1)import os, shutil, sys, tempfile, threading, time root = tempfile.mkdtemp() sys.path.insert(0, root) for path, text in { "a/__init__.py": "import time; time.sleep(0.2)", # any package whose __init__ takes a while "a/leaf.py": "from b.leaf import X", "b/__init__.py": "import a", "b/leaf.py": "X = 1", }.items(): os.makedirs(os.path.dirname(os.path.join(root, path)), exist_ok=True) with open(os.path.join(root, path), "w") as f: f.write(text + "\n") def thread_1(): import a.leaf def thread_2(): time.sleep(0.1) # start while a/__init__.py is running import b.leaf threads = [threading.Thread(target=thread_1), threading.Thread(target=thread_2)] for t in threads: t.start() for t in threads: t.join() shutil.rmtree(root) print("a.leaf imported:", "a.leaf" in sys.modules, "- b.leaf imported:", "b.leaf" in sys.modules) sys.exit("a.leaf" not in sys.modules or "b.leaf" not in sys.modules) The dependencies are
The cause shows without threads. With an empty import importlib._bootstrap as b ref = b._module_locks.get("w"); lock = ref() if ref else None print("lock of w held while w.main's body runs:", bool(lock and lock.locked()))
directional thoughts...
What happens, step by stepLine numbers are those of
Lock trace of reproducer 1Every On 3.14.x the line after How the three failures arise:
"Parent before child" orders the locks along one dotted name, but gives no order between two chains, ( This trade-off (ancestors' locks held through the leaf's body) isn't discussed in the class docstring (437-441), the comment at 462-466, the two new tests, the NEWS entry (3.15.0 beta 1: "so the two threads serialise instead of raising Reproducer 1: both errors, without sleepsStandard library only. It writes three or four small files to a temporary folder and first checks that both orders work in one thread. Exit code 1: an import failed; 0: none did. """Two threads make first imports in two packages. No import cycle, and either order works in one thread, yet thread 1 fails on 3.15 (passes on 3.11 to 3.14). python repro_error.py -> _DeadlockError python repro_error.py partial -> ImportError: cannot import name 'X' from partially initialized module 'b' """ import importlib._bootstrap as bootstrap import os, shutil, sys, tempfile, threading, time, traceback, types CASES = { # Needs: a.leaf -> b.leaf -> b -> a. "a" imports nothing, so there is no cycle. "deadlock": ("import a.leaf", "import b.leaf", { "a/__init__.py": "import sync; sync.slow_init()", "a/leaf.py": "from b.leaf import X", "b/__init__.py": "import a", "b/leaf.py": "X = 1"}), # Needs: a.leaf -> b -> a. No cycle either. "partial": ("import a.leaf", "import b", { "a/__init__.py": "import sync; sync.slow_init()", "a/leaf.py": "from b import X", "b.py": "import a; X = 1"}), } stmt1, stmt2, files = CASES[sys.argv[1] if len(sys.argv) > 1 else "deadlock"] sys.dont_write_bytecode = True root = tempfile.mkdtemp() sys.path.insert(0, root) for path, text in files.items(): os.makedirs(os.path.dirname(os.path.join(root, path)), exist_ok=True) with open(os.path.join(root, path), "w") as f: f.write(text + "\n") # "sync" is not a file: it lets a/__init__.py call back into this script. sync = sys.modules["sync"] = types.ModuleType("sync") # Control: one thread, both orders. sync.slow_init = lambda: None for order in ((stmt1, stmt2), (stmt2, stmt1)): for stmt in order: exec(stmt) for name in [n for n in sys.modules if n.split(".")[0] in ("a", "b")]: del sys.modules[name] print("one thread, both orders: ok") # Two threads. a/__init__.py stands for any package whose __init__ takes a while: # it returns once thread 2 has got as far as waiting for the module lock of "a". # No sleep decides the order; the private _blocking_on is only read, to see that moment. in_a = threading.Event() failed = [] def slow_init(): in_a.set() while "_ModuleLock('a')" not in repr(bootstrap._blocking_on.get(t2.ident)): time.sleep(0.001) sync.slow_init = slow_init def run(stmt, start_after=None): if start_after: start_after.wait() try: exec(stmt) except BaseException: failed.append(stmt) print(f"{threading.current_thread().name}, {stmt!r}:") traceback.print_exc(file=sys.stdout) t1 = threading.Thread(target=run, args=(stmt1,), name="thread 1", daemon=True) t2 = threading.Thread(target=run, args=(stmt2, in_a), name="thread 2", daemon=True) t2.start(); t1.start() t1.join(30); t2.join(30) if t1.is_alive() or t2.is_alive(): failed.append("still running after 30 s") shutil.rmtree(root, ignore_errors=True) print("two threads:", f"FAILED {failed}" if failed else "ok", "on", sys.version.split()[0]) sys.exit(1 if failed else 0) On 3.15.0 (on 3.14.7 both print The second message blames a circular import that doesn't exist. Reproducer 2: the wait and the hang"""Thread 1 makes the first import of w.main. Thread 2 imports w.other, and starts while w/__init__.py is still running. On 3.15 thread 2 then has to wait until the BODY of w.main has ended, though "w" was complete long before. Here that body waits for thread 2, so both would hang forever; no _DeadlockError is raised. The wait is cut off after 5 s so that the script ends. Passes on 3.11 to 3.14. """ import importlib._bootstrap as bootstrap import os, shutil, sys, tempfile, threading, time, traceback, types files = { "w/__init__.py": "import sync; sync.slow_init()", "w/main.py": "import sync; sync.wait_for_thread_2()", "w/other.py": "", } sys.dont_write_bytecode = True root = tempfile.mkdtemp() sys.path.insert(0, root) os.mkdir(os.path.join(root, "w")) for path, text in files.items(): with open(os.path.join(root, path), "w") as f: f.write(text + "\n") in_w = threading.Event() thread_2_done = threading.Event() hung = [] def slow_init(): # Stands for any __init__ that takes a while: returns once thread 2 waits for # the module lock of "w" (or is done: before gh-130094 it doesn't wait at all). in_w.set() while not (thread_2_done.is_set() or "_ModuleLock('w')" in repr(bootstrap._blocking_on.get(t2.ident))): time.sleep(0.001) def wait_for_thread_2(): w = sys.modules["w"] print("w.main's body runs; w.__spec__._initializing =", w.__spec__._initializing) if not thread_2_done.wait(5): hung.append(True) print("after 5 s thread 2 is still here:") traceback.print_stack(sys._current_frames()[t2.ident], limit=4, file=sys.stdout) # "sync" is not a file: it lets the modules call back into this script. sync = sys.modules["sync"] = types.ModuleType("sync") sync.slow_init, sync.wait_for_thread_2 = slow_init, wait_for_thread_2 def thread_1(): import w.main def thread_2(): in_w.wait() t0 = time.monotonic() import w.other thread_2_done.set() print(f"thread 2: 'import w.other' took {time.monotonic() - t0:.1f} s") t1 = threading.Thread(target=thread_1, daemon=True) t2 = threading.Thread(target=thread_2, daemon=True) t2.start(); t1.start() t1.join(30); t2.join(30) shutil.rmtree(root, ignore_errors=True) print("HANG (cut off by the script)" if hung else "ok", "on", sys.version.split()[0]) sys.exit(1 if hung else 0) On 3.15.0: On 3.14.7 (the first two lines can come in either order): The same happens if thread 2 says The plain form of this failure is the wait: 5 s here against 0.0 s, for a module whose parent is complete and which has nothing to do with CPython versions tested on:CPython main branch, 3.15 Operating systems tested on:Linux Linked PRsActivity
MetadataMetadataAssigneesLabelsstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-importlibtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error ProjectsMilestoneRelationshipsDevelopmentIssue actions |
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |