GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows · Issue #107219 · python/cpython

test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219

New issue

Copy link

New issue

Copy link

Closed

test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows#107219

Copy link

Labels

OS-windowstestsTests in the Lib/test dirTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

vstinner

opened

on Jul 25, 2023

Issue body actions

GHA Windows x86 job, test_crash_big_data() hangs on ProcessPoolExecutor.shutdown(): https://github.com/python/cpython/actions/runs/5651960914/job/15310873235?pr=107217

Main thread: ProcessPoolExecutor.shutdown()
Thread 2: Threading.join()
Thread 3: queue _feed() => connection send_bytes()

(...)
0:39:52 load avg: 0.06 running: test_concurrent_futures (19 min 2 sec)
0:40:22 load avg: 0.05 running: test_concurrent_futures (19 min 32 sec)
0:40:51 load avg: 0.03 [447/447/2] test_concurrent_futures crashed (Exit code 1)
Timeout (0:20:00)!
Thread 0x000007e8 (most recent call first):
  File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 282 in _send_bytes
  File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 199 in send_bytes
  File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 246 in _feed
  (...)
  File "D:\a\cpython\cpython\Lib\threading.py", line 1009 in _bootstrap

Thread 0x00001738 (most recent call first):
  File "D:\a\cpython\cpython\Lib\threading.py", line 1146 in _wait_for_tstate_lock
  File "D:\a\cpython\cpython\Lib\threading.py", line 1126 in join
  (...)
  File "D:\a\cpython\cpython\Lib\threading.py", line 1009 in _bootstrap

Thread 0x0000103c (most recent call first):
  File "D:\a\cpython\cpython\Lib\threading.py", line 1146 in _wait_for_tstate_lock
  File "D:\a\cpython\cpython\Lib\threading.py", line 1126 in join
  File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 836 in shutdown
  File "D:\a\cpython\cpython\Lib\concurrent\futures\_base.py", line 647 in __exit__
  File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures.py", line 1386 in test_crash_big_data
  (...)
  File "D:\a\cpython\cpython\Lib\test\support\__init__.py", line 1241 in run_unittest
  File "D:\a\cpython\cpython\Lib\test\libregrtest\runtest.py", line 294 in _test_module
  (...)

Linked PRs

Activity

vstinner
added
type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
on Jul 25, 2023
vstinner commented on Jul 25, 2023
vstinner
on Jul 25, 2023
MemberAuthor
More actions
By the way, test.regrtest doesn't work with test_concurrent_futures: when test_concurrent_futures is re-run in verbose mode, no tests is ran!

0:40:52 Re-running test_concurrent_futures in verbose mode ---------------------------------------------------------------------- Ran 0 tests in 0.001s NO TESTS RAN

This second bug maybe hides the first bug (test_concurrent_futures hangs sometimes).

vstinner commented on Jul 25, 2023

vstinner

on Jul 25, 2023

MemberAuthor

More actions

Windows x64 job, also blocked on test_crash_big_data(): https://github.com/python/cpython/actions/runs/5652189009/job/15311447694

Similar threads state.

0:22:04 load avg: 5.60 [447/447/2] test_concurrent_futures crashed (Exit code 1)
Timeout (0:20:00)!
Thread 0x00001be0 (most recent call first):
  File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 282 in _send_bytes
  File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 199 in send_bytes
  File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 246 in _feed
  (...)
  File "D:\a\cpython\cpython\Lib\threading.py", line 1009 in _bootstrap

Thread 0x000002a0 (most recent call first):
  File "D:\a\cpython\cpython\Lib\threading.py", line 1146 in _wait_for_tstate_lock
  File "D:\a\cpython\cpython\Lib\threading.py", line 1126 in join
  (...)
  File "D:\a\cpython\cpython\Lib\threading.py", line 1009 in _bootstrap

Thread 0x00000c18 (most recent call first):
  File "D:\a\cpython\cpython\Lib\threading.py", line 1146 in _wait_for_tstate_lock
  File "D:\a\cpython\cpython\Lib\threading.py", line 1126 in join
  File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 836 in shutdown
  File "D:\a\cpython\cpython\Lib\concurrent\futures\_base.py", line 647 in __exit__
  File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures.py", line 1386 in test_crash_big_data
  (...)

vstinner commented on Jul 25, 2023

vstinner

on Jul 25, 2023

MemberAuthor

More actions

Azure Pipelines: Windows PR Tests win32 hangs on test_interpreter_shutdown(): https://dev.azure.com/Python/cpython/_build/results?buildId=133201&view=logs&j=d554cd63-f8f4-5b2d-871b-33e4ea76e915&t=5a14d0eb-dbd4-5b80-f5d0-7909f950a1cc

Main thread: run_python_until_end() => subprocess.Popen.communicate()
Thread 2 (stdout?): subprocess _readerthread()
Thread 3 (stderr?): subprocess _readerthread()

In short, the main thread is waiting until the process completes and the process is killed after 20 minutes.

On the same CI, the win64 job ran test_concurrent_futures in 1 min 33 sec.

(...)
test_hang_gh94440 (test.test_concurrent_futures.ProcessPoolSpawnProcessPoolShutdownTest.test_hang_gh94440)
shutdown(wait=True) doesn't hang when a future was submitted and ... skipped 'Tested platform does not support the alarm signal'
test_hang_issue12364 (test.test_concurrent_futures.ProcessPoolSpawnProcessPoolShutdownTest.test_hang_issue12364) ... ok

Timeout (0:20:00)!
Thread 0x00001184 (most recent call first):
  File "D:\a\1\s\Lib\subprocess.py", line 1597 in _readerthread
  File "D:\a\1\s\Lib\threading.py", line 989 in run
  File "D:\a\1\s\Lib\threading.py", line 1052 in _bootstrap_inner
  File "D:\a\1\s\Lib\threading.py", line 1009 in _bootstrap

Thread 0x00001224 (most recent call first):
  File "D:\a\1\s\Lib\subprocess.py", line 1597 in _readerthread
  File "D:\a\1\s\Lib\threading.py", line 989 in run
  File "D:\a\1\s\Lib\threading.py", line 1052 in _bootstrap_inner
  File "D:\a\1\s\Lib\threading.py", line 1009 in _bootstrap

Thread 0x00001560 (most recent call first):
  File "D:\a\1\s\Lib\threading.py", line 1146 in _wait_for_tstate_lock
  File "D:\a\1\s\Lib\threading.py", line 1126 in join
  File "D:\a\1\s\Lib\subprocess.py", line 1626 in _communicate
  File "D:\a\1\s\Lib\subprocess.py", line 1209 in communicate
  File "D:\a\1\s\Lib\test\support\script_helper.py", line 139 in run_python_until_end
  File "D:\a\1\s\Lib\test\support\script_helper.py", line 149 in _assert_python
  File "D:\a\1\s\Lib\test\support\script_helper.py", line 166 in assert_python_ok
  File "D:\a\1\s\Lib\test\test_concurrent_futures.py", line 302 in test_interpreter_shutdown
  (...)
  File "<frozen runpy>", line 198 in _run_module_as_main

AlexWaygood commented on Jul 26, 2023
AlexWaygood
on Jul 26, 2023
Member
More actions
I reported a similar issue in May 2022, but closed it as it seemed the issue stopped occurring in CI:

test_concurrent_futures sometimes hangs indefinitely on Windows #92222
AlexWaygood
added
testsTests in the Lib/test dirTests in the Lib/test dir

OS-windows
on Jul 26, 2023

Eclips4 commented on Aug 24, 2023

Eclips4

on Aug 24, 2023

Member

More actions

./python -m test -v test_concurrent_futures -m test_crash_big_data --forever get me this:

many lines..
0:00:05 [  9] test_concurrent_futures
test_crash_big_data (test.test_concurrent_futures.ProcessPoolForkExecutorDeadlockTest.test_crash_big_data) ... skipped 'require un
ix system'
test_crash_big_data (test.test_concurrent_futures.ProcessPoolForkserverExecutorDeadlockTest.test_crash_big_data) ... skipped 'requ
ire unix system'
test_crash_big_data (test.test_concurrent_futures.ProcessPoolSpawnExecutorDeadlockTest.test_crash_big_data) ... Warning -- Uncaugh
t thread exception: InvalidStateError
Exception in thread Thread-9:
Traceback (most recent call last):
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\threading.py", line 1059, in _bootstrap_inner
    self.run()
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 344, in run
    self.terminate_broken(cause)
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 492, in terminate_broken
    work_item.future.set_exception(bpe)
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\_base.py", line 559, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2cf561cf260 state=cancelled>
0.55s Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
Warning -- Dangling thread: <_MainThread(MainThread, started 11880)>
Warning -- Dangling thread: <Thread(QueueFeederThread, started daemon 4112)>
ok
Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
Warning -- Dangling thread: <_MainThread(MainThread, started 11880)>
Warning -- Dangling thread: <Thread(QueueFeederThread, started daemon 4112)>

Sadly, but it's hard to reproduce.

vstinner commented on Aug 25, 2023
vstinner
on Aug 25, 2023
MemberAuthor
More actions

Sadly, but it's hard to reproduce.

You can stress the system to make the issue more likely. For example, open a second terminal and run:

python -m test -j2

You can use -j4 or more depending on the number of CPUs and how much you want your machine to be stressed :-)
Eclips4 commented on Aug 25, 2023
Eclips4
on Aug 25, 2023
Member
More actions

Sadly, but it's hard to reproduce.

You can stress the system to make the issue more likely. For example, open a second terminal and run:

python -m test -j2

You can use -j4 or more depending on the number of CPUs and how much you want your machine to be stressed :-)

Oh, that's right! With -j8 (in a separate terminal) I can reproduce bug more easily.
lazka commented on Aug 25, 2023
lazka
on Aug 25, 2023
Contributor
More actions
(I'm also seeing this hang in the mingw fork after updating from 3.11.4 to 3.11.5)

vstinner commented on Aug 26, 2023

vstinner

on Aug 26, 2023

MemberAuthor

More actions

Error on Linux, not sure if it's related.

aarch64 Fedora Stable LTO + PGO 3.x buildbot: https://buildbot.python.org/all/#/builders/524/builds/4310

Log (reformatted for readability):

FAIL: test_interpreter_shutdown (test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)

AssertionError: b'Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/threading.py", line 1059, in _bootstrap_inner
    self.run()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/concurrent/futures/process.py", line 339, in run
    self.add_call_item_to_queue()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/concurrent/futures/process.py", line 394, in add_call_item_to_queue
    self.call_queue.put(_CallItem(work_id,
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/queues.py", line 94, in put
    self._start_thread()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/queues.py", line 177, in _start_thread
    self._thread.start()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/threading.py", line 978, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can\'t create new thread at interpreter shutdown
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 274, in main
    code = _serve_one(child_r, fds,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 313, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
' is not false

vstinner commented on Aug 26, 2023

vstinner

on Aug 26, 2023

MemberAuthor

More actions

Similar error on another Linux machine.

ARM Raspbian 3.x: https://buildbot.python.org/all/#/builders/424/builds/4736

Logs (reformatted):

FAIL: test_interpreter_shutdown (test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)
AssertionError: b'Exception in thread Thread-1:
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 1059, in _bootstrap_inner
    self.run()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/concurrent/futures/process.py", line 339, in run
    self.add_call_item_to_queue()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/concurrent/futures/process.py", line 394, in add_call_item_to_queue
    self.call_queue.put(_CallItem(work_id,
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/queues.py", line 94, in put
    self._start_thread()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/queues.py", line 177, in _start_thread
    self._thread.start()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 978, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can\'t create new thread at interpreter shutdown
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/forkserver.py", line 274, in main
    code = _serve_one(child_r, fds,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/forkserver.py", line 313, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
' is not false

lazka
added 2 commits that reference this issue on Aug 26, 2023
tests: skip a broken test
4df3c29
tests: skip a broken test
8e7cde8
35 remaining items
serhiy-storchaka
added 2 commits that reference this issue on Nov 10, 2023
[3.12] pythongh-109370: Fix unexpected traceback output in test_concu…
7d01d51
[3.12] gh-109370: Fix unexpected traceback output in test_concurrent_…
4018209
encukou
added 2 commits that reference this issue on Jan 23, 2024
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
188228f
[3.12] gh-114440: Close writer pipe in multiprocessing.Queue, not con…
c912bc3
naveen521kk
added 5 commits that reference this issue on Feb 19, 2024
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
c4877f2
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
fd8463b
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
df2288a
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
df5f451
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
69198de
naveen521kk
added 3 commits that reference this issue on Jul 11, 2024
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
6a3e15b
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
23a82b7
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
701b465
naveen521kk
added a commit that references this issue on Aug 5, 2024
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
2ce1791
Glyphack
added a commit that references this issue on Sep 2, 2024
pythongh-109370: Fix unexpected traceback output in test_concurrent_f…
25b6222
naveen521kk
added a commit that references this issue on Sep 4, 2024
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
1ca1659

Metadata

Assignees

No one assigned

Labels

OS-windowstestsTests in the Lib/test dirTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Projects

Multiprocessing issues
Status
Done
Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219

Description

Linked PRs

Activity

vstinner commented on Jul 25, 2023

vstinner commented on Jul 25, 2023

vstinner commented on Jul 25, 2023

AlexWaygood commented on Jul 26, 2023

Eclips4 commented on Aug 24, 2023

vstinner commented on Aug 25, 2023

Eclips4 commented on Aug 25, 2023

lazka commented on Aug 25, 2023

vstinner commented on Aug 26, 2023

vstinner commented on Aug 26, 2023

35 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions