Support for multiprocessing? If not, maybe rename to MultithreadingMock?

Spoke offline with @voidspace and suggested to keep this PR for the multithreading use case only as it is not trivial to get this working for multiprocess. It was also suggested to rename the mock to make it clear it is about multithreading and keep the Event as private as we are not exposing it anymore.

@tirkarthi @voidspace @lisroach if this looks good I'll get started with the docs.

Gentle ping @tirkarthi @voidspace @lisroach! Let me know if the general idea/naming looks good and I'll get to work on the documentation.

Sorry, got caught up with new job. I will try to get to the PR after two weeks.

I am fine with the PR. The primary concern in my implementation was to have mock_timeout fit in the API along with either to have a default or wait forever and hope we have reached some consensus on it as per the PR. It would be helpful if one of the core dev accepts this idea for merge. Also does it needs some more discussion from other channels like discourse, mailing list etc since I don't have experience in proposing new API. Thanks Mario for picking up the PR.

As a note, IIRC when speaking with @voidspace, he liked the idea of having the timeout namespaced. He can further comment.

What does it mean to have timeout namespaced? Have a parameter called "mock_timeout"?

Yeah, just prepending it with “mock_”, mothibg fancy.

having just ‘timeout’ can easily collide with existing functions that take a timeout parameter.

Gentle ping @lisroach @voidspace, just wished yesterday that I had this mock in a codebase :).

@mariocj89 - might be worth emailing or otherwise contacting @lisroach and @voidspace offline, I know GitHub notifications can be noisy for some people and a ping on here may not get to them.

I don't think adding an additional class is justified enough. It does not really make Mock anymore "threaded" than it already is, just adds a convenience method on top of the threading facility. @cjw296 was already confused with the name above. Recall that unittest is somewhat hard to navigate already.

I'm also not convinced that implementation is convenient enough: e.g. the implementation does not allow to wait for arbitrary conditions.

One way or another let's get it through.

Perhaps with the release of 3.9 it's time to consider this change for 3.10?

I love the functionality of this, obviously it is solving some real-world problems.

I think @Kentzo makes a good point, does it add enough functionality to justify creating a new class? Adding the capability to Mock on the backend is a nicer experience for end-users, who don't have to learn about the new class to get the same functionality.

@Kentzo have the backwards compatibility issues with #20759 been resolved? I agree with Pablo that backwards compatibility is a must, even if we have to sacrifice some features. ThreadingMock fits with the Mock code-style and behavior, and the addition of a new class would not cause any breaking changes.

@lisroach Yes, the issue that was pointed out (I changed the called property from a boolean to a boolean-alike object) was resolved: I added the call_event property instead.

Just one plea here: many people are stuck on older versions of Python, and so only get to use new stuff via the backport, if at all possible, please have a thought to writing code in such a way that it'll run on Python 3.6. We already had to skip the "position-only args" commit in the backport, and that was pretty painful.

I think @Kentzo makes a good point, does it add enough functionality to justify creating a new class

I'm absolutely fine with that. A year ago there was a discussion where it seems that you and @voidspace where leaning more towards having specific mocks for different functionality, where it was discussed to have this implemented as such. But if things have changed I'm 100% fine with that :).

Should we then close this PR go with @Kentzo proposal?

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-emscripten node (dynamic linking) 3.x has failed when building commit d65b783.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1056/builds/2437) and take a look at the build logs.
4. Check if the failure is related to this commit (d65b783) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1056/builds/2437

Failed tests:

• test_unittest

Failed subtests:

• test_wait_magic_method - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_magic_method
• test_wait_success_with_instance_timeout - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_success_with_instance_timeout
• test_wait_success - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_success
• test_wait_until_any_call_change_global_and_override - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_change_global_and_override
• test_wait_success_with_timeout_override - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_success_with_timeout_override
• test_wait_failed_with_instance_timeout - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_failed_with_instance_timeout
• test_wait_until_any_call_keywords - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_keywords
• test_wait_until_any_call_positional - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_positional
• test_wait_failed_with_timeout_override - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_failed_with_timeout_override

Summary of the results of the build (if available):

== Tests result: FAILURE ==

325 tests OK.

10 slowest tests:

• test_math: 1 min 44 sec
• test_hashlib: 1 min 41 sec
• test_unparse: 49.0 sec
• test_tokenize: 43.6 sec
• test_capi: 29.1 sec
• test_fstring: 28.5 sec
• test_tarfile: 27.2 sec
• test_unicodedata: 25.7 sec
• test_zipfile: 18.5 sec
• test_pickle: 18.2 sec

1 test failed:
test_unittest

120 tests skipped:
test.test_asyncio.test_base_events
test.test_asyncio.test_buffered_proto
test.test_asyncio.test_context
test.test_asyncio.test_eager_task_factory
test.test_asyncio.test_events test.test_asyncio.test_futures
test.test_asyncio.test_futures2 test.test_asyncio.test_locks
test.test_asyncio.test_pep492
test.test_asyncio.test_proactor_events
test.test_asyncio.test_protocols test.test_asyncio.test_queues
test.test_asyncio.test_runners
test.test_asyncio.test_selector_events
test.test_asyncio.test_sendfile test.test_asyncio.test_server
test.test_asyncio.test_sock_lowlevel test.test_asyncio.test_ssl
test.test_asyncio.test_sslproto test.test_asyncio.test_streams
test.test_asyncio.test_subprocess
test.test_asyncio.test_taskgroups test.test_asyncio.test_tasks
test.test_asyncio.test_threads test.test_asyncio.test_timeouts
test.test_asyncio.test_transports
test.test_asyncio.test_unix_events test.test_asyncio.test_waitfor
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test__xxinterpchannels
test__xxsubinterpreters test_asyncgen test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_httplib test_httpservers test_idle test_imaplib
test_interpreters test_ioctl test_kqueue test_launcher test_lzma
test_mmap test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_openpty test_pdb
test_perf_profiler test_perfmaps test_poll test_poplib test_pty
test_pwd test_queue test_readline test_regrtest test_repl
test_resource test_select test_selectors test_smtplib test_smtpnet
test_socket test_socketserver test_ssl test_stable_abi_ctypes
test_startfile test_subprocess test_sys_settrace test_syslog
test_tcl test_thread test_threadedtempfile test_threading
test_threading_local test_tkinter test_tools test_ttk
test_ttk_textonly test_turtle test_urllib2 test_urllib2_localnet
test_urllib2net test_urllibnet test_venv test_wait3 test_wait4
test_webbrowser test_winconsoleio test_winreg test_winsound
test_wmi test_wsgiref test_xmlrpc test_zipfile64
test_zipimport_support test_zoneinfo
0:25:03 load avg: 11.72
0:25:03 load avg: 11.72 Re-running failed tests is not supported with --python host runner option.

Total duration: 25 min 3 sec

Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 107, in test_wait_success_with_instance_timeout
    self.run_async(something.method_1, delay=0.01)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 185, in test_wait_until_any_call_keywords
    self.run_async(something.method_1, a=1, delay=0.1)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 157, in test_wait_magic_method
    self.run_async(something.method_1.__str__, delay=0.01)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 117, in test_wait_failed_with_instance_timeout
    self.run_async(something.method_1, delay=0.5)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 97, in test_wait_success
    self.run_async(something.method_1, delay=0.01)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 231, in test_wait_until_any_call_change_global_and_override
    self.run_async(m1, delay=0.1)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 128, in test_wait_success_with_timeout_override
    self.run_async(something.method_1, delay=0.05)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 136, in test_wait_failed_with_timeout_override
    self.run_async(something.method_1, delay=0.1)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 166, in test_wait_until_any_call_positional
    self.run_async(something.method_1, 1, delay=0.1)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-emscripten node (pthreads) 3.x has failed when building commit d65b783.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1050/builds/2426) and take a look at the build logs.
4. Check if the failure is related to this commit (d65b783) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1050/builds/2426

Failed tests:

• test_unittest

Failed subtests:

• test_wait_until_any_call_keywords - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_keywords

Summary of the results of the build (if available):

== Tests result: FAILURE ==

329 tests OK.

10 slowest tests:

• test_hashlib: 2 min 16 sec
• test_math: 2 min 5 sec
• test_tarfile: 1 min 14 sec
• test_tokenize: 43.7 sec
• test_unparse: 43.1 sec
• test_io: 37.7 sec
• test_capi: 37.0 sec
• test_unicodedata: 29.5 sec
• test_pathlib: 27.8 sec
• test_pickle: 23.2 sec

1 test failed:
test_unittest

116 tests skipped:
test.test_asyncio.test_base_events
test.test_asyncio.test_buffered_proto
test.test_asyncio.test_context
test.test_asyncio.test_eager_task_factory
test.test_asyncio.test_events test.test_asyncio.test_futures
test.test_asyncio.test_futures2 test.test_asyncio.test_locks
test.test_asyncio.test_pep492
test.test_asyncio.test_proactor_events
test.test_asyncio.test_protocols test.test_asyncio.test_queues
test.test_asyncio.test_runners
test.test_asyncio.test_selector_events
test.test_asyncio.test_sendfile test.test_asyncio.test_server
test.test_asyncio.test_sock_lowlevel test.test_asyncio.test_ssl
test.test_asyncio.test_sslproto test.test_asyncio.test_streams
test.test_asyncio.test_subprocess
test.test_asyncio.test_taskgroups test.test_asyncio.test_tasks
test.test_asyncio.test_threads test.test_asyncio.test_timeouts
test.test_asyncio.test_transports
test.test_asyncio.test_unix_events test.test_asyncio.test_waitfor
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test__xxinterpchannels
test__xxsubinterpreters test_asyncgen test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_httplib test_httpservers test_idle test_imaplib
test_interpreters test_ioctl test_kqueue test_launcher test_lzma
test_mmap test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_openpty test_pdb
test_perf_profiler test_perfmaps test_poll test_poplib test_pty
test_pwd test_readline test_regrtest test_repl test_resource
test_select test_selectors test_smtplib test_smtpnet test_socket
test_socketserver test_ssl test_stable_abi_ctypes test_startfile
test_subprocess test_sys_settrace test_syslog test_tcl
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
test_winreg test_winsound test_wmi test_wsgiref test_xmlrpc
test_xxlimited test_zipfile64 test_zipimport_support test_zoneinfo
0:25:59 load avg: 7.86
0:25:59 load avg: 7.86 Re-running failed tests is not supported with --python host runner option.

Total duration: 25 min 59 sec

Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 188, in test_wait_until_any_call_keywords
    self.assertNotIn(call(a=1), something.method_1.mock_calls)
AssertionError: call(a=1) unexpectedly found in [call(a=1)]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit d65b783.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/2414) and take a look at the build logs.
4. Check if the failure is related to this commit (d65b783) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/2414

Failed tests:

• test_unittest

Failed subtests:

• test_wait_magic_method - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_magic_method
• test_wait_success_with_instance_timeout - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_success_with_instance_timeout
• test_wait_success - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_success
• test_wait_until_any_call_change_global_and_override - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_change_global_and_override
• test_wait_success_with_timeout_override - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_success_with_timeout_override
• test_wait_failed_with_instance_timeout - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_failed_with_instance_timeout
• test_wait_until_any_call_keywords - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_keywords
• test_wait_until_any_call_positional - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_positional
• test_wait_failed_with_timeout_override - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_failed_with_timeout_override

Summary of the results of the build (if available):

== Tests result: FAILURE ==

317 tests OK.

10 slowest tests:

• test_hashlib: 1 min 12 sec
• test_math: 1 min 11 sec
• test_tokenize: 42.9 sec
• test_capi: 32.1 sec
• test_unparse: 30.2 sec
• test_unicodedata: 25.1 sec
• test_pickle: 22.6 sec
• test_tarfile: 19.1 sec
• test_decimal: 12.7 sec
• test_fstring: 11.8 sec

1 test failed:
test_unittest

128 tests skipped:
test.test_asyncio.test_base_events
test.test_asyncio.test_buffered_proto
test.test_asyncio.test_context
test.test_asyncio.test_eager_task_factory
test.test_asyncio.test_events test.test_asyncio.test_futures
test.test_asyncio.test_futures2 test.test_asyncio.test_locks
test.test_asyncio.test_pep492
test.test_asyncio.test_proactor_events
test.test_asyncio.test_protocols test.test_asyncio.test_queues
test.test_asyncio.test_runners
test.test_asyncio.test_selector_events
test.test_asyncio.test_sendfile test.test_asyncio.test_server
test.test_asyncio.test_sock_lowlevel test.test_asyncio.test_ssl
test.test_asyncio.test_sslproto test.test_asyncio.test_streams
test.test_asyncio.test_subprocess
test.test_asyncio.test_taskgroups test.test_asyncio.test_tasks
test.test_asyncio.test_threads test.test_asyncio.test_timeouts
test.test_asyncio.test_transports
test.test_asyncio.test_unix_events test.test_asyncio.test_waitfor
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test__xxinterpchannels
test__xxsubinterpreters test_asyncgen test_bz2 test_clinic
test_cmd_line test_concurrent_futures test_contextlib_async
test_ctypes test_curses test_dbm_gnu test_dbm_ndbm test_devpoll
test_doctest test_docxmlrpc test_dtrace test_embed test_epoll
test_faulthandler test_fcntl test_file_eintr test_fork1
test_ftplib test_gdb test_grp test_gzip test_httplib
test_httpservers test_idle test_imaplib test_interpreters
test_ioctl test_kqueue test_launcher test_lzma test_mailbox
test_mmap test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_openpty test_pdb
test_perf_profiler test_perfmaps test_poll test_poplib test_pty
test_pwd test_queue test_readline test_regrtest test_repl
test_resource test_select test_selectors test_smtplib test_smtpnet
test_socket test_socketserver test_sqlite3 test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_thread
test_threadedtempfile test_threading test_threading_local
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib test_urllib2 test_urllib2_localnet test_urllib2net
test_urllib_response test_urllibnet test_venv test_wait3
test_wait4 test_webbrowser test_winconsoleio test_winreg
test_winsound test_wmi test_wsgiref test_xmlrpc test_xxlimited
test_zipfile64 test_zipimport_support test_zlib test_zoneinfo
0:06:19 load avg: 2.14
0:06:19 load avg: 2.14 Re-running failed tests is not supported with --python host runner option.

Total duration: 6 min 19 sec

Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 107, in test_wait_success_with_instance_timeout
    self.run_async(something.method_1, delay=0.01)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 185, in test_wait_until_any_call_keywords
    self.run_async(something.method_1, a=1, delay=0.1)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 128, in test_wait_success_with_timeout_override
    self.run_async(something.method_1, delay=0.05)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 231, in test_wait_until_any_call_change_global_and_override
    self.run_async(m1, delay=0.1)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 117, in test_wait_failed_with_instance_timeout
    self.run_async(something.method_1, delay=0.5)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 136, in test_wait_failed_with_timeout_override
    self.run_async(something.method_1, delay=0.1)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 166, in test_wait_until_any_call_positional
    self.run_async(something.method_1, 1, delay=0.1)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 157, in test_wait_magic_method
    self.run_async(something.method_1.__str__, delay=0.01)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread


Traceback (most recent call last):
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 97, in test_wait_success
    self.run_async(something.method_1, delay=0.01)
  File "/Lib/test/test_unittest/testmock/testthreadingmock.py", line 28, in run_async
    self._executor.submit(
  File "/Lib/concurrent/futures/thread.py", line 179, in submit
    self._adjust_thread_count()
  File "/Lib/concurrent/futures/thread.py", line 202, in _adjust_thread_count
    t.start()
  File "/Lib/threading.py", line 971, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

OK! Let's see the results:

1. wasm32-emscripten node:

RuntimeError: can't start new thread

2. wasm32-emscripten node (pthreads) 3.x

Fails with:

======================================================================
FAIL: test_wait_until_any_call_keywords (test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_wait_until_any_call_keywords)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 188, in test_wait_until_any_call_keywords
    self.assertNotIn(call(a=1), something.method_1.mock_calls)
AssertionError: call(a=1) unexpectedly found in [call(a=1)]

3. wasm32-wasi 3.x

RuntimeError: can't start new thread

I'm worried 2 might come from flaky tests, will review them all to make sure we can make them more resilient (as you know, with timeouts we are exposed to the host).

Need also to investigate what is going on with the error to start a thread.

@mariocj89 Since the failures are in wasm related builds where threads might not be supported maybe you need to wrap the test file to skip when threads are not working. I see requires_working_threading being used in many tests. The docstring also notes threading related tests might fail in wasm builds. I guess wasm related builds are only during full test run and is not present in GitHub actions.

Indeed, just spoke with @pablogsal who told me the same :P.

Will send a PR before EOD.

It seems below tests are totally skipped and maybe a related one needs to be added to the mock related tests as well. Since the test will be backported to mock backport project maybe the helper function needs to be inlined in case mock backport runs on wasm builds to not rely on test.support

rg '^threading_helper.requires_working_threading' Lib/test
Lib/test/test_queue.py
14:threading_helper.requires_working_threading(module=True)

Lib/test/test_thread.py
13:threading_helper.requires_working_threading(module=True)

Lib/test/test_threading_local.py
15:threading_helper.requires_working_threading(module=True)

Lib/test/test_threading.py
29:threading_helper.requires_working_threading(module=True)

Lib/test/test_threadedtempfile.py
24:threading_helper.requires_working_threading(module=True)

Lib/test/test_importlib/test_locks.py
15:threading_helper.requires_working_threading(module=True)

Lib/test/test_importlib/test_threaded_import.py
22:threading_helper.requires_working_threading(module=True)

PR to add the marker: #106366

Will followup trying to harden the rests of the tests.

@cjw296 even though the API looks neat to me, I showed it to a colleague who does not have a lot of mocking experiencet and suggested to use wait_for_called_with rather than wait_for_any_call.

Whilst I like how wait_for_any_call mirrors the naming convention with assert_any_call and leaves the door open to use wait_for_called_with to fail if the next call does not match the specified call, it might be easier to read to just call it wait_for_called_with and if we ever need a "wait for called with X and fail if different" we could name it "wait_for_next_call".

What do you think?

mock = ThreadingMock()
mock.wait_for_any_call("arg1", kwarg2=2)   #1
mock.wait_for_call_with("arg1", kwarg2=2)  #2

1 or 2?

I'm fine with one, but if we want to change it, it is possibly now or never.

Maybe it should be wait_for_any_call_with?

Sounds good. PR ready: #106414

There seems to be another problem with this -- I just got a test flake, apparently due to the use of short fixed timeouts in the test added here. See https://github.com/python/cpython/actions/runs/5559403783/jobs/10155484726.

Not sure what a "test flake" means in this context?

Looking at the test again, I'm unclear it's sensible:

Why the delays of 0.2, 0.5 and 0.6?
The assertions on lines 169, 175 and 176 seem likely but not guaranteed?

Lines 178-180 suppose that because call(3) has occurred, then call(2) will have occurred. Again, that seems likely given the 0.1s delay but not guaranteed, and indeed seems to have not happened here. What could lead these to be out of order? Some funny OS-level scheduler thing on that macOS runner? Something on @gvanrossum's branch?

FWIW, this is the first time I'm aware of this test failing, and it's run nightly on a bunch of different python versions in the backport CI: https://app.circleci.com/pipelines/github/testing-cabal/mock?branch=master
Those are all on Linux, mind...

@mariocj89 - what are your thoughts on this? What can we do to make this test reliable. A bonus would be to make them faster to run, these threading tests are noticeably much slower than the other mock tests...

It is a hard balance to keep the tests fast but reliable.

I will try changing this to have short retrying timeouts to make that happen. Will send a PR on Monday if there is not rush as I don't have access to a PC this weekend

Not sure what a "test flake" means in this context?

A flaky test is a test that usually passes but sometimes fails. Tests that rely on timeouts are often flaky (but there are other causes). Any large complex system has flaky tests. In this case, the test failed in CI on macOS only; Windows and Ubuntu were fine. Re-running the test on my machine locally it passes. That makes it flaky in my book.

Note that, as defense against flaky tests, our CI machinery re-runs a failed test once and if it passes the second time, it is accepted. In this case we were apparently unlucky with the timeouts twice in a row.

Thanks for having a second look at these tests!

#106822 should prevent any future failure.

Is this change going to be included in 3.12?

I haven’t checked the source but I see no evidence of a backport to the 3.12 branch, so presumably not.

@Kentzo - it doesn't look like ThreadingMock made it into https://github.com/python/cpython/blob/3.12/Lib/unittest/mock.py, so no need for this fix.

ThreadingMock is available in the the mock rolling backport if you want to use it before 3.13 is out:
https://mock.readthedocs.io/en/latest/changelog.html#id1
https://pypi.org/project/mock/5.1.0/

Is it possible to use ThreadingMock with auto spec feature?

i.e. let patch or create_autospec create a ThreadingMock instance instead of a MagicMock instance.

Per the mocks docs:

You can specify an alternative class of Mock using the new_callable argument to patch().

That should work in theory, give it a try!

Per the mocks docs:

You can specify an alternative class of Mock using the new_callable argument to patch().

That should work in theory, give it a try!

This is not possible with autospec. The #1 reason for using patch :-(.

ValueError: Cannot use 'autospec' and 'new_callable' together