GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-108388: Split test_multiprocessing_spawn by vstinner · Pull Request #108396 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Aug 24, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.

  • Issue: Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@vstinner
pythongh-108388: Split test_multiprocessing_spawn
bfd362d 
Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
@bedevere-bot bedevere-bot mentioned this pull request Aug 24, 2023
Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Aug 24, 2023
@vstinner

vstinner commented Aug 24, 2023

Copy link
Copy Markdown
Member Author

Example running 4 slow tests:

./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest

Timing (reference => this PR): 3 min 22 sec => 2 min 22 sec (-1 min)

On the main branch:

10 slowest tests:
- test_multiprocessing_spawn: 3 min 22 sec
- test_asyncio: 2 min 10 sec
- test_multiprocessing_forkserver: 1 min 43 sec
- test_multiprocessing_fork: 1 min 17 sec

Total duration: 3 min 22 sec

With this PR + PR #108393 (needed for this test):

40 tests OK.

10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec

2 tests skipped:
    test_asyncio.test_windows_events test_asyncio.test_windows_utils

Total duration: 2 min 22 sec

All CPUs are well used, the system load peak is 12.33 which is a good thing (the machine has 12 threads/6 cores).

Full logs:

Details 
vstinner@mona$ ./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest
0:00:00 load avg: 6.17 Run tests in parallel using 14 child processes
0:00:01 load avg: 6.17 [ 1/42] test_asyncio.test_protocols passed
0:00:01 load avg: 6.17 [ 2/42] test_asyncio.test_futures2 passed
0:00:01 load avg: 6.17 [ 3/42] test_asyncio.test_buffered_proto passed
0:00:01 load avg: 6.17 [ 4/42] test_asyncio.test_context passed
0:00:01 load avg: 6.17 [ 5/42] test_asyncio.test_pep492 passed
0:00:02 load avg: 6.17 [ 6/42] test_asyncio.test_runners passed
0:00:02 load avg: 6.17 [ 7/42] test_asyncio.test_proactor_events passed
0:00:02 load avg: 6.17 [ 8/42] test_asyncio.test_queues passed
0:00:03 load avg: 6.17 [ 9/42] test_asyncio.test_server passed
0:00:03 load avg: 6.17 [10/42] test_asyncio.test_selector_events passed
0:00:03 load avg: 6.17 [11/42] test_asyncio.test_eager_task_factory passed
0:00:04 load avg: 6.80 [12/42] test_asyncio.test_locks passed
0:00:04 load avg: 6.80 [13/42] test_asyncio.test_threads passed
0:00:06 load avg: 6.80 [14/42] test_asyncio.test_timeouts passed
0:00:06 load avg: 6.80 [15/42] test_asyncio.test_transports passed
0:00:07 load avg: 6.80 [16/42] test_asyncio.test_streams passed
0:00:07 load avg: 6.80 [17/42] test_asyncio.test_windows_events skipped
test_asyncio.test_windows_events skipped -- Windows only
0:00:07 load avg: 6.80 [18/42] test_asyncio.test_base_events passed
0:00:07 load avg: 6.80 [19/42] test_asyncio.test_sslproto passed
0:00:07 load avg: 6.80 [20/42] test_asyncio.test_windows_utils skipped
test_asyncio.test_windows_utils skipped -- Windows only
0:00:08 load avg: 6.80 [21/42] test_asyncio.test_unix_events passed
0:00:09 load avg: 7.70 [22/42] test_asyncio.test_waitfor passed
0:00:10 load avg: 7.70 [23/42] test_asyncio.test_sock_lowlevel passed
0:00:10 load avg: 7.70 [24/42] test_asyncio.test_sendfile passed
0:00:11 load avg: 7.70 [25/42] test_asyncio.test_taskgroups passed
0:00:12 load avg: 7.70 [26/42] test_asyncio.test_futures passed
0:00:18 load avg: 8.68 [27/42] test_asyncio.test_ssl passed
0:00:28 load avg: 10.42 [28/42] test_multiprocessing_fork.test_threads passed
0:00:29 load avg: 11.02 [29/42] test_asyncio.test_subprocess passed
Unknown child process pid 313199, will report returncode 255
Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that handles pid 313199 is closed
0:00:32 load avg: 11.02 [30/42] test_multiprocessing_fork.test_manager passed -- running: test_asyncio.test_events (32.0 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310261) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313364) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:32 load avg: 11.02 [31/42] test_multiprocessing_forkserver.test_threads passed -- running: test_asyncio.test_events (32.9 sec)
0:00:34 load avg: 11.50 [32/42] test_multiprocessing_fork.test_misc passed -- running: test_asyncio.test_events (34.6 sec), test_asyncio.test_tasks (31.7 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313266) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:40 load avg: 12.02 [33/42] test_asyncio.test_events passed (39.3 sec) -- running: test_multiprocessing_forkserver.test_misc (30.9 sec), test_asyncio.test_tasks (37.1 sec), test_multiprocessing_fork.test_processes (32.4 sec), test_multiprocessing_forkserver.test_manager (31.1 sec)
0:00:51 load avg: 12.33 [34/42] test_multiprocessing_spawn.test_threads passed -- running: test_multiprocessing_forkserver.test_processes (40.9 sec), test_multiprocessing_spawn.test_processes (32.8 sec), test_multiprocessing_forkserver.test_misc (42.1 sec), test_multiprocessing_spawn.test_misc (38.9 sec), test_asyncio.test_tasks (48.2 sec), test_multiprocessing_fork.test_processes (43.5 sec), test_multiprocessing_spawn.test_manager (39.7 sec), test_multiprocessing_forkserver.test_manager (42.2 sec)
0:00:51 load avg: 12.33 [35/42] test_multiprocessing_fork.test_processes passed (43.4 sec) -- running: test_multiprocessing_forkserver.test_processes (41.5 sec), test_multiprocessing_spawn.test_processes (33.5 sec), test_multiprocessing_forkserver.test_misc (42.7 sec), test_multiprocessing_spawn.test_misc (39.5 sec), test_asyncio.test_tasks (48.8 sec), test_multiprocessing_spawn.test_manager (40.4 sec), test_multiprocessing_forkserver.test_manager (42.9 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:55 load avg: 12.06 [36/42] test_multiprocessing_forkserver.test_manager passed (45.5 sec) -- running: test_multiprocessing_forkserver.test_processes (45.0 sec), test_multiprocessing_spawn.test_processes (37.0 sec), test_multiprocessing_forkserver.test_misc (46.2 sec), test_multiprocessing_spawn.test_misc (43.0 sec), test_asyncio.test_tasks (52.3 sec), test_multiprocessing_spawn.test_manager (43.9 sec)
0:00:56 load avg: 12.06 [37/42] test_asyncio.test_tasks passed (52.9 sec) -- running: test_multiprocessing_forkserver.test_processes (46.3 sec), test_multiprocessing_spawn.test_processes (38.2 sec), test_multiprocessing_forkserver.test_misc (47.5 sec), test_multiprocessing_spawn.test_misc (44.3 sec), test_multiprocessing_spawn.test_manager (45.1 sec)
0:01:04 load avg: 10.75 [38/42] test_multiprocessing_forkserver.test_misc passed (54.3 sec) -- running: test_multiprocessing_forkserver.test_processes (53.8 sec), test_multiprocessing_spawn.test_processes (45.7 sec), test_multiprocessing_spawn.test_misc (51.8 sec), test_multiprocessing_spawn.test_manager (52.6 sec)
0:01:14 load avg: 9.63 [39/42] test_multiprocessing_spawn.test_manager passed (1 min 2 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 4 sec), test_multiprocessing_spawn.test_processes (56.1 sec), test_multiprocessing_spawn.test_misc (1 min 2 sec)
0:01:35 load avg: 12.00 [40/42] test_multiprocessing_spawn.test_misc passed (1 min 22 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 24 sec), test_multiprocessing_spawn.test_processes (1 min 16 sec)
0:01:52 load avg: 13.34 [41/42] test_multiprocessing_forkserver.test_processes passed (1 min 41 sec) -- running: test_multiprocessing_spawn.test_processes (1 min 34 sec)
0:02:22 load avg: 8.84 running: test_multiprocessing_spawn.test_processes (2 min 4 sec)
0:02:22 load avg: 8.84 [42/42] test_multiprocessing_spawn.test_processes passed (2 min 3 sec)

== Tests result: SUCCESS ==

40 tests OK.

10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec

2 tests skipped:
    test_asyncio.test_windows_events test_asyncio.test_windows_utils

Total duration: 2 min 22 sec
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner
vstinner merged commit aa9a359 into python:main Aug 24, 2023
@bedevere-bot bedevere-bot removed the awaiting core review label Aug 24, 2023
@vstinner
vstinner deleted the test_mp_packages branch August 24, 2023 03:50
@vstinner vstinner mentioned this pull request Aug 24, 2023
gh-108388: test_concurrent_futures requires cpu #108389
Closed
gpshead
gpshead reviewed Aug 24, 2023
View reviewed changes
Comment thread Lib/test/_test_multiprocessing.py

def install_tests_in_module_dict(remote_globs, start_method):
def install_tests_in_module_dict(remote_globs, start_method,
only_type=None, exclude_types=False):

@gpshead gpshead Aug 24, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function needs proper documentation as to what it does and how it behaves. The new parameters behaviors are unobvious.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@gpshead gpshead added the needs backport to 3.12 only security fixes label Aug 24, 2023
@miss-islington

miss-islington commented Aug 24, 2023

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Aug 24, 2023

Copy link
Copy Markdown

GH-108442 is a backport of this pull request to the 3.12 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label Aug 24, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 24, 2023
@vstinner @miss-islington
pythongh-108388: Split test_multiprocessing_spawn (pythonGH-108396)
89579a8 
Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
(cherry picked from commit aa9a359)

Co-authored-by: Victor Stinner <[email protected]>
Yhg1s pushed a commit that referenced this pull request Aug 25, 2023
@miss-islington @vstinner
[3.12] gh-108388: Split test_multiprocessing_spawn (GH-108396) (#108442)
2262190 
gh-108388: Split test_multiprocessing_spawn (GH-108396)

Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
(cherry picked from commit aa9a359)

Co-authored-by: Victor Stinner <[email protected]>
@vstinner vstinner added the needs backport to 3.11 only security fixes label Sep 21, 2023
@miss-islington

miss-islington commented Sep 21, 2023

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 21, 2023
@vstinner @miss-islington
pythongh-108388: Split test_multiprocessing_spawn (pythonGH-108396)
b2cea8f 
Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
(cherry picked from commit aa9a359)

Co-authored-by: Victor Stinner <[email protected]>
@bedevere-app

bedevere-app Bot commented Sep 21, 2023

Copy link
Copy Markdown

GH-109688 is a backport of this pull request to the 3.11 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.11 only security fixes label Sep 21, 2023
vstinner added a commit to miss-islington/cpython that referenced this pull request Sep 21, 2023
@vstinner
pythongh-108388: Split test_multiprocessing_spawn (pythonGH-108396)
9ae7650 
Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
(cherry picked from commit aa9a359)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this pull request Sep 21, 2023
@miss-islington @vstinner
[3.11] gh-108388: Split test_multiprocessing_spawn (GH-108396) (#109688)
b3af888 
gh-108388: Split test_multiprocessing_spawn (GH-108396)

Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
(cherry picked from commit aa9a359)

Co-authored-by: Victor Stinner <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@vstinner @miss-islington @bedevere-bot @gpshead
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.