GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-84559: Deprecate fork being the multiprocessing default. by gpshead · Pull Request #100618 · python/cpython

Conversation

@gpshead

@gpshead gpshead commented Dec 30, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start().

  • Issue: multiprocessing's default posix start method of 'fork' is broken: change to `'forkserver' || 'spawn' #84559

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@gpshead
pythonGH-84559: Deprecate fork being the multiprocessing default.
1cecbdb 
This starts the process. Users who don't specify their own start method
and use the default on platforms where it is 'fork' will see a
DeprecationWarning upon multiprocessing.Pool() construction or upon
multiprocessing.Process.start().
@gpshead gpshead added the topic-multiprocessing label Dec 30, 2022
@bedevere-bot bedevere-bot mentioned this pull request Dec 30, 2022
multiprocessing's default posix start method of 'fork' is broken: change to `'forkserver' || 'spawn' #84559
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Dec 30, 2022
@gpshead gpshead self-assigned this Dec 30, 2022
gpshead added 13 commits December 31, 2022 04:10
@gpshead
address CI test failures?
df83fba
@gpshead
the new warning needs no pickle name mapping
ec1e868
@gpshead
avoid DeprecationWarnings vs -Werror
a1064de
@gpshead
prefer forkserver over fork for concurrent.futures
223b748
@gpshead
explicitly use spawn for simple process launchers.
fc64626
@gpshead
warn from concurrent.futures, reword things.
e45fab6
@gpshead
avoid double warning in futures
29764a2
@gpshead
Cleanup messages, test futures warning.
c52b9db
@gpshead
cleanup warn stacklevel logic
4c8cc34
@gpshead
Document all the things!
7179684
@gpshead
Have all doctests use 'spawn'.
f7e3161
@gpshead
Correct warning stacklevel for Manager.
9947860
@gpshead
more doctest cleanup.
5834f2d
@gpshead

gpshead commented Jan 1, 2023

Copy link
Copy Markdown
Member Author

This code could be a lot cleaner if #39615 were implemented...
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead
NEWS entry.
7d48e86
FFY00
FFY00 reviewed Jan 9, 2023
View reviewed changes
Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@gpshead
docs: add the implicit i
28c0cfc
@gpshead gpshead mentioned this pull request Jan 21, 2023
gh-39615: Add warnings.warn() skip_file_prefixes support #100840
Merged
gpshead added 4 commits January 27, 2023 18:36
@gpshead
Merge branch 'main' into multiprocessing/deprecate-fork-default
2820bb8
@gpshead
Use skip_file_prefixes.
4c16205
@gpshead
use the modern warnings API, doc updates.
7d84a72
@gpshead
documentation cleanup
8b2029e
@gpshead gpshead added the stdlib Standard Library Python modules in the Lib/ directory label Jan 28, 2023
@gpshead
gpshead requested a review from Yhg1s January 28, 2023 04:28
@gpshead
gpshead marked this pull request as ready for review January 28, 2023 04:28
@gpshead
gpshead requested review from 1st1, asvetlov, gvanrossum, kumaraditya303 and vsajip as code owners January 28, 2023 04:28
gvanrossum
gvanrossum reviewed Jan 28, 2023
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes to test_asyncio LGTM. I hope I won't have to review the rest, I know basically nothing about multiprocessing. :-(

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@kumaraditya303
kumaraditya303 removed their request for review January 31, 2023 16:53
kumaraditya303
kumaraditya303 reviewed Jan 31, 2023
View reviewed changes

@kumaraditya303 kumaraditya303 left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve the asyncio change.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Yhg1s
Yhg1s approved these changes Feb 1, 2023
View reviewed changes
Comment thread Lib/multiprocessing/context.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Feb 1, 2023
gpshead added 2 commits February 1, 2023 14:40
@gpshead
Remove an unused variable.
846e9a0
@gpshead
doc typo fix, unnecessary method change revert.
7078986
@hugovk

hugovk commented Feb 2, 2023

Copy link
Copy Markdown
Member

Please could you add this to What's New, under Pending Removal in Python 3.14?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead
Add What's New entries.
e3e3b3c
@gpshead

gpshead commented Feb 2, 2023

Copy link
Copy Markdown
Member Author

Please could you add this to What's New, under Pending Removal in Python 3.14?

Ahha, I knew we had a list of upcoming things somewhere. Good idea, done! What's New text added.
hugovk reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead
gpshead merged commit 0ca67e6 into python:main Feb 2, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label Feb 2, 2023
@gpshead
gpshead deleted the multiprocessing/deprecate-fork-default branch February 2, 2023 23:50
@bedevere-bot

bedevere-bot commented Feb 3, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-emscripten node (dynamic linking) 3.x has failed when building commit 0ca67e6.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1056/builds/1286) and take a look at the build logs.
  4. Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1056/builds/1286

Failed tests:

  • test_multiprocessing_defaults

Failed subtests:

  • test_no_mp_warning_when_using_explicit_fork_context - test.test_multiprocessing_defaults.DefaultWarningsTest.test_no_mp_warning_when_using_explicit_fork_context
  • test_default_fork_start_method_warning_pool - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_pool
  • test_default_fork_start_method_warning_manager - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_manager
  • test_default_fork_start_method_warning_process - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_process

Summary of the results of the build (if available):

== Tests result: FAILURE ==

335 tests OK.

10 slowest tests:

  • test_math: 1 min 57 sec
  • test_tokenize: 1 min 9 sec
  • test_unparse: 48.0 sec
  • test_lib2to3: 45.6 sec
  • test_capi: 27.1 sec
  • test_argparse: 20.3 sec
  • test_unicodedata: 18.7 sec
  • test_pickle: 17.7 sec
  • test_zipfile: 16.2 sec
  • test_decimal: 12.9 sec

1 test failed:
test_multiprocessing_defaults

98 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_httplib test_httpservers test_idle test_imaplib
test_interpreters test_ioctl test_kqueue test_launcher test_lzma
test_mmap test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
test_pdb test_peg_generator test_perf_profiler test_poll
test_poplib test_pty test_pwd test_queue test_readline
test_regrtest test_repl test_resource test_select test_selectors
test_smtplib test_smtpnet test_socket test_socketserver test_spwd
test_ssl test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_thread
test_threadedtempfile test_threading test_threading_local test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
test_winreg test_winsound test_wmi test_wsgiref test_xmlrpc
test_xmlrpc_net test_zipfile64 test_zipimport_support
test_zoneinfo
0:21:59 load avg: 11.21
0:21:59 load avg: 11.21 Re-running failed tests is not supported with --python host runner option.

Total duration: 21 min 59 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 34, in test_default_fork_start_method_warning_process
    process.start()  # warning should point here.
    ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 227, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 302, in _Popen
    return super()._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 285, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/popen_fork.py", line 66, in _launch
    self.pid = os.fork()
               ^^^^^^^^^
OSError: [Errno 52] Function not implemented


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 47, in test_default_fork_start_method_warning_pool
    pool = multiprocessing.Pool(1)  # warning should point here.
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 75, in test_no_mp_warning_when_using_explicit_fork_context
    pool = fork_mp.Pool(1)
           ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 61, in test_default_fork_start_method_warning_manager
    manager = multiprocessing.Manager()  # warning should point here.
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 58, in Manager
    from .managers import SyncManager
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/managers.py", line 29, in <module>
    from . import connection
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Feb 3, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-emscripten node (pthreads) 3.x has failed when building commit 0ca67e6.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1050/builds/1283) and take a look at the build logs.
  4. Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1050/builds/1283

Failed tests:

  • test_multiprocessing_defaults

Failed subtests:

  • test_no_mp_warning_when_using_explicit_fork_context - test.test_multiprocessing_defaults.DefaultWarningsTest.test_no_mp_warning_when_using_explicit_fork_context
  • test_default_fork_start_method_warning_pool - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_pool
  • test_default_fork_start_method_warning_manager - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_manager
  • test_default_fork_start_method_warning_process - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_process

Summary of the results of the build (if available):

== Tests result: FAILURE ==

339 tests OK.

10 slowest tests:

  • test_math: 2 min 1 sec
  • test_tokenize: 1 min 11 sec
  • test_unparse: 47.9 sec
  • test_lib2to3: 43.0 sec
  • test_io: 39.3 sec
  • test_capi: 36.4 sec
  • test_unicodedata: 27.4 sec
  • test_zipfile: 25.3 sec
  • test_pickle: 22.0 sec
  • test_decimal: 19.5 sec

1 test failed:
test_multiprocessing_defaults

94 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_httplib test_httpservers test_idle test_imaplib
test_interpreters test_ioctl test_kqueue test_launcher test_lzma
test_mmap test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
test_pdb test_peg_generator test_perf_profiler test_poll
test_poplib test_pty test_pwd test_readline test_regrtest
test_repl test_resource test_select test_selectors test_smtplib
test_smtpnet test_socket test_socketserver test_spwd test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
test_winreg test_winsound test_wmi test_wsgiref test_xmlrpc
test_xmlrpc_net test_xxlimited test_zipfile64
test_zipimport_support test_zoneinfo
0:22:12 load avg: 7.44
0:22:12 load avg: 7.44 Re-running failed tests is not supported with --python host runner option.

Total duration: 22 min 12 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 75, in test_no_mp_warning_when_using_explicit_fork_context
    pool = fork_mp.Pool(1)
           ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 47, in test_default_fork_start_method_warning_pool
    pool = multiprocessing.Pool(1)  # warning should point here.
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 61, in test_default_fork_start_method_warning_manager
    manager = multiprocessing.Manager()  # warning should point here.
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 58, in Manager
    from .managers import SyncManager
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/managers.py", line 29, in <module>
    from . import connection
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 34, in test_default_fork_start_method_warning_process
    process.start()  # warning should point here.
    ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 227, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 302, in _Popen
    return super()._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 285, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/popen_fork.py", line 66, in _launch
    self.pid = os.fork()
               ^^^^^^^^^
OSError: [Errno 52] Function not implemented
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Feb 3, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit 0ca67e6.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/1328) and take a look at the build logs.
  4. Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/1328

Failed tests:

  • test_multiprocessing_defaults

Failed subtests:

  • test_no_mp_warning_when_using_explicit_fork_context - test.test_multiprocessing_defaults.DefaultWarningsTest.test_no_mp_warning_when_using_explicit_fork_context
  • test_default_fork_start_method_warning_pool - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_pool
  • test_default_fork_start_method_warning_manager - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_manager
  • test_default_fork_start_method_warning_process - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_process

Summary of the results of the build (if available):

== Tests result: FAILURE ==

326 tests OK.

10 slowest tests:

  • test_tokenize: 1 min 14 sec
  • test_math: 1 min 14 sec
  • test_lib2to3: 31.5 sec
  • test_capi: 31.4 sec
  • test_unparse: 29.1 sec
  • test_unicodedata: 22.7 sec
  • test_decimal: 19.1 sec
  • test_statistics: 18.5 sec
  • test_pickle: 13.4 sec
  • test_zipfile: 10.9 sec

1 test failed:
test_multiprocessing_defaults

107 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio test_bz2
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_gzip test_httplib test_httpservers test_idle
test_imaplib test_interpreters test_ioctl test_kqueue
test_launcher test_lzma test_mailbox test_mmap test_msilib
test_multiprocessing_fork test_multiprocessing_forkserver
test_multiprocessing_main_handling test_multiprocessing_spawn
test_nis test_openpty test_ossaudiodev test_pdb test_peg_generator
test_perf_profiler test_pipes test_poll test_poplib test_pty
test_pwd test_queue test_readline test_regrtest test_repl
test_resource test_select test_selectors test_smtplib test_smtpnet
test_socket test_socketserver test_spwd test_sqlite3 test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_thread
test_threadedtempfile test_threading test_threading_local test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib test_urllib2 test_urllib2_localnet test_urllib2net
test_urllib_response test_urllibnet test_venv test_wait3
test_wait4 test_webbrowser test_winconsoleio test_winreg
test_winsound test_wmi test_wsgiref test_xmlrpc test_xmlrpc_net
test_xxlimited test_zipfile64 test_zipimport_support test_zlib
test_zoneinfo
0:05:45 load avg: 3.51
0:05:45 load avg: 3.51 Re-running failed tests is not supported with --python host runner option.

Total duration: 5 min 45 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 47, in test_default_fork_start_method_warning_pool
    pool = multiprocessing.Pool(1)  # warning should point here.
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 34, in test_default_fork_start_method_warning_process
    process.start()  # warning should point here.
    ^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 227, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 302, in _Popen
    return super()._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 284, in _Popen
    from .popen_fork import Popen
  File "/Lib/multiprocessing/popen_fork.py", line 12, in <module>
    class Popen(object):
  File "/Lib/multiprocessing/popen_fork.py", line 24, in Popen
    def poll(self, flag=os.WNOHANG):
                        ^^^^^^^^^^
AttributeError: module 'os' has no attribute 'WNOHANG'


Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 61, in test_default_fork_start_method_warning_manager
    manager = multiprocessing.Manager()  # warning should point here.
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 58, in Manager
    from .managers import SyncManager
  File "/Lib/multiprocessing/managers.py", line 29, in <module>
    from . import connection
  File "/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 75, in test_no_mp_warning_when_using_explicit_fork_context
    pool = fork_mp.Pool(1)
           ^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Feb 3, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x has failed when building commit 0ca67e6.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/469/builds/4156) and take a look at the build logs.
  4. Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/469/builds/4156

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

417 tests OK.

10 slowest tests:

  • test_gdb: 7 min
  • test_concurrent_futures: 2 min 49 sec
  • test_asyncio: 2 min 28 sec
  • test_capi: 2 min 18 sec
  • test_multiprocessing_spawn: 2 min 11 sec
  • test_math: 1 min 49 sec
  • test_tokenize: 1 min 18 sec
  • test_multiprocessing_forkserver: 1 min 16 sec
  • test_multiprocessing_fork: 1 min 11 sec
  • test_subprocess: 50.6 sec

1 test altered the execution environment:
test_multiprocessing_defaults

16 tests skipped:
test_check_c_globals test_devpoll test_ioctl test_kqueue
test_launcher test_msilib test_peg_generator test_startfile
test_tix test_tkinter test_ttk test_winconsoleio test_winreg
test_winsound test_wmi test_zipfile64

Total duration: 13 min 33 sec

Click to see traceback logs 
remote: Enumerating objects: 64, done.        
remote: Counting objects:   1% (1/64)        
remote: Counting objects:   3% (2/64)        
remote: Counting objects:   4% (3/64)        
remote: Counting objects:   6% (4/64)        
remote: Counting objects:   7% (5/64)        
remote: Counting objects:   9% (6/64)        
remote: Counting objects:  10% (7/64)        
remote: Counting objects:  12% (8/64)        
remote: Counting objects:  14% (9/64)        
remote: Counting objects:  15% (10/64)        
remote: Counting objects:  17% (11/64)        
remote: Counting objects:  18% (12/64)        
remote: Counting objects:  20% (13/64)        
remote: Counting objects:  21% (14/64)        
remote: Counting objects:  23% (15/64)        
remote: Counting objects:  25% (16/64)        
remote: Counting objects:  26% (17/64)        
remote: Counting objects:  28% (18/64)        
remote: Counting objects:  29% (19/64)        
remote: Counting objects:  31% (20/64)        
remote: Counting objects:  32% (21/64)        
remote: Counting objects:  34% (22/64)        
remote: Counting objects:  35% (23/64)        
remote: Counting objects:  37% (24/64)        
remote: Counting objects:  39% (25/64)        
remote: Counting objects:  40% (26/64)        
remote: Counting objects:  42% (27/64)        
remote: Counting objects:  43% (28/64)        
remote: Counting objects:  45% (29/64)        
remote: Counting objects:  46% (30/64)        
remote: Counting objects:  48% (31/64)        
remote: Counting objects:  50% (32/64)        
remote: Counting objects:  51% (33/64)        
remote: Counting objects:  53% (34/64)        
remote: Counting objects:  54% (35/64)        
remote: Counting objects:  56% (36/64)        
remote: Counting objects:  57% (37/64)        
remote: Counting objects:  59% (38/64)        
remote: Counting objects:  60% (39/64)        
remote: Counting objects:  62% (40/64)        
remote: Counting objects:  64% (41/64)        
remote: Counting objects:  65% (42/64)        
remote: Counting objects:  67% (43/64)        
remote: Counting objects:  68% (44/64)        
remote: Counting objects:  70% (45/64)        
remote: Counting objects:  71% (46/64)        
remote: Counting objects:  73% (47/64)        
remote: Counting objects:  75% (48/64)        
remote: Counting objects:  76% (49/64)        
remote: Counting objects:  78% (50/64)        
remote: Counting objects:  79% (51/64)        
remote: Counting objects:  81% (52/64)        
remote: Counting objects:  82% (53/64)        
remote: Counting objects:  84% (54/64)        
remote: Counting objects:  85% (55/64)        
remote: Counting objects:  87% (56/64)        
remote: Counting objects:  89% (57/64)        
remote: Counting objects:  90% (58/64)        
remote: Counting objects:  92% (59/64)        
remote: Counting objects:  93% (60/64)        
remote: Counting objects:  95% (61/64)        
remote: Counting objects:  96% (62/64)        
remote: Counting objects:  98% (63/64)        
remote: Counting objects: 100% (64/64)        
remote: Counting objects: 100% (64/64), done.        
remote: Compressing objects:   3% (1/32)        
remote: Compressing objects:   6% (2/32)        
remote: Compressing objects:   9% (3/32)        
remote: Compressing objects:  12% (4/32)        
remote: Compressing objects:  15% (5/32)        
remote: Compressing objects:  18% (6/32)        
remote: Compressing objects:  21% (7/32)        
remote: Compressing objects:  25% (8/32)        
remote: Compressing objects:  28% (9/32)        
remote: Compressing objects:  31% (10/32)        
remote: Compressing objects:  34% (11/32)        
remote: Compressing objects:  37% (12/32)        
remote: Compressing objects:  40% (13/32)        
remote: Compressing objects:  43% (14/32)        
remote: Compressing objects:  46% (15/32)        
remote: Compressing objects:  50% (16/32)        
remote: Compressing objects:  53% (17/32)        
remote: Compressing objects:  56% (18/32)        
remote: Compressing objects:  59% (19/32)        
remote: Compressing objects:  62% (20/32)        
remote: Compressing objects:  65% (21/32)        
remote: Compressing objects:  68% (22/32)        
remote: Compressing objects:  71% (23/32)        
remote: Compressing objects:  75% (24/32)        
remote: Compressing objects:  78% (25/32)        
remote: Compressing objects:  81% (26/32)        
remote: Compressing objects:  84% (27/32)        
remote: Compressing objects:  87% (28/32)        
remote: Compressing objects:  90% (29/32)        
remote: Compressing objects:  93% (30/32)        
remote: Compressing objects:  96% (31/32)        
remote: Compressing objects: 100% (32/32)        
remote: Compressing objects: 100% (32/32), done.        
remote: Total 36 (delta 30), reused 7 (delta 4), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '0ca67e6313c11263ecaef7ce182308eeb5aa6814'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 0ca67e6313 GH-84559: Deprecate fork being the multiprocessing default. (#100618)
Switched to and reset branch 'main'

make: *** [Makefile:1912: buildbottest] Error 3
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead added a commit to gpshead/cpython that referenced this pull request Feb 3, 2023
@gpshead
pythongh-84559: Remove the new mp warning, too disruptive.
20cec00 
This reverts the core of python#100618 while leaving relevant documentation
improvements and minor refactorings in place.
@gpshead gpshead mentioned this pull request Feb 3, 2023
gh-84559: Remove the new multiprocessing warning, too disruptive. #101551
Merged
gpshead added a commit that referenced this pull request Feb 3, 2023
@gpshead
gh-84559: Remove the new multiprocessing warning, too disruptive. (#1…
d4c410f 
…01551)

This reverts the core of #100618 while leaving relevant documentation
improvements and minor refactorings in place.
russellb added a commit to russellb/vllm that referenced this pull request Sep 18, 2024
@russellb
[Core] Change multiproc method default to spawn
515c7ae 
The default of `fork` is known to be problematic. Python itself is
changing the default to `spawn`. The new default is expected to be in
place for Python 3.14.

Python references for the change to the default:

* python/cpython#84559
* python/cpython#100618

We also have several places where this option had to be set to `spawn`
to make tests work. The AMD code even checks and overrides the value if
it's not set to `spawn`.

Simplify things for everyone and just default to `spawn`, but leave the
option in place just in case, at least for now.

Signed-off-by: Russell Bryant <[email protected]>
@russellb russellb mentioned this pull request Sep 18, 2024
[Core] Change multiproc method default to spawn vllm-project/vllm#8576
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gvanrossum gvanrossum gvanrossum left review comments
@FFY00 FFY00 FFY00 left review comments
@kumaraditya303 kumaraditya303 kumaraditya303 left review comments
@Yhg1s Yhg1s Yhg1s approved these changes
@vsajip vsajip Awaiting requested review from vsajip vsajip is a code owner
@1st1 1st1 Awaiting requested review from 1st1 1st1 is a code owner
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

stdlib Standard Library Python modules in the Lib/ directory topic-multiprocessing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@gpshead @hugovk @bedevere-bot @gvanrossum @Yhg1s @FFY00 @kumaraditya303
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.