GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-90549: Fix leak of global named resources using multiprocessing spawn by jxdabc · Pull Request #30617 · python/cpython

gh-90549: Fix leak of global named resources using multiprocessing spawn #30617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

pitrou merged 3 commits into python:main from jxdabc:fix-issue-46391

Jun 9, 2022

Uh oh!

There was an error while loading. Please reload this page.

Merged

gh-90549: Fix leak of global named resources using multiprocessing spawn #30617

Show file tree

Hide file tree

Changes from all commits

Commits

Show all changes

3 commits

Select commit Hold shift + click to select a range

f202fff

bpo-46391: Library multiprocess leaks named resources.

update test code and news

Jun 7, 2022

File filter

Filter by extension .py (3) .rst (1) All 2 file types selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

There are no files selected for viewing

14 changes: 14 additions & 0 deletions 14 Lib/multiprocessing/context.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -223,6 +223,10 @@ class Process(process.BaseProcess):
		def _Popen(process_obj):
		return _default_context.get_context().Process._Popen(process_obj)

		@staticmethod
		def _after_fork():
Comment thread pitrou marked this conversation as resolved. Resolved Uh oh! There was an error while loading. Please reload this page. Comment thread pitrou marked this conversation as resolved. Resolved Uh oh! There was an error while loading. Please reload this page.
		return _default_context.get_context().Process._after_fork()

		class DefaultContext(BaseContext):
		Process = Process

Expand Down Expand Up		@@ -283,6 +287,11 @@ def _Popen(process_obj):
		from .popen_spawn_posix import Popen
		return Popen(process_obj)

		@staticmethod
		def _after_fork():
		# process is spawned, nothing to do
		pass

		class ForkServerProcess(process.BaseProcess):
		_start_method = 'forkserver'
		@staticmethod
Expand Down Expand Up		@@ -326,6 +335,11 @@ def _Popen(process_obj):
		from .popen_spawn_win32 import Popen
		return Popen(process_obj)

		@staticmethod
		def _after_fork():
		# process is spawned, nothing to do
		pass

		class SpawnContext(BaseContext):
		_name = 'spawn'
		Process = SpawnProcess
Expand Down

10 changes: 8 additions & 2 deletions 10 Lib/multiprocessing/process.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -304,8 +304,7 @@ def _bootstrap(self, parent_sentinel=None):
		if threading._HAVE_THREAD_NATIVE_ID:
		threading.main_thread()._set_native_id()
		try:
		util._finalizer_registry.clear()
		util._run_after_forkers()
		self._after_fork()
		finally:
		# delay finalization of the old process object until after
		# _run_after_forkers() is executed
Expand Down Expand Up		@@ -336,6 +335,13 @@ def _bootstrap(self, parent_sentinel=None):

		return exitcode

		@staticmethod
		def _after_fork():
		from . import util
		util._finalizer_registry.clear()
		util._run_after_forkers()


		#
		# We subclass bytes to avoid accidental transmission of auth keys over network
		#
Expand Down

30 changes: 30 additions & 0 deletions 30 Lib/test/_test_multiprocessing.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -5,6 +5,7 @@
		import unittest
		import unittest.mock
		import queue as pyqueue
		import textwrap
		import time
		import io
		import itertools
Expand Down Expand Up		@@ -5760,6 +5761,35 @@ def test_namespace(self):
		self.run_worker(self._test_namespace, o)


		class TestNamedResource(unittest.TestCase):
		def test_global_named_resource_spawn(self):
		#
		# gh-90549: Check that global named resources in main module
Comment thread jxdabc marked this conversation as resolved. Resolved Uh oh! There was an error while loading. Please reload this page.
		# will not leak by a subprocess, in spawn context.
		#
		testfn = os_helper.TESTFN
		self.addCleanup(os_helper.unlink, testfn)
		with open(testfn, 'w', encoding='utf-8') as f:
		f.write(textwrap.dedent('''\
		import multiprocessing as mp

		ctx = mp.get_context('spawn')

		global_resource = ctx.Semaphore()

		def submain(): pass

		if __name__ == '__main__':
		p = ctx.Process(target=submain)
		p.start()
		p.join()
		'''))
		rc, out, err = test.support.script_helper.assert_python_ok(testfn)
		# on error, err = 'UserWarning: resource_tracker: There appear to
		# be 1 leaked semaphore objects to clean up at shutdown'
		self.assertEqual(err, b'')


		class MiscTestCase(unittest.TestCase):
		def test__all__(self):
		# Just make sure names in not_exported are excluded
Expand Down

2 changes: 2 additions & 0 deletions 2 Misc/NEWS.d/next/Library/2022-06-07-14-53-46.gh-issue-90549.T4FMKY.rst

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		Fix a multiprocessing bug where a global named resource (such as a semaphore)
		could leak when a child process is spawned (as opposed to forked).

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.