GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — bpo-19675: Terminate processes if construction of a pool is failing. by JulienPalard · Pull Request #5614 · python/cpython

bpo-19675: Terminate processes if construction of a pool is failing. #5614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

JulienPalard merged 5 commits into python:master from JulienPalard:bpo-19675

Nov 4, 2018

Uh oh!

There was an error while loading. Please reload this page.

Merged

bpo-19675: Terminate processes if construction of a pool is failing. #5614

Show file tree

Hide file tree

Changes from all commits

Commits

Show all changes

5 commits

Select commit Hold shift + click to select a range

53c89fb

bpo-19675: Terminate processes if construction of a pool is failing.

JulienPalard Feb 10, 2018

a8b3924

bpo-19675: Add a test.

JulienPalard Apr 7, 2018

3bdbf79

Merge remote-tracking branch 'upstream/master' into bpo-19675

taleinat Sep 10, 2018

f569cae

bpo-19675: clean up and fix typos in test and NEWS

taleinat Sep 10, 2018

7670d4a

Merge remote-tracking branch 'upstream/master' into bpo-19675

taleinat Nov 4, 2018

File filter

Filter by extension .py (2) .rst (1) All 2 file types selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

There are no files selected for viewing

12 changes: 10 additions & 2 deletions 12 Lib/multiprocessing/pool.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -174,7 +174,15 @@ def __init__(self, processes=None, initializer=None, initargs=(),

		self._processes = processes
		self._pool = []
		self._repopulate_pool()
		try:
		self._repopulate_pool()
		except Exception:
		for p in self._pool:
		if p.exitcode is None:
		p.terminate()
		for p in self._pool:
		p.join()
		raise

		self._worker_handler = threading.Thread(
		target=Pool._handle_workers,
Expand Down Expand Up		@@ -251,10 +259,10 @@ def _repopulate_pool_static(ctx, Process, processes, pool, inqueue,
		initargs, maxtasksperchild,
		wrap_exception)
		)
		pool.append(w)
		w.name = w.name.replace('Process', 'PoolWorker')
		w.daemon = True
		w.start()
		pool.append(w)
		util.debug('added worker')

		@staticmethod
Expand Down

43 changes: 43 additions & 0 deletions 43 Lib/test/_test_multiprocessing.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -3,6 +3,7 @@
		#

		import unittest
		import unittest.mock
		import queue as pyqueue
		import contextlib
		import time
Expand Down Expand Up		@@ -4635,6 +4636,48 @@ def test_empty(self):
		proc.join()


		class TestPoolNotLeakOnFailure(unittest.TestCase):

		def test_release_unused_processes(self):
		# Issue #19675: During pool creation, if we can't create a process,
		# don't leak already created ones.
		will_fail_in = 3
		forked_processes = []

		class FailingForkProcess:
		def __init__(self, **kwargs):
		self.name = 'Fake Process'
		self.exitcode = None
		self.state = None
		forked_processes.append(self)

		def start(self):
		nonlocal will_fail_in
		if will_fail_in <= 0:
		raise OSError("Manually induced OSError")
		will_fail_in -= 1
		self.state = 'started'

		def terminate(self):
		self.state = 'stopping'

		def join(self):
		if self.state == 'stopping':
		self.state = 'stopped'

		def is_alive(self):
		return self.state == 'started' or self.state == 'stopping'

		with self.assertRaisesRegex(OSError, 'Manually induced OSError'):
		p = multiprocessing.pool.Pool(5, context=unittest.mock.MagicMock(
		Process=FailingForkProcess))
		p.close()
		p.join()
		self.assertFalse(
		any(process.is_alive() for process in forked_processes))



		class MiscTestCase(unittest.TestCase):
		def test__all__(self):
		# Just make sure names in blacklist are excluded
Expand Down

1 change: 1 addition & 0 deletions 1 Misc/NEWS.d/next/Library/2018-02-10-23-41-05.bpo-19675.-dj35-.rst

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		``multiprocessing.Pool`` no longer leaks processes if its initialization fails.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.