GeezerVison is showing github.com — Library multiprocess leaks named resources. · Issue #90549 · python/cpython
Library multiprocess leaks named resources. #90549Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-01-15.18:04:57.278>
labels = ['3.8', '3.9', '3.10', 'performance', '3.11', 'library']
title = 'Library multiprocess leaks named resources.'
updated_at = <Date 2022-02-18.19:39:09.294>
user = 'https://github.com/jxdabc' bugs.python.org fields:
activity = <Date 2022-02-18.19:39:09.294>
actor = 'BarkingBad'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-01-15.18:04:57.278>
creator = 'milestonejxd'
dependencies = []
files = ['50580']
hgrepos = []
issue_num = 46391
keywords = ['patch']
message_count = 5.0
messages = ['410654', '410657', '410853', '411456', '413504']
nosy_count = 8.0
nosy_names = ['pitrou', 'benjamin.peterson', 'sbt', 'yselivanov', 'davin', 'arhadthedev', 'milestonejxd', 'BarkingBad']
pr_nums = ['30617']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46391'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11'] Activityjxdabc commented on Jan 15, 2022 MannequinAuthor More actions Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python.
Many Repo users have encountered resource leak warnings with Python 3.9.
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5
I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.
To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )
import multiprocessing as mp
global_resource = mp.Semaphore()
def submain(): pass
if __name__ == '__main__':
p = mp.Process(target=submain)
p.start()
p.join()
Tested on macOS with Python 3.9.7
python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.
This bug will 100% reproduce when then main module uses named resources as global variables and uses spawn context, which is the case of Repo on macOS.
This is caused by multiprocess::BaseProcess::_bootstrap.
When a new process is started with multiprocessing.Process.start() in spawn context.
- The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
- Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
- multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.
When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.
And I uploaded a MR. arhadthedev commented on Jan 15, 2022 I added core devs related to multiprocessing into a nosy list so they got a notification and the PR will be evaluated and merged faster. FYI, the devs are Davin Potts and Antoine Pitrou (as per <https://devguide.python.org/experts/\>). jxdabc commented on Jan 18, 2022 MannequinAuthor More actions jxdabc commented on Jan 24, 2022 MannequinAuthor More actions update,
Confirmed to affect all versions above 3.8+.
It's really annoying.
Command line tools like Repo get a warning after every (success) command.
And mess up with shell.
-------------------------------------------------------------------
$ repo status .
project ksbox/ branch dev
$ /usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '$
$
$
$ repo sync .
Fetching: 100% (1/1), done in 1.184s
Garbage collecting: 100% (1/1), done in 0.013s
repo sync has finished successfully.
$ /usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
$
$
$
$ repo status .
project ksbox/ branch dev
$ /usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d ' added a commit that references this issue on Jun 9, 2022gh-90549: Fix leak of global named resources using multiprocessing sp… added 4 commits that reference this issue on Jun 9, 2022pythongh-90549: Fix leak of global named resources using multiprocess… pythongh-90549: Fix leak of global named resources using multiprocess… gh-90549: Fix leak of global named resources using multiprocessing sp… gh-90549: Fix leak of global named resources using multiprocessing sp… pitrou commented on Jun 10, 2022 Okay, this is fixed now. Thanks @jxdabc for the contribution! dturner commented on Aug 17, 2023 Which version(s) was this fix applied to? I'm still getting this error in the latest version of Python 3.11.4 when running repo status
/Users/donturner/homebrew/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Reacted by Pat Patterson, Redoubts, Tyler Ferrara and kosiek siennathesane commented on Sep 11, 2023 This is also happening in 3.11.5
/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d 'Reacted by Pat Patterson, Redoubts, Li Mu Zhi and kosiek 3DMiller commented on Mar 4, 2024 Has this been resolved? Issue on macOS running python 3.9.12. gpshead commented on Mar 4, 2024 3.10.6+ and 3.11.0+ contain PR #30617. The reproducer code listed in the opening message does not cause the Resource warning for me (including when explicitly setting the start method to 'spawn'). If you're still seeing this on recent Python versions, we need a simple way to reproduce it.
I recommend opening a new issue when you have one.
(Just because you see this error message and find your way here does not mean that Python itself is the cause -- it can also be an application bug, we currently lack sufficient information to say one way or the other.)
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
