Description

assignee = None
closed_at = None
created_at = <Date 2022-02-18.15:16:10.733>
labels = ['library', '3.9', 'performance']
title = 'ProcessPoolExecutor exception memory leak'
updated_at = <Date 2022-02-18.15:23:21.328>
user = 'https://github.com/themylogin'

activity = <Date 2022-02-18.15:23:21.328>
actor = 'python-dev'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-02-18.15:16:10.733>
creator = 'themylogin'
dependencies = []
files = ['50628']
hgrepos = []
issue_num = 46787
keywords = ['patch']
message_count = 1.0
messages = ['413485']
nosy_count = 2.0
nosy_names = ['python-dev', 'themylogin']
pr_nums = ['31408']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46787'
versions = ['Python 3.9']

Activity

1. themylogin commented on Feb 18, 2022

   

   themyloginmannequin

   on Feb 18, 2022

   MannequinAuthor

   More actions

   If an exception occurs in ProcessPoolExecutor work item, all the exception frame local variables are not garbage collected (or are garbage collected too lately) because they are referenced by the exception's traceback.

   Attached file is a test case. With unpatched python 3.9 (debian bullseye) it prints:

   root@truenas[~/freenas/freenas]# python test.py
   At iteration 0 memory usage is 226070528
   At iteration 1 memory usage is 318763008
   At iteration 2 memory usage is 318509056
   At iteration 3 memory usage is 321662976
   At iteration 4 memory usage is 321404928
   At iteration 5 memory usage is 324292608
   At iteration 6 memory usage is 324296704
   At iteration 7 memory usage is 326922240
   At iteration 8 memory usage is 326922240
   At iteration 9 memory usage is 329543680

   With the proposed patch there is no memory usage growth:

   At iteration 0 memory usage is 226410496
   At iteration 1 memory usage is 226451456
   At iteration 2 memory usage is 226451456
   At iteration 3 memory usage is 226443264
   At iteration 4 memory usage is 226443264
   At iteration 5 memory usage is 226435072
   At iteration 6 memory usage is 226426880
   At iteration 7 memory usage is 226426880
   At iteration 8 memory usage is 226435072
   At iteration 9 memory usage is 226426880
2. themylogin

   added

   3.9 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   performancePerformance or resource usagePerformance or resource usage

   on Feb 18, 2022
3. ezio-melotti

   transferred this issue fromon Apr 10, 2022
4. kumaraditya303

   closed this as completedon May 6, 2022
5. clchiou commented on Sep 25, 2022

   

   clchiou

   on Sep 25, 2022

   More actions

   I wonder if this patch also needs to be applied to Lib/multiprocessing/pool.py as it appears to me that Lib/concurrent/futures/process.py and Lib/multiprocessing/pool.py are quite similar in this aspect: sending exceptions with (stringified) traceback across process boundary. What do you think? @themylogin

Issue actions