Description

assignee = None
closed_at = <Date 2022-01-27.13:42:43.202>
created_at = <Date 2007-01-15.10:26:05.000>
labels = ['expert-subinterpreters', 'interpreter-core', '3.11', 'performance']
title = "Py_Finalize() doesn't clear all Python objects at exit"
updated_at = <Date 2022-02-25.11:34:18.230>
user = 'https://bugs.python.org/kylotan'

activity = <Date 2022-02-25.11:34:18.230>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-27.13:42:43.202>
closer = 'vstinner'
components = ['Interpreter Core', 'Subinterpreters']
creation = <Date 2007-01-15.10:26:05.000>
creator = 'kylotan'
dependencies = []
files = []
hgrepos = []
issue_num = 1635741
keywords = ['patch']
message_count = 185.0
messages = ['61054', '110895', '111024', '130729', '248761', '355187', '355189', '355191', '355193', '355194', '355201', '359342', '359482', '359830', '360063', '361427', '361428', '361466', '361798', '362068', '362124', '362143', '362144', '362195', '363935', '363936', '363937', '363940', '363941', '364234', '364330', '364379', '364463', '364521', '364609', '364656', '364833', '364836', '364845', '364871', '364883', '364906', '364909', '364951', '364952', '364968', '364971', '364972', '364973', '364975', '364987', '365008', '365017', '365043', '365386', '365388', '365484', '365584', '365611', '367686', '367796', '368096', '368318', '370561', '370763', '371119', '371569', '371683', '372061', '372077', '372096', '372102', '372103', '372797', '372912', '372938', '372942', '375169', '375272', '375286', '375287', '375288', '375290', '375311', '375567', '376218', '376221', '376240', '376288', '376360', '376461', '376474', '376477', '376480', '376493', '376500', '376552', '376554', '376555', '376556', '376566', '376567', '376571', '376572', '376616', '376687', '376993', '377253', '377364', '377365', '377523', '378115', '378685', '379280', '379287', '379667', '380216', '380270', '380280', '380317', '380318', '380331', '380332', '380340', '380346', '380351', '380733', '380758', '380854', '380857', '381124', '381300', '381317', '381318', '381351', '381352', '381376', '381379', '381391', '381392', '381395', '381396', '381400', '381411', '381423', '381426', '381431', '381432', '381434', '381476', '381477', '381571', '381572', '383155', '383168', '383276', '383283', '383285', '383287', '383454', '383636', '383785', '383799', '383801', '383844', '383847', '383880', '384039', '384042', '384326', '384352', '384646', '384978', '384987', '385010', '388728', '410809', '411116', '411278', '411311', '411872', '411961', '411997', '413938', '413979']
nosy_count = 26.0
nosy_names = ['amaury.forgeotdarc', 'orsenthil', 'kylotan', 'vstinner', 'christian.heimes', 'tlesher', 'phsilva', 'petr.viktorin', 'ysj.ray', 'santoso.wijaya', 'lukasz.langa', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'pconnell', 'isoschiz', 'Jim Fasarakis-Hilliard', 'corona10', 'miss-islington', 'shihai1991', 'erlendaasland', 'koubaa', 'h-vetinari', 'TCsaba', 'kumaraditya', 'CharlieZhao']
pr_nums = ['17835', '17883', '18030', '18032', '18049', '18050', '18065', '18066', '18358', '18365', '18374', '18404', '18486', '18608', '18613', '19012', '19015', '19018', '19022', '19021', '19044', '19057', '19069', '19071', '19074', '19084', '19100', '19107', '19122', '19128', '19135', '19140', '19150', '19151', '19242', '19243', '19252', '19307', '19382', '19459', '19798', '19822', '19907', '19923', '20540', '20763', '20808', '20848', '20920', '21087', '21168', '21189', '21269', '21270', '21294', '21319', '21371', '21375', '21378', '21418', '21605', '21818', '21855', '21856', '21896', '21902', '21985', '21986', '21995', '22003', '22049', '22050', '22051', '22131', '22134', '22139', '22145', '22148', '22149', '22164', '22165', '22220', '22222', '22240', '22242', '22328', '22415', '22478', '22489', '22490', '22712', '22713', '22838', '22880', '23091', '23122', '23146', '23147', '23148', '23151', '23170', '23139', '23188', '23234', '23236', '23304', '23358', '23359', '23360', '23361', '23376', '23377', '23378', '23379', '23381', '23390', '23393', '23398', '23401', '23402', '23404', '23406', '23409', '23462', '23443', '23443', '23535', '23598', '23793', '23811', '23972', '24095', '24175', '24205', '30522', '30525', '30744', '31555', '31560']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue1635741'
versions = ['Python 3.11']

Activity

1. kylotan commented on Jan 15, 2007

   

   kylotanmannequin

   on Jan 15, 2007

   MannequinAuthor

   More actions

   This C code:

   #include <Python.h>
   int main(int argc, char *argv[])
   {
       Py_Initialize(); Py_Finalize();
       Py_Initialize(); Py_Finalize();
       Py_Initialize(); Py_Finalize();
       Py_Initialize(); Py_Finalize();
       Py_Initialize(); Py_Finalize();
       Py_Initialize(); Py_Finalize();
       Py_Initialize(); Py_Finalize();
   }

   Produces this output:
   [7438 refs]
   [7499 refs]
   [7550 refs]
   [7601 refs]
   [7652 refs]
   [7703 refs]
   [7754 refs]

   A similar program configured to call the Py_Initialize()/Py_Finalize() 1000 times ends up with:
   ...
   [58295 refs]
   [58346 refs]
   [58397 refs]

   This is with a fresh debug build of Python 2.5.0 on Windows XP, using Visual C++ 2003.
2. kylotan

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Jan 15, 2007
3. devdanzin

   added

   performancePerformance or resource usagePerformance or resource usage

   on Mar 30, 2009
4. BreamoreBoy commented on Jul 20, 2010

   

   BreamoreBoymannequin

   on Jul 20, 2010

   Mannequin

   More actions

   Does the title of this issue accurately reflect the current status of the Python interpreter?
5. amauryfa commented on Jul 21, 2010

   

   amauryfa

   on Jul 21, 2010

   Contributor

   More actions

   Yes, some objects are not cleaned in finalization.
   This is not a problem in usual cases though, when the interpreter is
   started only once.
6. ysjray commented on Mar 13, 2011

   

   ysjraymannequin

   on Mar 13, 2011

   Mannequin

   More actions

   Does the title of this issue accurately reflect the current status of the Python interpreter?

   Yes, here is the running result on current 3.3 latest code:
   [37182 refs]
   [39415 refs]
   [41607 refs]
   [43799 refs]
   [45991 refs]
   [48183 refs]
   [50375 refs]

   This seems to be a known bug that Py_Finalize() doesn't free all objects according doc http://docs.python.org/dev/c-api/init.html?highlight=py_finalize#Py_Finalize
7. AlexBudovski commented on Aug 18, 2015

   

   AlexBudovskimannequin

   on Aug 18, 2015

   Mannequin

   More actions

   Interestingly enough, some of the leaked memory came from the finalize routine itself! Here's one example:

   0:004> !heap -p -a 0x000000DB144346F0
   address 000000db144346f0 found in
   _HEAP @ db0cae0000
   HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
   000000db14434690 030a 0000 [00] 000000db144346c0 03074 - (busy)
   7ffc55628b04 ntdll!RtlpCallInterceptRoutine+0x0000000000000040
   7ffc555f9f36 ntdll!RtlAllocateHeap+0x0000000000079836
   7ffc2a60c4da ucrtbased!calloc_base+0x000000000000123a
   7ffc2a60c27d ucrtbased!calloc_base+0x0000000000000fdd
   7ffc2a60f34f ucrtbased!malloc_dbg+0x000000000000002f
   7ffc2a60fdde ucrtbased!malloc+0x000000000000001e
   5a5e6ef9 python36_d!_PyMem_RawMalloc+0x0000000000000029
   5a5e78c7 python36_d!_PyMem_DebugAlloc+0x0000000000000087
   5a5e5e6f python36_d!_PyMem_DebugMalloc+0x000000000000001f
   5a5e7230 python36_d!PyMem_Malloc+0x0000000000000030
   5a582047 python36_d!new_keys_object+0x0000000000000077
   5a57f7c5 python36_d!dictresize+0x0000000000000085
   5a57a4b2 python36_d!PyDict_Merge+0x0000000000000112
   5a57bf33 python36_d!PyDict_Update+0x0000000000000023
   5a75fb1d python36_d!PyImport_Cleanup+0x000000000000045d
   5a778f9e python36_d!Py_Finalize+0x000000000000005e
8. vstinner commented on Oct 23, 2019

   

   vstinner

   on Oct 23, 2019

   Member

   More actions

   I tested on the master branch of Python:
   ---

   #include <Python.h>
   
   void func()
   {
       Py_Initialize(); Py_Finalize();
       Py_ssize_t cnt = _Py_GetRefTotal();
       printf("sys.gettotalrefcount(): %zd\n", cnt);
   }
   
   int main(int argc, char *argv[])
   {
       Py_SetProgramName(L"./_testembed");
       for (int i=0; i < 10; i++) {
           func();
       }
   }

   ---

   Each iteration leaks around 5,000 Python objects:
   ---
   sys.gettotalrefcount(): 15113
   sys.gettotalrefcount(): 19527
   sys.gettotalrefcount(): 23941
   sys.gettotalrefcount(): 28355
   sys.gettotalrefcount(): 32769
   sys.gettotalrefcount(): 37183
   sys.gettotalrefcount(): 41597
   sys.gettotalrefcount(): 46011
   sys.gettotalrefcount(): 50425
   sys.gettotalrefcount(): 54839
   ---
9. vstinner

   added

   3.9 (EOL)end of lifeend of life

   on Oct 23, 2019
10. vstinner

    changed the title [-]Interpreter seems to leak references after finalization[/-] [+]Py_Finalize() doesn't clear all Python objects at exit[/+] on Oct 23, 2019
11. vstinner

    added

    3.9 (EOL)end of lifeend of life

    on Oct 23, 2019
12. vstinner

    changed the title [-]Interpreter seems to leak references after finalization[/-] [+]Py_Finalize() doesn't clear all Python objects at exit[/+] on Oct 23, 2019
13. vstinner commented on Oct 23, 2019

    

    vstinner

    on Oct 23, 2019

    Member

    More actions

    I marked bpo-6741 as a duplicate of this issue.
14. 199 remaining items

    Load more actions

15. vstinner commented on Jan 27, 2022

    

    vstinner

    on Jan 27, 2022

    Member

    More actions

    Thanks to recent enhancements, epecially in bpo-46417, the last memory blocks are now released at Python exit! The initial issue has been fixed!!! This bug was 15 years old! Current main branch:

    $ ./python -I -X showrefcount -c pass
    [-5 refs, 0 blocks]

    "0 blocks" means that Python no longer leaks memory at exit. You can use Valgrind to check it ;-)

    The negative reference count is being discussed in bpo-46449.

    I close this issue since it has an insane history: too many pull requests, changes, etc.

    While the work is not 100% done (we need to convert remaining static types to heap types, and convert extensions to multi-phase init), I prefer to address remaining issues in (existing or new) separated issues. See for example the bpo-40077: "Convert static types to heap types: use PyType_FromSpec()".

    I would like to thank everybody who helped to fix this issue! It's hard to list all names since this issue is a meta issues made of many sub-issues which also have sub-issues. Authors of PEP-573 and PEP-630 also helped fixing this issue!
16. vstinner

    added

    3.11only security fixesonly security fixes

    and removed

    3.10 (EOL)end of lifeend of life

    on Jan 27, 2022
17. vstinner

    closed this as completedon Jan 27, 2022
18. vstinner

    added

    3.11only security fixesonly security fixes

    and removed

    3.10 (EOL)end of lifeend of life

    on Jan 27, 2022
19. vstinner

    closed this as completedon Jan 27, 2022
20. vstinner commented on Jan 28, 2022

    

    vstinner

    on Jan 28, 2022

    Member

    More actions

    I marked bpo-35774 as a duplicate of this issue.
21. vstinner commented on Jan 28, 2022

    

    vstinner

    on Jan 28, 2022

    Member

    More actions

    The negative refcount issue has been fixed ;-)

    At commit 9a24127, Python now leaks exactly 0 reference count and 0 memory block at exit:

    $ ./python -I -X showrefcount -c pass
    [0 refs, 0 blocks]
22. vstinner commented on Feb 24, 2022

    

    vstinner

    on Feb 24, 2022

    Member

    More actions

    New changeset c9c178f by Victor Stinner in branch 'main':
    bpo-1635741: test_embed cheks that Python does not leak (GH-31555)
    c9c178f
23. vstinner commented on Feb 25, 2022

    

    vstinner

    on Feb 25, 2022

    Member

    More actions

    New changeset 4657bf7 by Victor Stinner in branch 'main':
    bpo-1635741: Fix winreg reference leaks (GH-31560)
    4657bf7
24. ezio-melotti

    transferred this issue fromon Apr 10, 2022
25. AlexSoft73 commented on Sep 19, 2022

    

    AlexSoft73

    on Sep 19, 2022

    More actions

    I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call
    Py_IsInitialized();
    Py_FinalizeEx();
26. vstinner commented on Sep 19, 2022

    

    vstinner

    on Sep 19, 2022

    Member

    More actions

    This issue is closed, please open a new issue.
27. vstinner commented on Sep 19, 2022

    

    vstinner

    on Sep 19, 2022

    Member

    More actions

    I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call

    Ah, it seems like it has been reported as a new issue: #96853

Issue actions