Description

brandtbucher

opened on Nov 16, 2019

Issue body actions

BPO	38823
Nosy	@Yhg1s, @vstinner, @asvetlov, @serhiy-storchaka, @miss-islington, @brandtbucher, @shihai1991, @erlend-aasland
PRs	bpo-38823: Clean up refleaks in _asyncio initialization. #17195 [3.8] bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195) #17196 [3.7] bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195) #17197 bpo-38823: Clean up refleaks in _contextvars initialization. #17198 [3.7] bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198) #17199 [3.8] bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198) #17200 bpo-38823: Clean up refleaks in _tkinter initialization. #17206 bpo-38823: Clean up _statistics initialization. #17215 bpo-38823: Clean up _xxtestfuzz initialization. #17216 [3.8] bpo-38823: Clean up refleaks in _tkinter initialization. (GH-17206) #17226 [3.7] bpo-38823: Clean up refleaks in _tkinter initialization. (GH-17206) #17227 bpo-38823: Clean up refleak in _tracemalloc initialization. #17235 bpo-38823: Clean up refleak in fcntl initialization. #17236 bpo-38823: Clean up refleaks in faulthandler initialization. #17250 bpo-38823: Clean up refleak in marshal initialization. #17260 [3.8] bpo-38823: Fix refleaks in faulthandler init error path on Windows (GH-17250) #17263 [3.7] bpo-38823: Fix refleaks in faulthandler init error path on Windows (GH-17250) #17264 [3.8] bpo-38823: Fix refleak in marshal init error path (GH-17260) #17271 [3.7] bpo-38823: Fix refleak in marshal init error path (GH-17260) #17272 bpo-38823: Clean up refleaks in _ast initialization. #17276 [3.8] bpo-38823: Fix refleak in marshal init error path (GH-17260) #17280 [3.7] bpo-38823: Fix refleak in marshal init error path (GH-17260) #17281 [3.8] bpo-38823: Fix refleak in _tracemalloc init error handling (GH-17235) #17282 [3.7] bpo-38823: Fix refleak in _tracemalloc init error handling (GH-17235) #17283 bpo-38823: Add a private _PyModule_StealObject API. #17298 bpo-1635741: Calling Py_INCREF() after PyModule_AddObject() success to run #18365 bpo-38823: Fix refleaks in _ctypes extension init #23247 bpo-38823: Always build _ctypes with wchar_t #23248 bpo-38823: Fix compiler warning in _ctypes on Windows #23258

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brandtbucher'
closed_at = None
created_at = <Date 2019-11-16.18:54:25.878>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'Improve stdlib module initialization error handling.'
updated_at = <Date 2020-11-13.13:44:17.121>
user = 'https://github.com/brandtbucher'

bugs.python.org fields:

activity = <Date 2020-11-13.13:44:17.121>
actor = 'vstinner'
assignee = 'brandtbucher'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-11-16.18:54:25.878>
creator = 'brandtbucher'
dependencies = []
files = []
hgrepos = []
issue_num = 38823
keywords = ['patch']
message_count = 30.0
messages = ['356764', '356765', '356773', '356774', '356775', '356781', '356782', '356783', '356809', '356820', '356872', '356873', '356874', '356984', '357002', '357003', '357004', '357014', '357019', '357029', '357044', '357046', '357047', '357050', '357051', '357114', '361486', '380822', '380824', '380883']
nosy_count = 8.0
nosy_names = ['twouters', 'vstinner', 'asvetlov', 'serhiy.storchaka', 'miss-islington', 'brandtbucher', 'shihai1991', 'erlendaasland']
pr_nums = ['17195', '17196', '17197', '17198', '17199', '17200', '17206', '17215', '17216', '17226', '17227', '17235', '17236', '17250', '17260', '17263', '17264', '17271', '17272', '17276', '17280', '17281', '17282', '17283', '17298', '18365', '23247', '23248', '23258']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38823'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

• PR: gh-83004: Clean up refleak in _io initialisation #98840

• PR: gh-83004: Clean up refleak in _pickle initialisation #98841

• PR: gh-83004: Clean up refleak in _zoneinfo initialisation #98842

• PR: gh-83004: Clean up refleaks in _decimal initialisation #99043

• PR: gh-83004: Clean up refleaks in ssl, socket initialisation #99044

• PR: [3.11] gh-83004: Clean up refleak in _io initialisation (GH-98840) #99059

• PR: [3.10] gh-83004: Clean up refleak in _io initialisation (GH-98840) #99060

• PR: [3.11] gh-83004: Clean up refleak in _zoneinfo initialisation (GH-98842) #99163

• PR: [3.10] gh-83004: Clean up refleak in _zoneinfo initialisation (GH-98842) #99164

• PR: [3.11] gh-83004: Clean up refleak in _pickle initialisation (GH-98841) #99165

• PR: [3.10] gh-83004: Clean up refleak in _pickle initialisation (GH-98841) #99166

• PR: gh-83004: Harden _socket init #103261

• PR: gh-83004: Harden msvcrt init #103383

• PR: gh-83004: Harden winsound init #103385

• PR: gh-83004: Harden winreg init #103386

• PR: gh-83004: Harden msvcrt further #103420

Activity

1. brandtbucher commented on Nov 16, 2019

   

   brandtbucher

   on Nov 16, 2019

   MemberAuthor

   More actions

   Many of the C stdlib modules can benefit from improved error handling during initialization. I've now had two PRs where the authors had reference leaks on error conditions, but defended their decisions by pointing to examples of similar idioms all over the stdlib.

   The problems fall into two related categories, mostly:

   • Not DECREF'ing the new module object on failure.
   • Not handling errors raised by the PyModule_Add* family of functions... specifically, the weird steal-on-success semantics of PyModule_AddObject. I've already improved the docs for this, so we should see the issue less, but our own code should still be fixed.

   I intend to turn this into a longer term project. I'll be working my way through these modules bit-by-bit over time, using this issue to track all of them (since there are a few dozen cases). I'd rather not make one huge one that spams all of the code owners and is impossible to review.

   If anybody want to make themselves available to review/merge these as I go along, that would be great! Many of the ones I'll start with are just adding trivial DECREFs to the more popular modules (_asyncio, _contextvars, _functools, _random, _warnings, etc...).
2. brandtbucher

   added

   3.9 (EOL)end of lifeend of life

   on Nov 16, 2019
3. brandtbucher

   self-assigned this

   on Nov 16, 2019
4. brandtbucher

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Nov 16, 2019
5. brandtbucher commented on Nov 16, 2019

   

   brandtbucher

   on Nov 16, 2019

   MemberAuthor

   More actions

   How do others feel about the creation of a new private API? It would keep these diffs smaller and ease refactoring... and it would probably be good to have around anyways:

   /* Like PyModule_AddObject, but steals o on success AND failure. */

   int
   _PyModule_StealObject(PyObject *m, const char *name, PyObject *o)
   {
       if (PyModule_AddObject(m, name, o) < 0) {
           Py_XDECREF(o);
           return -1;
       }
       return 0;
   }
6. miss-islington commented on Nov 16, 2019

   

   miss-islington

   on Nov 16, 2019

   Contributor

   More actions

   New changeset c3f6bdc by Miss Islington (bot) (Brandt Bucher) in branch 'master':
   bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195)
   c3f6bdc
7. miss-islington commented on Nov 16, 2019

   

   miss-islington

   on Nov 16, 2019

   Contributor

   More actions

   New changeset 48f4f75 by Miss Islington (bot) in branch '3.8':
   bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195)
   48f4f75
8. miss-islington commented on Nov 16, 2019

   

   miss-islington

   on Nov 16, 2019

   Contributor

   More actions

   New changeset 825e91b by Miss Islington (bot) in branch '3.7':
   bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195)
   825e91b
9. miss-islington commented on Nov 16, 2019

   

   miss-islington

   on Nov 16, 2019

   Contributor

   More actions

   New changeset 143a97f by Miss Islington (bot) (Brandt Bucher) in branch 'master':
   bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198)
   143a97f
10. miss-islington commented on Nov 17, 2019

    

    miss-islington

    on Nov 17, 2019

    Contributor

    More actions

    New changeset 8a334af by Miss Islington (bot) in branch '3.7':
    bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198)
    8a334af
11. miss-islington commented on Nov 17, 2019

    

    miss-islington

    on Nov 17, 2019

    Contributor

    More actions

    New changeset 1fe79a4 by Miss Islington (bot) in branch '3.8':
    bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198)
    1fe79a4
12. asvetlov commented on Nov 17, 2019

    

    asvetlov

    on Nov 17, 2019

    Contributor

    More actions

    Is anything left to be done for the issue?
13. asvetlov

    added

    3.7 (EOL)end of lifeend of life

    3.8 (EOL)end of lifeend of life

    on Nov 17, 2019
14. 62 remaining items
15. erlend-aasland

    added 2 commits that reference this issue on Apr 10, 2023

    pythongh-83004: Harden msvcrt further

    88bb327

    gh-83004: Harden winreg init (#103386)

    5ed2f19
16. warsaw

    added 4 commits that reference this issue on Apr 11, 2023

    pythongh-83004: Harden _socket init (pythonGH-103261)

    e7d348e

    pythongh-83004: Harden winsound init (python#103385)

    84ac912

    pythongh-83004: Harden `msvcrt` init (python#103383)

    1ce1aad

    pythongh-83004: Harden winreg init (python#103386)

    e5d9087
17. aisk

    added 3 commits that reference this issue on Apr 18, 2023

    pythongh-83004: Harden winsound init (python#103385)

    4ea911f

    pythongh-83004: Harden `msvcrt` init (python#103383)

    ce07012

    pythongh-83004: Harden winreg init (python#103386)

    e3f10e2
18. erlend-aasland

    added a commit that references this issue on Apr 19, 2023

    gh-83004: Harden msvcrt further (#103420)

    bd2ed06
19. carljm

    added a commit that references this issue on Apr 20, 2023

    Merge branch 'main' into superopt

    5c0a21c
20. erlend-aasland commented on Feb 8, 2024

    

    erlend-aasland

    on Feb 8, 2024

    Contributor

    More actions

    A lot of issues were fixed by Serhiy in #106858. Are there more actionable items left? (I did not check the _decimal PR yet.)
21. vstinner commented on Feb 8, 2024

    

    vstinner

    on Feb 8, 2024

    Member

    More actions

    I suggest to close the issue which has a long history, and maybe create a new issue with a better described scope (list C extensions) if more C extensions need to be fixed.

    Reacted by Erlend E. Aasland
22. erlend-aasland commented on Feb 9, 2024

    

    erlend-aasland

    on Feb 9, 2024

    Contributor

    More actions

    All right, there are no open PRs left (they were all obsoleted by Serhiy's recent work in #106858).

    I suggest to close the issue which has a long history, and maybe create a new issue with a better described scope (list C extensions) if more C extensions need to be fixed.

    I agree; let's close this :) Thanks to Brandt for starting this effort in the first place, and thanks to all who improved extension module error handling!
23. erlend-aasland

    closed this as completedon Feb 9, 2024

Issue actions