GeezerVison is showing github.com — Numerous refleaks on main · Issue #103879 · python/cpython
Numerous refleaks on main #103879Descriptionhttps://buildbot.python.org/all/#/release_status shows that the refleak buildbots are failing on current main. Example output on https://buildbot.python.org/all/#/builders/75/builds/745/steps/5/logs/warnings__111_.
Failing tests:
Details
test_functools leaked [1, 1, 2] references, sum=4
test_warnings leaked [4, 4, 4] references, sum=12
test_ipaddress leaked [6, 6, 6] references, sum=18
test_zoneinfo leaked [7, 6, 11] references, sum=24
FAIL: test_flock (__main__.FNTLEINTRTest.test_flock)
test_random leaked [1, 2, 4] references, sum=7
test_statistics leaked [175, 174, 175] references, sum=524
test_dataclasses leaked [3, 3, 2] references, sum=8
test_argparse leaked [69, 70, 76] references, sum=215
/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
test_tarfile leaked [17, 17, 16] references, sum=50
test_typing leaked [95, 121, 95] references, sum=311
test_typing leaked [1, 14, 1] memory blocks, sum=16
test_lib2to3 leaked [5, 8, 6] references, sum=19
test_shutil leaked [2, 2, 2] references, sum=6
test_unittest leaked [8, 12, 15] references, sum=35
test_logging leaked [3, 2, 1] references, sum=6
test__xxsubinterpreters leaked [46, 45, 45] references, sum=136
test_mailbox leaked [2, 3, 2] references, sum=7
test_socket leaked [24, 23, 23] references, sum=70
test_htmlparser leaked [2, 2, 2] references, sum=6
test_regrtest leaked [9, 12, 8] references, sum=29
test_datetime leaked [11, 11, 11] references, sum=33
test_nntplib leaked [1, 3, 2] references, sum=6
test_ftplib leaked [10, 13, 9] references, sum=32
test_interpreters leaked [54, 54, 54] references, sum=162
test_zipfile leaked [3, 4, 4] references, sum=11
test_sys_settrace leaked [6, 6, 6] references, sum=18
test_descr leaked [3, 65, 67] references, sum=135
test_compile leaked [7, 7, 7] references, sum=21
test_bytes leaked [2, 2, 2] references, sum=6
test_ast leaked [9, 9, 9] references, sum=27
test_poplib leaked [3, 3, 2] references, sum=8
test_types leaked [8, 50, 43] references, sum=101
test_concurrent_futures leaked [11, 12, 11] references, sum=34
test_genericalias leaked [2, 2, 3] references, sum=7
test_math leaked [32, 31, 33] references, sum=96
test_numeric_tower leaked [2, 1, 2] references, sum=5
test_capi leaked [99, 99, 99] references, sum=297
test_import leaked [170, 171, 168] references, sum=509
test_import leaked [71, 71, 70] memory blocks, sum=212
.test test_monitoring failed -- multiple errors occurred; run in verbose mode for details
test_selectors leaked [5, 6, 4] references, sum=15
test_email leaked [27, 26, 25] references, sum=78
test_subprocess leaked [1, 1, 1] references, sum=3
test_struct leaked [1, 1, 1] references, sum=3
test_weakref leaked [1, 1, 1] references, sum=3
test_imaplib leaked [2, 2, 1] references, sum=5
0:45:54 load avg: 0.13 Re-running failed tests in verbose mode
0:45:54 load avg: 0.13 Re-running test_functools in verbose mode
test_functools leaked [1, 1, 2] references, sum=4
0:45:55 load avg: 0.13 Re-running test_warnings in verbose mode
test_warnings leaked [4, 4, 4] references, sum=12
0:45:59 load avg: 0.20 Re-running test_ipaddress in verbose mode
test_ipaddress leaked [6, 6, 6] references, sum=18
0:46:00 load avg: 0.20 Re-running test_zoneinfo in verbose mode
test_zoneinfo leaked [7, 6, 11] references, sum=24
I saw similar leaks triggering the refleak buildbots on #103866 and #103764, before I realized the issue was probably on main.
Based on @sunmy2019's work in #103764 (comment), we're likely leaking references to the object.__setattr__ wrapper descriptor.
Linked PRs
ActivityJelleZijlstra commented on Apr 26, 2023 The failure on test.test_htmlparser bisects to
ef25febcf2ede92a03c5ea00a13e167e0b5cb274 is the first bad commit
commit ef25febcf2ede92a03c5ea00a13e167e0b5cb274
Author: Carl Meyer <[email protected]>
Date: Tue Apr 25 11:45:51 2023 -0600
gh-87729: specialize LOAD_SUPER_ATTR_METHOD (#103809)
@carljm carljm commented on Apr 26, 2023 Given that refleak buildbots passed on #103882, it seems this is fixed now. carljm commented on Apr 26, 2023 JelleZijlstra commented on Apr 26, 2023 JelleZijlstra commented on Apr 27, 2023 We might have another refleak on test_import, visible on main and on #103866 link. JelleZijlstra commented on Apr 27, 2023 I wasn't able to reproduce the new leak locally (tried both MacOS and Linux). It seems to happen only on one buildbot. sunmy2019 commented on Apr 28, 2023 The leak of test_import is still visible on that buildbot, and the issue persists for a long time
Visible at
https://buildbot.python.org/all/#/builders/205/builds/696
... (note: logs will be cleared for old tests, soon the log will not be visible)
Usually, rerunning the tests will let it pass. (Warning, no errors)
I suspect something changed in the test environment. sunmy2019 commented on Apr 28, 2023 sunmy2019 commented on Apr 28, 2023 @JelleZijlstra I can reproduce it with
./python -m test -j1 -R 3:3 test_import
Single processing will not cause leakage. -j is necessary. JelleZijlstra commented on Apr 28, 2023 Good catch, that reproduces it for me too. sunmy2019 commented on Apr 29, 2023 Current status:
Found most leakage is from test_import.SinglephaseInitTests.
ref change of type dict: +7
ref change of type float: +21
ref change of type getset_descriptor: +5
ref change of type int: +16
ref change of type str: ... (immortal str not counted)
ref change of type tuple: +7
ref change of type type: +12
ref change of type weakref.ReferenceType: +5
I took a quick look, ref change of type type: +12 contains many instances created by
state->error = PyErr_NewException("_testsinglephase.error", NULL, NULL);Modules/_testsinglephase.c:init_module is called several times during -j tests, but only once with the single-threaded one.
|
static int |
|
init_module(PyObject *module, module_state *state) |
|
{ |
|
if (PyModule_AddObjectRef(module, "error", state->error) != 0) { |
|
return -1; |
|
} |
|
if (PyModule_AddObjectRef(module, "int_const", state->int_const) != 0) { |
|
return -1; |
|
} |
|
if (PyModule_AddObjectRef(module, "str_const", state->str_const) != 0) { |
|
return -1; |
|
} |
|
|
|
double d = _PyTime_AsSecondsDouble(state->initialized); |
|
PyObject *initialized = PyFloat_FromDouble(d); |
|
if (initialized == NULL) { |
|
return -1; |
|
} |
|
if (PyModule_AddObjectRef(module, "_module_initialized", initialized) != 0) { |
|
return -1; |
|
} |
|
|
|
return 0; |
|
} |
At least, I found initialized is leaked here. PyModule_AddObjectRef creates a new reference. The previous leak statistics indicate state->error may also be leaked elsewhere.
@ericsnowcurrently Can you please take a look since the last change is yours? sunmy2019 commented on Apr 29, 2023 @ericvsmith I tracked down the problem. Hope you can fix it and add me as a co-author.
1
initialized is leaked as mentioned above.
2
_testsinglephase_with_state module. Its md_state is not released by _testinternalcapi.clear_extension or _clear_globals.
When I manually add and call the clean-up function. Ref leaks are gone.
3
|
# Start with an interpreter that gets destroyed right away. |
|
base = self.import_in_subinterp(postscript=''' |
|
# Attrs set after loading are not in m_copy. |
|
mod.spam = 'spam, spam, mash, spam, eggs, and spam' |
|
''') |
No cleanup was made against base, which is leaking ref. This line looks intentional from the comment, but it does leak things.
Most of the ref leaks in test_import are here. Hopefully, it should be all. ericsnowcurrently commented on May 1, 2023 @sunmy2019, there are known refleaks in test_import. See gh-102251. Are you talking about more than that?
Also, your analysis is helpful. Thanks! sunmy2019 commented on May 1, 2023
Are you talking about more than that?
Yes, I am talking about test_import. The link you give seems to be test_imp.
The ref leak in test_import is introduced at least two months ago:
https://buildbot.python.org/all/#/builders/205/builds/696
and still visible and reproducible on main:
./python -m test -j1 -R 3:3 test_import
The above guide should help you clear major ref leaks in test_import. ericsnowcurrently commented on May 2, 2023 The tests in question were moved from test_imp to test_import. JelleZijlstra commented on May 12, 2023 I think the main refleaks are fixed now, closing this as the test_import refleaks are tracked in #102251.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|