GeezerVison is showing github.com — Improve bigmem tests · Issue #75876 · python/cpython
Improve bigmem tests #75876Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-10-04.19:22:19.577>
labels = ['type-feature', 'tests']
title = 'Improve bigmem tests'
updated_at = <Date 2017-10-07.15:45:09.628>
user = 'https://github.com/serhiy-storchaka' bugs.python.org fields:
activity = <Date 2017-10-07.15:45:09.628>
actor = 'pitrou'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2017-10-04.19:22:19.577>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31695
keywords = []
message_count = 2.0
messages = ['303729', '303885']
nosy_count = 3.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31695'
versions = [] Linked PRs
Activityserhiy-storchaka commented on Oct 4, 2017 I think that the process of running bigmem tests should be improved.
-
Every bigmem test should run in separate process. It's effect on memory fragmentation shouldn't affect other tests.
-
Only one bigmem test should be run at the same time. Otherwise several parallel bigmem tests can exhaust RAM and lead to swapping. In multiprocessing mode the process running a bigmem test should send a signal to other processes, wait until they finished their current tests and paused, run a bigmem test, and send a signal to other processes that they can continue.
-
The hard limit of addressed memory should be set for every bigmem test, according to the declared requirements. It is better to crash one test and report the failure, than hang on swapping.
-
All other tests should be run with the limit set to 2 GiB (or 1 GiB?). This will help to find memory consuming tests which should be made bigmem tests.
-
The maxrss of the process running a bigmem test should be reported in verbose mode after finishing a test.
Reacted by Zachary Ware Reacted by Zachary Ware pitrou commented on Oct 7, 2017 cmaloney commented on May 9, 2026 Currently bigmemtest in verbose mode only reports memory usage on Linux by way of reading /proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improve bigmemtest. With that also fix 5 above, printing the maxrss after finishing each test.
To support that I'd like to add a private C extension function that uses platform-specific APIs to get process memory usage and update _MemoryWatchdog / memory_watchdog.py to use that information in its print. That makes it so the platform-specific code lives entirely inside C conditional compilation.
The code currently pre-opens the /proc file and passes that through as stdin to reduce PID recycling issues if the parent exits. Unfortunately there is no cross-platform pidfd type mechanism to make this really simple. Instead the new code relies on a pipe to the parent to determine "has the parent been lost" which lets us detect parent process termination and prevent issues if the PID is recycled. It also means we should exit more quickly when the parent intentionally exits and gives a good point to add the end of test print. Reacted by Serhiy Storchaka vstinner commented on May 11, 2026 This issue has not been touched for 9 years, I suggest you opening a new issue for your idea. We can add a private function in a test module such as _testcapi. I didn't understand the pidfd issue, I should see the code to understand :-) Reacted by Cody Maloney and Serhiy Storchaka StanFromIreland commented on Jun 25, 2026 Currently on our buildbot the test takes over one and a half hours, while some of that is unavoidable, it would be good to decrease it a little if possible. vstinner commented on Jun 25, 2026
Currently bigmemtest in verbose mode only reports memory usage on Linux by way of reading /proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improve bigmemtest.
I modified test.libregrtest to log the memory usage with a new get_process_memory_usage() function in test.libregrtest.utils. The function works on Linux, Windows, macOS and FreeBSD.
I also modified @support.bigmemtest to reuse this get_process_memory_usage() function.
Example:
Using random seed: 2542641886
0:00:00 load avg: 2.46 mem: 23.8 MiB Run 505 tests in parallel using 8 worker processes (timeout: 1 hour 40 min, worker timeout: 1 hour 45 min)
(...)
0:22:04 load avg: 5.86 mem: 14.8 GiB [485/505/1] test_descr passed -- running (3): test_bigmem (22 min 3 sec), test_lzma (22 min 3 sec), test_hashlib (4 min 8 sec)
0:22:05 load avg: 5.86 mem: 14.8 GiB [486/505/1] test.test_asyncio.test_streams passed -- running (3): test_bigmem (22 min 4 sec), test_lzma (22 min 4 sec), test_hashlib (4 min 8 sec)
0:22:06 load avg: 5.86 mem: 17.2 GiB [487/505/1] test_mimetypes passed -- running (3): test_bigmem (22 min 5 sec), test_lzma (22 min 5 sec), test_hashlib (4 min 9 sec)
(...)
0:41:07 load avg: 2.82 mem: 30.7 GiB [504/505/1] test_lzma passed (41 min 6 sec) -- running (1): test_bigmem (41 min 6 sec)
0:41:38 load avg: 2.29 mem: 30.1 GiB running (1): test_bigmem (41 min 37 sec)
(...)
1:31:12 load avg: 1.52 mem: 30.1 GiB running (1): test_bigmem (1 hour 31 min)
1:31:44 load avg: 1.62 mem: 30.1 GiB running (1): test_bigmem (1 hour 31 min)
1:32:14 load avg: 1.72 mem: 16.1 GiB running (1): test_bigmem (1 hour 32 min)
1:32:45 load avg: 1.43 mem: 4.4 GiB running (1): test_bigmem (1 hour 32 min)
1:32:51 load avg: 1.48 mem: 74.4 MiB [505/505/1] test_bigmem passed (1 hour 32 min)
The mem: 14.8 GiB part is new! serhiy-storchaka commented on Aug 7, 2026 #155302 makes the bigmem tests running in a separate subprocess -- point 1 in my plan. Thanks to reusing the runInSubprocess machinery it is transparent to the user -- you can use all TestCase methods. The follow-ups will implement other points (I already have 3 and 5).
#155307 corrects underspecified memory usage. It was needed to be fixed before imposing the hard limit (point 3). There were only 6 incorrect values in 233 bigmem tests -- for most tests the estimate was very accurate, as result of my past work. serhiy-storchaka commented on Aug 19, 2026 #156024 implements point 5 -- reports the maxrss and majflt of the process running a bigmem test. serhiy-storchaka commented on Aug 22, 2026 #156023 implements point 2 -- limits the address space of a bigmem test. There are a few subtleties. Glibc reserves some amount of address space per thread per CPU core (it is not really used), so there is an option to skip the address space limitation for tests with large amount of threads. ASan build and macOS also reserve huge address space -- it is skipped there too. For these cases (and for Windows which does not have such API), the watchdog kills the test when it's real use exceeds the limit.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|