GeezerVison

Copy link

Copy Markdown

Member

vstinner requested a review from rhettinger as a code owner April 12, 2024 10:31

colesbury reviewed Apr 12, 2024

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner force-pushed the mimalloc_s390x branch from 0ab6087 to 6f0d611 Compare April 15, 2024 08:27

colesbury approved these changes Apr 16, 2024

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gdb -args ./python -m test test_bigaddrspace -u all -v

(gdb) py-bt
Traceback (most recent call first):
  File "/root/vstinner/cpython/Lib/test/test_bigaddrspace.py", line 99, in allocate
    return b'x' * length
  File "/root/vstinner/cpython/Lib/test/test_bigaddrspace.py", line 113, in test_large_alloc
    allocate(size)
(...)

(gdb) where
#0  0x000003fffdaa38f0 in __memset_mvcle () from /lib64/libc.so.6
#1  0x00000000010ce708 in _PyBytes_Repeat (
    dest=0x5bf70010030 'x' <repeats 200 times>..., 
    len_dest=842105263157894695, src=0x5bf31174330 "x", len_src=1)
    at Objects/bytesobject.c:3640
#2  0x00000000010c9e3a in bytes_repeat (a=0x5bf31174300, n=842105263157894695)
    at Objects/bytesobject.c:1484
(...)

(gdb) frame 1
(gdb) p len_dest
$1 = 842105263157894695
(gdb) p/x len_dest
$2 = 0xbafc24672035e27

# grep ^Vm /proc/2070842/status
VmPeak:	822368422334080 kB
VmSize:	822368422334080 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	 7319628 kB
VmRSS:	 7086188 kB
VmData:	822368422110644 kB
VmStk:	     388 kB
VmExe:	    6576 kB
VmLib:	    3624 kB
VmPTE:	   26876 kB
VmSwap:	 6529488 kB

# free -m
              total        used        free      shared  buff/cache   available
Mem:           7782        7355         102         183         324         129
Swap:          8167        6723        1444

Remove unreliable tests on huge memory allocations:

* Remove test_maxcontext_exact_arith() of test_decimal.
  Stefan Krah, test author, agreed on removing the test:
  python#114331 (comment)
* Remove test_constructor() tests of test_io.
  Sam Gross suggests remove them:
  python#117809 (review)

On Linux, depending how overcommit is configured, especially on Linux
s390x, a huge memory allocation (half or more of the full address
space) can succeed, but then the process will eat the full system
swap and make the system slower and slower until the whole system
becomes unusable.

Moreover, these tests had to be skipped when Python is built with
sanitizers.

colesbury approved these changes Apr 16, 2024

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner enabled auto-merge (squash) April 16, 2024 16:40

Remove unreliable tests on huge memory allocations:

* Remove test_maxcontext_exact_arith() of test_decimal.
  Stefan Krah, test author, agreed on removing the test:
  #114331 (comment)
* Remove test_constructor() tests of test_io.
  Sam Gross suggests remove them:
  #117809 (review)

On Linux, depending how overcommit is configured, especially on Linux
s390x, a huge memory allocation (half or more of the full address
space) can succeed, but then the process will eat the full system
swap and make the system slower and slower until the whole system
becomes unusable.

Moreover, these tests had to be skipped when Python is built with
sanitizers.

vstinner deleted the mimalloc_s390x branch April 16, 2024 20:34

Remove unreliable tests on huge memory allocations:

* Remove test_maxcontext_exact_arith() of test_decimal.
  Stefan Krah, test author, agreed on removing the test:
  python#114331 (comment)
* Remove test_constructor() tests of test_io.
  Sam Gross suggests remove them:
  python#117809 (review)

On Linux, depending how overcommit is configured, especially on Linux
s390x, a huge memory allocation (half or more of the full address
space) can succeed, but then the process will eat the full system
swap and make the system slower and slower until the whole system
becomes unusable.

Moreover, these tests had to be skipped when Python is built with
sanitizers.

…7809)

Fix mimalloc allocator for huge memory allocation (around
8,589,934,592 GiB) on s390x.

Abort allocation early in mimalloc if the number of slices doesn't
fit into uint32_t, to prevent a integer overflow (cast 64-bit
size_t to uint32_t).