GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Python 3.13.0a6 freethreading on s390x: `test.test_io.CBufferedReaderTest.test_constructor` crash with `Floating point exception` · Issue #117755 · python/cpython

Python 3.13.0a6 freethreading on s390x: `test.test_io.CBufferedReaderTest.test_constructor` crash with `Floating point exception` #117755

New issue

Copy link

New issue

Copy link

Closed

Python 3.13.0a6 freethreading on s390x: test.test_io.CBufferedReaderTest.test_constructor crash with Floating point exception#117755

Copy link

Labels

type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

befeleme

opened

on Apr 11, 2024

Issue body actions

Bug report

Bug description:

Since #114331 was solved, we once again attempted to build Python with freethreading on s390x Fedora Linux.

test.test_io.CBufferedReaderTest.test_constructor fails. The traceback:

test_constructor (test.test_io.CBufferedReaderTest.test_constructor) ... Fatal Python error: Floating point exception
Current thread 0x000003ff8aa77b60 (most recent call first):
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/case.py", line 238 in handle
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/case.py", line 795 in assertRaises
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/test_io.py", line 1710 in test_constructor
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/case.py", line 651 in run
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/case.py", line 707 in __call__
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/unittest/runner.py", line 240 in run
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 132 in test_func
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 88 in regrtest_runner
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 135 in _load_run_test
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 178 in _runtest_env_changed_exc
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 278 in _runtest
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/single.py", line 306 in run_single_test
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/worker.py", line 77 in worker_process
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/worker.py", line 112 in main
  File "/builddir/build/BUILD/Python-3.13.0a6/Lib/test/libregrtest/worker.py", line 116 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main
1 test failed again:
    test_io

Is this freethreading related? I don't know. Hoping to raise visibility and pointers as where the issue comes from. cc @vstinner

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

befeleme
added
type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
on Apr 11, 2024
befeleme
changed the title ~~[-]3.13.0a6: `test.test_io.CBufferedReaderTest.test_constructor` ends with `Fatal Python error: Floating point exception`[/-]~~ [+]3.13.0a6: `test.test_io.CBufferedReaderTest.test_constructor` ends with `Fatal Python error: Floating point exception` on s390x[/+] on Apr 11, 2024
befeleme
changed the title ~~[-]3.13.0a6: `test.test_io.CBufferedReaderTest.test_constructor` ends with `Fatal Python error: Floating point exception` on s390x[/-]~~ [+]3.13.0a6: `test.test_io.CBufferedReaderTest.test_constructor` ends with `Fatal Python error: Floating point exception` on s390x with freethreading[/+] on Apr 11, 2024
vstinner
added a commit that references this issue on Apr 12, 2024
pythongh-117755: Skip test_io.test_constructor() on s390x
a7b9090

vstinner commented on Apr 12, 2024

vstinner

on Apr 12, 2024

Member

More actions

The root issue is a division by zero in mimalloc when requested memory is huge: 0x7fffffffffffffff bytes.

It can be reproduced on s390x without test_io:

$ ./python
Python 3.13.0a6+ (heads/main-dirty:396b831, Apr 12 2024, 04:17:51) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] on linux
>>> import sys
>>> size = 0x7fffffffffffffff
>>> len = (size - sys.getsizeof(b''))
>>> b'x'*len
Floating point exception (core dumped)

$ uname -r
4.18.0-513.18.1.el8_9.s390x

Reacted by Miro Hrončok

vstinner commented on Apr 12, 2024

vstinner

on Apr 12, 2024

Member

More actions

I can reproduce the issue with a Python built with:

./configure --disable-gil CFLAGS="-O0 -g -ggdb"
time make -j6

I disable all compiler optimizations (-O0) to ease debugging in gdb.

gdb logs when the bug occurs:

$ gdb -args ./python bug.py
(gdb) run

Program received signal SIGFPE, Arithmetic exception.
0x000000000112b508 in mi_page_init (tld=<optimized out>, block_size=<optimized out>, page=0x40000000168, heap=0x1595ce8 <_PyRuntime+295592>)
    at Objects/mimalloc/page.c:696
696	  page->reserved = (uint16_t)(page_size / block_size);

(gdb) p page_size
$1 = 0
(gdb) p block_size
$2 = 0

(gdb) up
#1  0x0000000001172d9e in mi_page_fresh_alloc (heap=0x16efaa8 <_PyRuntime+295592>, pq=0x16f0590 <_PyRuntime+298384>, 
    block_size=9223372036854775808, page_alignment=0) at Objects/mimalloc/page.c:295
295	  mi_page_init(heap, page, full_block_size, heap->tld);
(gdb) p full_block_size
$3 = 0
(gdb) p block_size
$4 = 9223372036854775808
(gdb) p /x block_size
$5 = 0x8000000000000000

(gdb) p pq == 0
$6 = 0
(gdb) p mi_page_queue_is_huge(pq)
$7 = true
(gdb) p mi_page_block_size(page)
$8 = 0
(gdb) p page->xblock_size
$9 = 0

vstinner commented on Apr 12, 2024
vstinner
on Apr 12, 2024
Member
More actions
(gdb) b PyObject_Malloc Breakpoint 1 at 0x11806c6: file Objects/obmalloc.c, line 1288. (gdb) condition 1 size >= 0x7fffffffffffffff

PyObject_Malloc():

mi_find_page()

mi_large_huge_page_alloc()

mi_page_fresh_alloc()

_mi_segment_page_alloc()

mi_segment_huge_page_alloc(): mi_segment_os_alloc() and _mi_segment_page_start()

mi_segment_huge_page_alloc() is called with size=0x8000000000000000:

mi_segment_alloc(size) returns 0x40000000000

uint8_t* start = _mi_segment_page_start(segment, page, &psize); with slice->slice_count = 0 sets psize to 0.

The problem is that slice->slice_count is 0: integer overflow. mi_page_t.slice_count type is uint32_t, whereas on s390x, we try to set it to 0x8000_00000000 (140737488355328) which doesn't fit:

static mi_page_t* mi_segment_span_allocate(mi_segment_t* segment, size_t slice_index, size_t slice_count, mi_segments_tld_t* tld) { ... slice->slice_count = (uint32_t)slice_count; ... return page; }

gdb:

Breakpoint 5, mi_segment_span_allocate (segment=0x40000000000, slice_index=1, slice_count=140737488355328, tld=0x16f1eb0 <_PyRuntime+304816>) at Objects/mimalloc/segment.c:711
vstinner
added a commit that references this issue on Apr 12, 2024
pythongh-117755: Fix mimalloc for huge allocation on s390x
dbbf181
vstinner commented on Apr 12, 2024
vstinner
on Apr 12, 2024
Member
More actions
On x86-64, we don't reach this bug, since mmap() fails before.

strace on the big mmap() call:

x86-64, fail with ENOMEM: mmap(NULL, 9223372036863164416, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM

s390x, success: mmap(NULL, 9223372036858970112, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x3ffdcf80000.

On x86-64, gdb traces around mmap():

_mi_arena_alloc_aligned(size=0x8000000000410000) returns NULL.

_mi_prim_alloc()

unix_mmap() with size=0x8000000000800000, try_alignment=0x2000000, protect_flags=PROT_WRITE | PROT_READ, allow_large=true => it fails with errno=ENOMEM and return NULL.
vstinner
changed the title ~~[-]3.13.0a6: `test.test_io.CBufferedReaderTest.test_constructor` ends with `Fatal Python error: Floating point exception` on s390x with freethreading[/-]~~ [+]Python 3.13.0a6 freethreading: `test.test_io.CBufferedReaderTest.test_constructor` crash with `Floating point exception` on s390x[/+] on Apr 12, 2024
vstinner
changed the title ~~[-]Python 3.13.0a6 freethreading: `test.test_io.CBufferedReaderTest.test_constructor` crash with `Floating point exception` on s390x[/-]~~ [+]Python 3.13.0a6 freethreading on s390x: `test.test_io.CBufferedReaderTest.test_constructor` crash with `Floating point exception`[/+] on Apr 12, 2024
colesbury commented on Apr 12, 2024
colesbury
on Apr 12, 2024
Contributor
More actions
One thing I'm curious about: even with overcommit, I think that mmap() still needs to create the page mappings. With 4KB pages, that's trillions of pages. The page table itself is too bit to store in memory.

Why does it succeed on s390x? Is mmap able to use much bigger pages on IBM Z? Something else?
vstinner commented on Apr 12, 2024
vstinner
on Apr 12, 2024
Member
More actions

Why does it succeed on s390x? Is mmap able to use much bigger pages on IBM Z? Something else?

Sorry, I have no idea 🤷🏻‍♂️
vstinner
added 5 commits that reference this issue on Apr 15, 2024
gh-117755: Skip test_io.test_constructor() on s390x (#117801)
a9107fe
pythongh-117755: Fix mimalloc for huge allocation on s390x
6f0d611
pythongh-117755: Remove tests on huge memory allocations
cf45a92
gh-117755: Remove tests on huge memory allocations (#117938)
e05d202
gh-117755: Fix mimalloc for huge allocation on s390x (#117809)
3fe03cc
diegorusso
added 3 commits that reference this issue on Apr 17, 2024
pythongh-117755: Skip test_io.test_constructor() on s390x (python#117801
866aff5
pythongh-117755: Remove tests on huge memory allocations (python#117938)
80cd6a2
pythongh-117755: Fix mimalloc for huge allocation on s390x (python#11…
31a474b
hugovk
closed this as completed on Jun 15, 2024

Metadata

Assignees

No one assigned

Labels

type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Python 3.13.0a6 freethreading on s390x: test.test_io.CBufferedReaderTest.test_constructor crash with Floating point exception #117755

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

vstinner commented on Apr 12, 2024

vstinner commented on Apr 12, 2024

vstinner commented on Apr 12, 2024

vstinner commented on Apr 12, 2024

colesbury commented on Apr 12, 2024

vstinner commented on Apr 12, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Python 3.13.0a6 freethreading on s390x: `test.test_io.CBufferedReaderTest.test_constructor` crash with `Floating point exception` #117755