GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23 · Issue #89452 · python/cpython

test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23 #89452

New issue

Copy link

New issue

Copy link

Closed

test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23#89452

Copy link

Labels

3.10 (EOL)end of lifeend of life3.11only security fixesonly security fixes3.9 (EOL)end of lifeend of lifeOS-mactype-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

Description

pablogsal

opened

on Sep 25, 2021

Issue body actions

BPO	45289
Nosy	@ronaldoussoren, @ned-deily, @corona10, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-09-25.19:56:32.069>
labels = ['OS-mac', '3.10', '3.9', 'type-crash', '3.11']
title = 'test_dbm and test_dbm_ndbm segfaults in M1 Mac'
updated_at = <Date 2021-09-26.05:38:13.631>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2021-09-26.05:38:13.631>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2021-09-25.19:56:32.069>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 45289
keywords = []
message_count = 2.0
messages = ['402643', '402645']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'corona10', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue45289'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Hand-linked PRs

Linked PRs

Activity

pablogsal

on Sep 25, 2021

MemberAuthor

More actions

0:04:14 load avg: 3.86 [141/427/1] test_dbm crashed (Exit code -11)
Fatal Python error: Segmentation fault
Current thread 0x0000000102e2bd40 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_dbm.py", line 58 in keys_helper
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_dbm.py", line 143 in read_helper
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_dbm.py", line 74 in test_anydbm_creation
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/case.py", line 547 in \_callTestMethod
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/case.py", line 591 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/case.py", line 646 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 84 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 84 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 84 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/runner.py", line 197 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/support/__init__.py", line 992 in \_run_suite
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/support/__init__.py", line 1118 in run_unittest
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 261 in \_test_module
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 297 in \_runtest_inner2
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 340 in \_runtest_inner
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 202 in \_runtest
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 245 in runtest
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest_mp.py", line 83 in run_tests_worker
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/main.py", line 678 in \_main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/main.py", line 658 in main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/main.py", line 736 in main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/regrtest.py", line 43 in \_main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/regrtest.py", line 47 in \<module\>
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/runpy.py", line 86 in \_run_code
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/runpy.py", line 196 in \_run_module_as_main
Extension modules: \_testcapi (total: 1)
0:04:15 load avg: 3.87 [142/427/1] test_c_

The backtrace according to lldb:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10365c000)
  * frame #0: 0x0000000182bc6994 libsystem_platform.dylib`_platform_memmove + 84
    frame #1: 0x00000001000b71d0 python.exe`PyBytes_FromStringAndSize(str="c", size=4294967297) at bytesobject.c:165:5
    frame #2: 0x000000010360b234 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys_impl(self=0x00000001034b3ca0, cls=0x000000010088a580) at _dbmmodule.c:249:16
    frame #3: 0x000000010360af34 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys(self=0x00000001034b3ca0, cls=0x000000010088a580, args=0x00000001007991d8, nargs=0, kwnames=0x0000000000000000) at _dbmmodule.c.h:46:20
    frame #4: 0x00000001000dca8c python.exe`method_vectorcall_FASTCALL_KEYWORDS_METHOD(func=0x00000001034db290, args=0x00000001007991d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at descrobject.c:367:24
    frame #5: 0x0000000100252374 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100808b30, callable=0x00000001034db290, args=0x00000001007991d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:114:11
    frame #6: 0x000000010024fc10 python.exe`PyObject_Vectorcall(callable=0x00000001034db290, args=0x00000001007991d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:123:12
    frame #7: 0x000000010024fd70 python.exe`call_function(tstate=0x0000000100808b30, pp_stack=0x000000016fdc8aa0, oparg=1, kwnames=0x0000000000000000, use_tracing=0) at ceval.c:6418:13
    frame #8: 0x000000010024a518 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100808b30, frame=0x0000000100799150, throwflag=0) at ceval.c:4568:19
    frame #9: 0x000000010023a0f0 python.exe`_PyEval_EvalFrame(tstate=0x0000000100808b30, frame=0x0000000100799150, throwflag=0) at pycore_ceval.h:46:12
    frame #10: 0x000000010023a00c python.exe`_PyEval_Vector(tstate=0x0000000100808b30, con=0x00000001034e0320, locals=0x0000000000000000, args=0x0000000100799118, argcount=2, kwnames=0x0000000000000000) at ceval.c:5609:24
    frame #11: 0x00000001000ccbe4 python.exe`_PyFunction_Vectorcall(func=0x00000001034e0310, stack=0x0000000100799118, nargsf=9223372036854775810, kwnames=0x0000000000000000) at call.c:342:16
    frame #12: 0x0000000100252374 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100808b30, callable=0x00000001034e0310, args=0x0000000100799118, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:114:11
    frame #13: 0x000000010024fc10 python.exe`PyObject_Vectorcall(callable=0x00000001034e0310, args=0x0000000100799118, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:123:12
    frame #14: 0x000000010024fd70 python.exe`call_function(tstate=0x0000000100808b30, pp_stack=0x000000016fdca950, oparg=2, kwnames=0x0000000000000000, use_tracing=0) at ceval.c:6418:13
    frame #15: 0x000000010024a518 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100808b30, frame=0x00000001007990a8, throwflag=0) at ceval.c:4568:19
    frame #16: 0x000000010023a0f0 python.exe`_PyEval_EvalFrame(tstate=0x0000000100808b30, frame=0x00000001007990a8, throwflag=0) at pycore_ceval.h:46:12
    frame #17: 0x000000010023a00c python.exe`_PyEval_Vector(tstate=0x0000000100808b30, con=0x00000001034e0c10, locals=0x0000000000000000, args=0x0000000100799080, argcount=2, kwnames=0x0000000000000000) at ceval.c:5609:24
    frame #18: 0x00000001000ccbe4 python.exe`_PyFunction_Vectorcall(func=0x00000001034e0c00, stack=0x0000000100799080, nargsf=9223372036854775810, kwnames=0x0000000000000000) at call.c:342:16
    frame #19: 0x0000000100252374 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100808b30, callable=0x00000001034e0c00, args=0x0000000100799080, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:114:11
    frame #20: 0x000000010024fc10 python.exe`PyObject_Vectorcall(callable=0x00000001034e0c00, args=0x0000000100799080, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:123:12
    frame #21: 0x000000010024fd70 python.exe`call_function(tstate=0x0000000100808b30, pp_stack=0x000000016fdcc800, oparg=2, kwnames=0x0000000000000000, use_tracing=0) at ceval.c:6418:13
    frame #22: 0x000000010024a518 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100808b30, frame=0x0000000100799018, throwflag=0) at ceval.c:4568:19
    frame #23: 0x000000010023a0f0 python.exe`_PyEval_EvalFrame(tstate=0x0000000100808b30, frame=0x0000000100799018, throwflag=0) at pycore_ceval.h:46:12
    frame #24: 0x000000010023a00c python.exe`_PyEval_Vector(tstate=0x0000000100808b30, con=0x00000001034e0530, locals=0x0000000000000000, args=0x0000000100798ff0, argcount=1, kwnames=0x0000000000000000) at ceval.c:5609:24
......

The last 3 frames:

frame #1: 0x00000001000b71d0 python.exe`PyBytes_FromStringAndSize(str="c", size=4294967297) at bytesobject.c:165:5
   162 	    if (str == NULL)
   163 	        return (PyObject *) op;
   164
-> 165 	    memcpy(op->ob_sval, str, size);
   166 	    /* share short strings */
   167 	    if (size == 1) {
   168 	        struct _Py_bytes_state *state = get_bytes_state();
(lldb)
frame #2: 0x000000010360b234 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys_impl(self=0x00000001034b3ca0, cls=0x000000010088a580) at _dbmmodule.c:249:16
   246 	    }
   247 	    for (key = dbm_firstkey(self->di_dbm); key.dptr;
   248 	         key = dbm_nextkey(self->di_dbm)) {
-> 249 	        item = PyBytes_FromStringAndSize(key.dptr, key.dsize);
   250 	        if (item == NULL) {
   251 	            Py_DECREF(v);
   252 	            return NULL;
(lldb)
frame #3: 0x000000010360af34 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys(self=0x00000001034b3ca0, cls=0x000000010088a580, args=0x00000001007991d8, nargs=0, kwnames=0x0000000000000000) at _dbmmodule.c.h:46:20
   43  	        )) {
   44  	        goto exit;
   45  	    }
-> 46  	    return_value = _dbm_dbm_keys_impl(self, cls);
   47
   48  	exit:
   49  	    return return_value;

The segfault happens here:

frame #0: 0x0000000182bc6994 libsystem_platform.dylib`_platform_memmove + 84
libsystem_platform.dylib`_platform_memmove:
->  0x182bc6994 <+84>: ldnp   q0, q1, [x1]
    0x182bc6998 <+88>: add    x1, x1, #0x20             ; =0x20
    0x182bc699c <+92>: subs   x2, x2, #0x20             ; =0x20
    0x182bc69a0 <+96>: b.hi   0x182bc698c               ; <+76>

pablogsal
added
testsTests in the Lib/test dirTests in the Lib/test dir

3.9 (EOL)end of lifeend of life

3.10 (EOL)end of lifeend of life

3.11only security fixesonly security fixes

type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

OS-mac
and removed
testsTests in the Lib/test dirTests in the Lib/test dir
on Sep 25, 2021
pablogsal commented on Sep 25, 2021
pablogsal
on Sep 25, 2021
MemberAuthor
More actions
See https://buildbot.python.org/all/#/builders/725/builds/4
pablogsal
changed the title ~~[-]test_gdbm segfaults in M1 Mac[/-]~~ [+]test_dbm and test_dbm_ndbm segfaults in M1 Mac[/+] on Sep 25, 2021
ezio-melotti
transferred this issue from on Apr 10, 2022

ambv commented on May 2, 2022

ambv

on May 2, 2022

Contributor

More actions

Looking at macOS native crash reporting, the issue is in this line of _dbm_dbm_keys_impl within the _dbmmodule.c:

    for (key = dbm_firstkey(self->di_dbm); key.dptr;
         key = dbm_nextkey(self->di_dbm)) {
        item = PyBytes_FromStringAndSize(key.dptr, key.dsize);  // <<<<<<<<<<
        if (item == NULL) {
            Py_DECREF(v);
            return NULL;
        }

Crash report:

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib            0x199490db8 __pthread_kill + 8
1   libsystem_pthread.dylib           0x1994c5ee0 pthread_kill + 288
2   libsystem_c.dylib                 0x1993c8674 raise + 32
3   Python                            0x10512fb00 faulthandler_fatal_error + 356 (faulthandler.c:375)
4   libsystem_platform.dylib          0x1994dd4c4 _sigtramp + 56
5   Python                            0x104ed5df0 PyBytes_FromStringAndSize + 272 (bytesobject.c:142)
6   _dbm.cpython-311d-darwin.so       0x104b9b494 _dbm_dbm_keys_impl + 220 (_dbmmodule.c:255)

ambv commented on May 2, 2022

ambv

on May 2, 2022

Contributor

More actions

A minimal reproduction is:

import dbm.ndbm
from pathlib import Path

db_path = Path("/tmp/test.dbm")
db_path.unlink(missing_ok=True)
f = dbm.ndbm.open(db_path, 'c')
print("Underlying lib:", dbm.ndbm.library)
print("f is an", repr(f))
print("The next line segfaults:")
f.keys()

This crashes with:

Underlying lib: GNU gdbm
f is an <_dbm.dbm object at 0x105a8fe80>
The next line segfaults:
Fatal Python error: Segmentation fault

Current thread 0x0000000105794580 (most recent call first):
  File "/Volumes/RAMDisk/cpython/_test.py", line 10 in <module>
fish: Job 1, './python.exe -X faulthandler _t…' terminated by signal SIGSEGV (Address boundary error)

ambv commented on May 2, 2022

ambv

on May 2, 2022

Contributor

More actions

Note that swapping dbm.ndbm for dbm.gnu causes the example to run without crashing:

import dbm.gnu
from pathlib import Path

db_path = Path("/tmp/test.dbm")
db_path.unlink(missing_ok=True)
f = dbm.gnu.open(db_path, 'c')
print("f is an", repr(f))
print("The next line runs fine:")
f.keys()

This just outputs:

❯ ./python.exe -X faulthandler _test.py                                                                                                                                              
f is an <_gdbm.gdbm object at 0x103afae90>
The next line runs fine:
❯

So looks like there is something off with the ndbm compatibility layer of gdbm on M1 Macs.

ambv commented on May 2, 2022
ambv
on May 2, 2022
Contributor
More actions
This is on Homebrew gdbm 1.23.
5 remaining items
ambv
removed a link to a pull requestgh-89452: Prefer gdbm-compat over ndbm #92208 on May 3, 2022
ambv commented on May 3, 2022
ambv
on May 3, 2022
Contributor
More actions
I unlinked the issue because this is not what we want: "linked a pull request that will close this issue"

We want to backport the fix.
ambv
added a commit that references this issue on May 3, 2022
gh-89452: Prefer gdbm-compat over ndbm (#92208)
6c25bf0
ambv commented on May 3, 2022
ambv
on May 3, 2022
Contributor
More actions
See PR, we actually don't want to backport this. So this is fixed for 3.11+. For older Python versions, the workaround is to build with:

./configure --with-dbmliborder=gdbm:ndbm
ambv
closed this as completed on May 3, 2022
erlend-aasland
added this to dbm and shelve issueson Jan 25, 2024
erlend-aasland
moved this to Done in dbm and shelve issueson Jan 25, 2024
encukou
added a commit that references this issue on Oct 8, 2024
[3.10] pythongh-89452: GHA: Set --with-dbmliborder to avoid issues wi…
eac7718
encukou commented on Oct 8, 2024
encukou
on Oct 8, 2024
Member
More actions
This is now blocking 3.10 & 3.9 backport PRs. I'll re-open.
encukou
reopened this on Oct 8, 2024
ambv
added a commit that references this issue on Oct 8, 2024
[3.10] gh-89452: GHA: Set --with-dbmliborder to avoid issues with hom…
850189a
encukou
added a commit that references this issue on Oct 9, 2024
[3.9] pythongh-89452: GHA: Set --with-dbmliborder to avoid issues wit…
993f70d
ambv
added a commit that references this issue on Oct 9, 2024
[3.9] gh-89452: GHA: Set --with-dbmliborder to avoid issues with home…
be988e8
ambv commented on Oct 9, 2024
ambv
on Oct 9, 2024
Contributor
More actions
Thanks! Fixed and merged.
ambv
closed this as completed on Oct 9, 2024

Metadata

Assignees

No one assigned

Labels

Projects

dbm and shelve issues
Status
Done
Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23 #89452

Description

Hand-linked PRs

Linked PRs

Activity

pablogsal commented on Sep 25, 2021

pablogsal commented on Sep 25, 2021

ambv commented on May 2, 2022

ambv commented on May 2, 2022

ambv commented on May 2, 2022

ambv commented on May 2, 2022

5 remaining items

ambv commented on May 3, 2022

ambv commented on May 3, 2022

encukou commented on Oct 8, 2024

ambv commented on Oct 9, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions