GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Crash During Subinterpreter Finalization · Issue #105699 · python/cpython

Crash During Subinterpreter Finalization #105699

New issue
Copy link
New issue
Copy link
Closed
Closed
Crash During Subinterpreter Finalization#105699
Copy link
Assignees
ericsnowcurrently
Labels
3.12only security fixesonly security fixes3.13only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

Description

@ericsnowcurrently
ericsnowcurrently
opened on Jun 12, 2023
Issue body actions

There's an isolation leak somewhere. It may be just in the _xxsubinterpreters module, but I suspect it's not.

See #99114 (comment).

Reproducers:

  • https://gist.github.com/tonybaloney/262986212e1061b97908657a53a605d6
  • https://gist.github.com/tonybaloney/504be6d1541610f50963c32a3f1c379d

FYI, I see crashes on this fairly infrequently.

Linked PRs

  • gh-105699: Use a Thread-Local Variable for PKGCONTEXT #105740
  • [3.12] gh-105699: Use a Thread-Local Variable for PKGCONTEXT (gh-105740) #105765
  • gh-105699: Use a Linked List for the PyModuleDef Cache #106899
  • gh-105699: Fix a Crasher Related to a Deprecated Global Variable #106923
  • gh-105699: Fix an Interned Strings Crasher #106930
  • [3.12] gh-105699: Fix an Interned Strings Crasher (gh-106930) #106963
  • [3.12] gh-105699: Fix a Crasher Related to a Deprecated Global Variable (gh-106923) #106964
  • gh-105699: Add Some Stress Tests for Subinterpreter Creation #106966
  • gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache #106974
  • [3.12] gh-105699: Add some stress tests for subinterpreter creation (GH-106966) #107012
  • gh-105699: Disable the Interpreters Stress Tests #107354
  • [3.12] gh-105699: Disable the Interpreters Stress Tests (gh-107354) #107357
  • [3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974) #107412
  • gh-105699: Re-enable the Multiple-Interpreters Stress Tests #107572
  • [3.12] gh-105699: Re-enable the Multiple-Interpreters Stress Tests (gh-107572) #107783
  • [3.12] gh-105699: Re-enable the Multiple-Interpreters Stress Tests (gh-107572) #112483

Activity

Next
  1. ericsnowcurrently
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)
    type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump
    topic-subinterpreters
    3.12only security fixesonly security fixes
    3.13only security fixesonly security fixes
    on Jun 12, 2023
  2. ericsnowcurrently commented on Jun 12, 2023

    @ericsnowcurrently
    ericsnowcurrently
    on Jun 12, 2023
    MemberAuthor
    More actions

    Probably the same thing:

    (AMD64 Arch Linux TraceRefs 3.12)

    test_one (test.test__xxsubinterpreters.DestroyTests.test_one) ... ok
    Objects/object.c:2211: _Py_ForgetReference: Assertion failed: invalid object chain
    Enable tracemalloc to get the memory block allocation traceback
    object address  : 0x7f6bf962d300
    object refcount : 0
    object type     : 0x55a320cd18c0
    object type name: bytes
    object repr     : <refcnt 0 at 0x7f6bf962d300>
    Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
    Python runtime state: initialized
    Current thread 0x00007f6c09e2e740 (most recent call first):
      <no Python frame>
    Debug memory block at address p=0x7f6c09d55e90: API '�'
        18302063728033398269 bytes originally requested
        The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
            at p-7: 0xdd *** OUCH
            at p-6: 0xdd *** OUCH
            at p-5: 0xdd *** OUCH
            at p-4: 0xdd *** OUCH
            at p-3: 0xdd *** OUCH
            at p-2: 0xdd *** OUCH
            at p-1: 0xdd *** OUCH
        Because memory is corrupted at the start, the count of bytes requested
           may be bogus, and checking the trailing pad bytes may segfault.
        The 8 pad bytes at tail=0xfdfe7d6a07d35c8d are Fatal Python error: Segmentation fault
    Current thread 0x00007f6c09e2e740 (most recent call first):
      <no Python frame>
    Extension modules: _testcapi, _xxsubinterpreters, _xxinterpchannels (total: 3)
    make: *** [Makefile:2015: buildbottest] Segmentation fault (core dumped)
    
  3. github-project-automation
    added this to Subinterpreterson Jun 12, 2023
  4. github-project-automation
    moved this to Todo in Subinterpreterson Jun 12, 2023
  5. ericsnowcurrently commented on Jun 12, 2023

    @ericsnowcurrently
    ericsnowcurrently
    on Jun 12, 2023
    MemberAuthor
    More actions

    maybe related: #105690

  6. ericsnowcurrently
    added a commit that references this issue on Jun 14, 2023

    gh-105699: Use a Thread-Local Variable for PKGCONTEXT (gh-105740)

    b87d288
  7. miss-islington
    added a commit that references this issue on Jun 14, 2023

    pythongh-105699: Use a Thread-Local Variable for PKGCONTEXT (pythongh…

    bfb6863
  8. ericsnowcurrently
    added a commit that references this issue on Jun 14, 2023

    [3.12] gh-105699: Use a Thread-Local Variable for PKGCONTEXT (gh-105740…

    bc997b3
  9. ericsnowcurrently commented on Jul 19, 2023

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 19, 2023
    MemberAuthor
    More actions

    After gh-106899, I'm only seeing 3 very infrequent crashers:

    1. failing assertion in _xxinterpreterchannelsmodule.c (not user-facing)
    2. bogus interned string state
    3. something bad in start_thread() (pthread)
    (stack track for that last one)
    Thread 1 (Thread 0x7f1c41efb700 (LWP 27875)):
    #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
    #1  0x00007f1c4a6bd7f1 in __GI_abort () at abort.c:79
    #2  0x00007f1c4a706837 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f1c4a833a7b "%s\n") at ../sysdeps/posix/libc_fatal.c:181
    #3  0x00007f1c4a70d8ba in malloc_printerr (str=str@entry=0x7f1c4a831c8e "free(): invalid size") at malloc.c:5342
    #4  0x00007f1c4a818a6c in _int_free (have_lock=0, p=0x7f1c28001490, av=0x7f1c28000020) at malloc.c:4171
    #5  __GI___libc_free (mem=0x7f1c280014a0) at malloc.c:3134
    #6  tcache_thread_shutdown () at malloc.c:2979
    #7  arena_thread_freeres () at arena.c:950
    #8  0x00007f1c4a819562 in __libc_thread_freeres () at thread-freeres.c:29
    #9  0x00007f1c4b21a700 in start_thread (arg=0x7f1c41efb700) at pthread_create.c:476
    #10 0x00007f1c4a79e61f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
    
  10. ericsnowcurrently
    self-assigned this
    on Jul 20, 2023
  11. ericsnowcurrently commented on Jul 20, 2023

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 20, 2023
    MemberAuthor
    More actions

    With the 3 PRs I have up I don't see any more crashes (other than in _xxinterpchannels).

  12. 13 remaining items

  13. miss-islington
    added a commit that references this issue on Jul 27, 2023

    pythongh-105699: Disable the Interpreters Stress Tests (pythongh-107354)

    f5e7b24
  14. ericsnowcurrently
    added 4 commits that reference this issue on Jul 27, 2023

    [3.12] gh-105699: Disable the Interpreters Stress Tests (gh-107354) (g…

    c580527

    gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974)

    8ba4df9

    pythongh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (pyt…

    5b3abd0

    [3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh…

    da151fd
  15. ericsnowcurrently commented on Jul 31, 2023

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 31, 2023
    MemberAuthor
    More actions

    I've disable the new stress tests as they were causing crashes regularly that I'm not convinced indicate bugs (outside _xxsubinterpreters). Once that's sorted out and I've re-enabled the tests, we can close this issue.

  16. ericsnowcurrently
    added a commit that references this issue on Aug 8, 2023

    gh-105699: Re-enable the Multiple-Interpreters Stress Tests (gh-107572)

    f9e3ff1
  17. miss-islington
    added a commit that references this issue on Aug 8, 2023

    pythongh-105699: Re-enable the Multiple-Interpreters Stress Tests (py…

    7329b6f
  18. ericsnowcurrently commented on Aug 8, 2023

    @ericsnowcurrently
    ericsnowcurrently
    on Aug 8, 2023
    MemberAuthor
    More actions

    I'm calling this good.

  19. ericsnowcurrently
    closed this as completedon Aug 8, 2023
  20. github-project-automation
    moved this from Todo to Done in Subinterpreterson Aug 8, 2023
  21. ericsnowcurrently
    added 4 commits that reference this issue on Nov 27, 2023

    [3.12] gh-105699: Re-enable the Multiple-Interpreters Stress Tests (g…

    a4aac7d

    Revert "[3.12] pythongh-105699: Re-enable the Multiple-Interpreters S…

    7cabbb7

    [3.12] gh-105699: Revert gh-107783 "Re-enable the Multiple-Interprete…

    ee7dc67

    [3.12] pythongh-105699: Re-enable the Multiple-Interpreters Stress Te…

    d697fb0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @ericsnowcurrently
    ericsnowcurrently

Labels

3.12only security fixesonly security fixes3.13only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

Projects

  • Subinterpreters
    Status
    Done
    Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.