GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Calling `ShareableList.count` in threads aborts: `Assertion 'self->exports == 0' failed` · Issue #127085 · python/cpython

Calling `ShareableList.count` in threads aborts: `Assertion 'self->exports == 0' failed` #127085

New issue

Copy link

New issue

Copy link

Closed

#127412

Closed

Calling ShareableList.count in threads aborts: Assertion 'self->exports == 0' failed#127085

#127412

Copy link

Labels

3.13only security fixesonly security fixes3.14bugs and security fixesbugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtopic-multiprocessingtype-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

Description

devdanzin

opened

on Nov 21, 2024

Issue body actions

Crash report

What happened?

It's possible to abort the interpreter by calling multiprocessing.shared_memory.ShareableList.count in threads with PYTHON_GIL=0 in a debug build:

import gc
import multiprocessing.shared_memory
from threading import Thread

obj = multiprocessing.shared_memory.ShareableList("Uq..SeDAmB+EBrkLl.SG.Z+Z.ZdsV..wT+zLxKwdN\b")

for x in range(10):
    Thread(target=obj.count, args=(1,)).start()

del obj
gc.collect()

Result:

Exception ignored in: <function SharedMemory.__del__ at 0x200006bbfb0>
Traceback (most recent call last):
  File "/home/danzin/projects/mycpython/Lib/multiprocessing/shared_memory.py", line 189, in __del__
    self.close()
  File "/home/danzin/projects/mycpython/Lib/multiprocessing/shared_memory.py", line 229, in close
    self._buf.release()
BufferError: memoryview has 2 exported buffers
python: Objects/memoryobject.c:1143: memory_dealloc: Assertion `self->exports == 0' failed.
Aborted

Found using fusil by @vstinner.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a2+ experimental free-threading build (heads/main-dirty:c9b399fbdb0, Nov 19 2024, 20:12:48) [GCC 11.4.0]

Linked PRs

Activity

devdanzin
added
type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump
on Nov 21, 2024
picnixz
added
topic-multiprocessing

interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

topic-free-threading

3.13only security fixesonly security fixes

3.14bugs and security fixesbugs and security fixes
on Nov 21, 2024
LindaSummer commented on Nov 22, 2024
LindaSummer
on Nov 22, 2024
Contributor
More actions
Hi @devdanzin ,

I'm a newbie of CPython and want to have a try on this issue.

Could it be assigned to me?😊

Best Regards,
Edward
JelleZijlstra commented on Nov 22, 2024
JelleZijlstra
on Nov 22, 2024
Member
More actions
@LindaSummer you don't need to be assigned an issue to work on it. If you're interested in this issue, I suggest you read through the code where the crash happens to understand it. If you feel you found something useful, you can post on this issue with your findings. If you find a possible solution, you can open a PR with your fix.
Reacted by Edward Xu
LindaSummer commented on Nov 23, 2024
LindaSummer
on Nov 23, 2024
Contributor
More actions

@LindaSummer you don't need to be assigned an issue to work on it. If you're interested in this issue, I suggest you read through the code where the crash happens to understand it. If you feel you found something useful, you can post on this issue with your findings. If you find a possible solution, you can open a PR with your fix.

Hi @JelleZijlstra ,

Got it! Thanks very much for your warm guidance! 😊

I will try to go through the code and search for the root cause.

Best Regards,
Edward

LindaSummer commented on Nov 26, 2024

LindaSummer

on Nov 26, 2024

Contributor

More actions

Hi @JelleZijlstra ,

Here are some updates during these days.

I constructed a new case to reproduce this issue. This issue should have no relation to gc.

import multiprocessing.shared_memory
from threading import Thread

def test_memoryview_buffer():
    import struct
    class TA:
        data = bytearray("Uq..SeDAmB+EBrkLl.SG.Z+Z.ZdsV..wT+zLxKwdN\b", encoding="utf8")
        _fmt = "b"*len(data)
        shm = multiprocessing.shared_memory.SharedMemory(create=True, size=struct.calcsize(_fmt))
        def __init__(self):
            struct.pack_into(self._fmt, self.shm.buf, 0, *self.data)
        def count(self, t):
            res = 0
            for i in range(len(self.data)):
                (v,) = struct.unpack_from('b', self.shm.buf, i)
                if i == v:
                    res += 1
            return res
    obj = TA()
    for x in range(2000):
        Thread(target=obj.count, args=(1,)).start()
    del obj

test_memoryview_buffer()

I print the ignored exception details and find below.

Exception ignored in: <function SharedMemory.__del__ at 0x20000c0efd0>
Traceback (most recent call last):
  File "/workspaces/cpython/Lib/multiprocessing/shared_memory.py", line 189, in __del__
    self.close()
  File "/workspaces/cpython/Lib/multiprocessing/shared_memory.py", line 230, in close
    self._buf.release()
BufferError: memoryview:  negative export count 
python.exe: Objects/memoryobject.c:1150: void memory_dealloc(PyObject *): Assertion `self->exports == 0' failed.

This should be inside the SharedMemory release function.

I use an bytearray memoryview and this issue doesn't happen again. I think it should be related to SharedMemory accessors.
I searched for the exports references and find it is incremented and decremented in file memory_getbuf and memory_releasebuf.

I think it should be a race condition issue on the shm.buf memoryview object.

I will try to learn more about the non-gil synchronization logic and try to find the root cause.

Please correct me if my analysis has any problems. 😊

Best Regards,
Edward

LindaSummer
added 5 commits that reference this issue on Nov 28, 2024
pythongh-127085: fix `memoryview->exports` race condition.
1c88b90
pythongh-127085: fix `memoryview->exports` race condition.
7a2cfa6
pythongh-127085: fix `memoryview->exports` race condition.
1cd98d0
pythongh-127085: fix `memoryview->exports` race condition.
c7872cd
Merge branch 'main' into bugfix/pythongh-127085
06f462f
2 remaining items
LindaSummer
added 10 commits that reference this issue on Nov 30, 2024
Merge branch 'main' into bugfix/pythongh-127085
376f382
pythongh-127085: refactor test case and add macro for exports atomic …
49957d1
Merge branch 'main' into bugfix/pythongh-127085
3991201
Merge branch 'main' into bugfix/pythongh-127085
d4dc897
Merge branch 'main' into bugfix/pythongh-127085
e287b88
pythongh-127085: fix formatting issue.
666fde3
Merge branch 'main' into bugfix/pythongh-127085
a3d58f1
Merge branch 'main' into bugfix/pythongh-127085
d348113
Merge branch 'main' of github.com:LindaSummer/cpython into bugfix/pyt…
9594f38
Merge branch 'main' of github.com:python/cpython into bugfix/pythongh…
0ca811e
kumaraditya303
added a commit that references this issue on Dec 16, 2024
gh-127085: fix some data races in memoryview in free-threading (#127412)
4937ba5
kumaraditya303
closed this as completedin #127412 on Dec 16, 2024
freakboy3742
added a commit that references this issue on Dec 17, 2024
gh-127085: Add a test skip if multiprocessing isn't available (#128019)
b9a492b
srinivasreddy
added 2 commits that reference this issue on Jan 8, 2025
pythongh-127085: fix some data races in memoryview in free-threading (p…
0be115d
pythongh-127085: Add a test skip if multiprocessing isn't available (p…
b5d061c

Metadata

Assignees

No one assigned

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Calling ShareableList.count in threads aborts: Assertion 'self->exports == 0' failed #127085

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

LindaSummer commented on Nov 22, 2024

JelleZijlstra commented on Nov 22, 2024

LindaSummer commented on Nov 23, 2024

LindaSummer commented on Nov 26, 2024

2 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Calling `ShareableList.count` in threads aborts: `Assertion 'self->exports == 0' failed` #127085