GeezerVison is showing github.com — `ctypes` thread safety auditing (and fixing) · Issue #127945 · python/cpython
ctypes thread safety auditing (and fixing) #127945
DescriptionFeature or enhancement
This is a tracking issue for all thread safety problems related to ctypes. I'll be working on this, but others can feel free to give me some help.
First of all, we need to find where the thread safety problems are.
### Auditing
- [x] Audit `_ctypes.c`
- [ ] Audit `_ctypes_test.c` (and probably the generated file too)
- [ ] Audit `callbacks.c`
- [ ] Audit `callproc.c`
- [ ] Audit `cfield.c`
- [x] Audit `malloc_closure.c`
- [ ] Audit `stgdict.c`
I'll be tracking the issues that get found when auditing here. The plan is to just create a new issue and link to it for each new problem instead of flooding this issue with PRs.
Generally, the workflow for fixes should follow most of the rules from #116738, but I suspect we'll need recursive mutexes for quite a few things related to callbacks, because it's difficult to tell what might be re-entrant, and we can't use critical sections for arbitrary function pointers.
### Known Issues
- [ ] https://github.com/python/cpython/issues/127946
- [ ] https://github.com/python/cpython/issues/128182
- [ ] https://github.com/python/cpython/issues/128485
- [ ] https://github.com/python/cpython/issues/128567
- [ ] https://github.com/python/cpython/issues/128570
- [ ] https://github.com/python/cpython/issues/131974
cc @encukou, as the ctypes genius, and @colesbury as the free-threading mastermind.
Linked PRs
Reacted by Enzo Bonnal, vfdev and Erlend E. Aasland Activityencukou commented on Jan 10, 2025 ZeroIntensity commented on Jan 10, 2025 Got it, thanks! I'll probably just move to callbacks.c next. kumaraditya303 commented on Mar 20, 2025 I discussed this with Peter and moving forward I'll be taking over the thread safety work and continue it. kumaraditya303 commented on Mar 24, 2025 I investigated the thread safety of ctypes and here's the (incomplete) list of issues:
colesbury commented on Mar 24, 2025 I think there are also races on StgInfo.flags, in particular with the DICTFLAG_FINAL bit.
|
info->flags &= ~DICTFLAG_FINAL; /* clear the 'final' flag in the subclass info */ |
|
baseinfo->flags |= DICTFLAG_FINAL; /* set the 'final' flag in the baseclass info */ |
kumaraditya303 commented on Mar 24, 2025
I think there are also races on StgInfo.flags, in particular with the DICTFLAG_FINAL bit.
Yes, I am covering that under "the initialization and concurrent modifications of stginfo is not thread safe". colesbury commented on Mar 24, 2025 This is different and more important than concurrent modification or initialization of stginfo, and I think it should be one of the early things we should focus on. Concurrent modifications of ctypes Structures is not an important use case.
However, creating separate instances of the same structure is an important use case:
from ctypes import Structure, c_int
from concurrent.futures import ThreadPoolExecutor
class POINT(Structure):
_fields_ = [("x", c_int), ("y", c_int)]
with ThreadPoolExecutor() as executor:
futures = [executor.submit(POINT, i, i) for i in range(1000)]
results = [future.result() for future in futures]This needs to be both efficient and thread-safe. added 3 commits that reference this issue on Mar 25, 2025gh-127945: fix thread safety of creating instances of ctypes structur… sergey-miryanov commented on May 15, 2025 Can this issue be closed now as all linked PRs have been merged? Is there anything else that needs to be done?
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|