GeezerVison is showing github.com — gh-125828 : Fix 'get_value' missing for [Bounded]Semaphore on multiprocessing MacOSX - C workaround by YvesDup · Pull Request #130913 · python/cpython
Conversation
YvesDup
commented
Mar 6, 2025
•
edited by bedevere-app
Bot
Loading
Uh oh!
There was an error while loading. Please reload this page.
Following #125829 discussions and after a long deeping in the _multiprocessing.semaphore.c C file, I suggest a workaround based on a shared memory that contains an array to store all the internal semaphore counters.
At the moment, all the tests are passing, but I'm having 2 side problems, the second of which is really embarrassing:
- How to calculate the size of the array stored in the shared memory, knowing that:
- I sized the array from the value of
sysconf(SC_SEM_NSEMS_MAX) which is set to 87381 on a mac OSX 15. This is very high.
- But I can't create more than about 4900 semaphores regardless of the number of python instances. I get an OSError(28, 'No space left on device'). What is the right value ?
- When and how to call the system function
shm_unlink to properly close the shared memory (First I called shm_open)?
Regarding this last point, I wonder if only the process that opened the shared memory should call this function? Or should all processes that called shm_open call shm_unlink? Unfortunately, the Posix documentation isn't precise.
I worked on the count of opened/closed processes, with the aim of calling shm_unlink when the last process has closed, without success. I also had a look to the posixshmem module, without success too.
Could the resource_tracker and/or the use of the Finalize class of the multiprocessing.util module help me to fix this problem ? Is it possible to use these 2 Python libs from C API ?
This memory management difficulty make me think that this choice may not be appropriate. I am a bit stuck.
Thank you for your feedback and ideas.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
This PR is stale because it has been open for 30 days with no activity.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
1. All past blocking points are resolved.
2. Atomicity,
+ Implementation and rebuild Semlock are atomic (protected via the dedicated semaphore),
+ Release, get_value and is_zero methods are atomic too (protected via the mutex of each semaphore),
+ Acquire function is nearly-atomic.
3. Shared memory,
+ The shared memory and its associated semphore have unique names based on
the main process ID.
+ a new counter was added to count pending thread
when blocking acquire operations happen.
+ When a Semlock is destroyed, the shared memory and its dediocted semaphore are destroyed too
if there is no more opened Semclock.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
+ refactoring code on SemLock_imp and SemLock__rebuild_impl,
+ change conditional compilation value PyDEBUG to Py_REF_DEBUG,
+ add sizeof CounteObject in the header
+ add 3 unit tests
YvesDup
commented
Jun 2, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
This last version is always based on a shared memory protected with a lock.
- when the first semaphore is created, the shared memory is created with its lock,
- each Semaphore updates its counters which are stored into the shared memory,
- when there is no more semaphore, the shared memory and its associated lock are destroyed.
Names of these shared memory and lock are unique for the main process and all its child processes.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
- Fix bug: the lost of errno value in SemLock_impl,
- Reduce shared memory sise, used kern.posix.max.sem value
instaed of SC_SEM_NSEMS_MAX value for array size of shared memeory,
- Add a unit test that raises an error when the maximum opened sems is reached.
Last code reviews are relaized with Claude code
…/check-c-globals.py`` tool
|
This PR is stale because it has been open for 90 days with no activity.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
gpshead
Awaiting requested review from gpshead
gpshead is a code owner
ericsnowcurrently
Awaiting requested review from ericsnowcurrently
ericsnowcurrently is a code owner
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|