GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-125828 : Fix 'get_value' missing for [Bounded]Semaphore on multiprocessing MacOSX - C workaround by YvesDup · Pull Request #130913 · python/cpython

Conversation

@YvesDup

@YvesDup YvesDup commented Mar 6, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Following #125829 discussions and after a long deeping in the _multiprocessing.semaphore.c C file, I suggest a workaround based on a shared memory that contains an array to store all the internal semaphore counters.

At the moment, all the tests are passing, but I'm having 2 side problems, the second of which is really embarrassing:

  1. How to calculate the size of the array stored in the shared memory, knowing that:
    1. I sized the array from the value of sysconf(SC_SEM_NSEMS_MAX) which is set to 87381 on a mac OSX 15. This is very high.
    2. But I can't create more than about 4900 semaphores regardless of the number of python instances. I get an OSError(28, 'No space left on device'). What is the right value ?
  2. When and how to call the system function shm_unlink to properly close the shared memory (First I called shm_open)?

Regarding this last point, I wonder if only the process that opened the shared memory should call this function? Or should all processes that called shm_open call shm_unlink? Unfortunately, the Posix documentation isn't precise.

I worked on the count of opened/closed processes, with the aim of calling shm_unlink when the last process has closed, without success. I also had a look to the posixshmem module, without success too.
Could the resource_tracker and/or the use of the Finalize class of the multiprocessing.util module help me to fix this problem ? Is it possible to use these 2 Python libs from C API ?

This memory management difficulty make me think that this choice may not be appropriate. I am a bit stuck.
Thank you for your feedback and ideas.

  • Issue: Multiprocessing Semaphore and BoundedSemaphore - incomplete features ('get_value' missing) on MacOSX. #125828

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@YvesDup
Initial commit
7095e9e
@YvesDup
YvesDup requested a review from gpshead as a code owner March 6, 2025 13:58
@bedevere-app bedevere-app Bot added the awaiting review label Mar 6, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Mar 6, 2025
Multiprocessing Semaphore and BoundedSemaphore - incomplete features ('get_value' missing) on MacOSX. #125828
Open
blurb-it Bot and others added 4 commits March 6, 2025 18:52
@blurb-it
📜🤖 Added by blurb_it.
89676c1
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
58df0a4
@YvesDup
Fix nits
954048f
@YvesDup
Update test_bounded_semaphore in order to test upper limit on MacOSX
52144ae
@YvesDup
YvesDup marked this pull request as draft April 15, 2025 12:32
@bedevere-app bedevere-app Bot removed the awaiting review label Apr 15, 2025
YvesDup added 15 commits April 15, 2025 14:33
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
8eb9f30
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
a8c3925
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
be6972f
@YvesDup
refactor code, remove global lock and shared memory when count of sem…
065921b
…aphore is 0
@YvesDup
remove unsed 'semlock_traverse' function
8c8d025
@YvesDup
Fix nits in news file
56cba95
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
2cdda38
@YvesDup
Fix another nits
20cde3b
@YvesDup
Move variable declaration and refactor code on dump tools
14399db
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
c8a7fbb
@YvesDup
Merge branch 'python:main' into sem-macosx-multiprocessing-module-C
006b77c
@YvesDup
Merge branch 'python:main' into sem-macosx-multiprocessing-module-C
36e298c
@YvesDup
Merge branch 'python:main' into sem-macosx-multiprocessing-module-C
79df976
@YvesDup
Merge branch 'python:main' into sem-macosx-multiprocessing-module-C
1a12f54
@YvesDup
Merge branch 'python:main' into sem-macosx-multiprocessing-module-C
10e3d6c
@github-actions

github-actions Bot commented Apr 22, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 22, 2026
YvesDup added 2 commits April 27, 2026 13:14
@YvesDup
Merge branch 'python:main' into sem-macosx-multiprocessing-module-C
b446eff
@YvesDup
Final version including:
9c8b9cb
1. All past blocking points are resolved.

2. Atomicity,
+ Implementation and rebuild Semlock are atomic (protected via the dedicated semaphore),
+ Release, get_value and is_zero methods are atomic too (protected via the mutex of each semaphore),
+ Acquire function is nearly-atomic.

3. Shared memory,
+ The shared memory and its associated semphore have unique names based on
    the main process ID.
+ a new counter  was added to count pending thread
    when blocking acquire operations happen.
+ When a Semlock is destroyed, the shared memory and its dediocted semaphore are destroyed too
    if there is no more opened Semclock.
@read-the-docs-community

read-the-docs-community Bot commented May 7, 2026

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #32588785 | 📁 Comparing 9c8b9cb against main (299cf3c)

  🔍 Preview build  

80 files changed · ± 78 modified · - 2 deleted

± Modified

  • glossary.html
  • c-api/concrete.html
  • c-api/exceptions.html
  • c-api/extension-modules.html
  • c-api/index.html
  • c-api/interp-lifecycle.html
  • c-api/lifecycle.html
  • c-api/module.html
  • c-api/perfmaps.html
  • c-api/stable.html
  • and 68 more...

- Deleted

  • c-api/sentinel.html
  • c-api/slots.html
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

YvesDup added 4 commits May 7, 2026 18:42
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
45017a9
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
8af3be7
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
3983281
@YvesDup
Fix nits in news
ce5dc4c
@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label May 14, 2026
YvesDup added 3 commits June 1, 2026 16:16
@YvesDup
Last update including:
6104c6e
+ refactoring code on SemLock_imp and SemLock__rebuild_impl,
+ change conditional compilation value PyDEBUG to Py_REF_DEBUG,
+ add sizeof CounteObject in the header
+ add 3	 unit tests
@YvesDup
Fix unused variable result in SemLock__rebuild
2722654
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
ee231f6
@YvesDup

YvesDup commented Jun 2, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

This last version is always based on a shared memory protected with a lock.

  • when the first semaphore is created, the shared memory is created with its lock,
  • each Semaphore updates its counters which are stored into the shared memory,
  • when there is no more semaphore, the shared memory and its associated lock are destroyed.

Names of these shared memory and lock are unique for the main process and all its child processes.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@YvesDup
YvesDup marked this pull request as ready for review June 2, 2026 09:52
@bedevere-app bedevere-app Bot added the awaiting review label Jun 2, 2026
YvesDup added 4 commits June 2, 2026 14:16
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
d870e18
@YvesDup
Fix bug and refactor code
72c74e7
- Fix bug: the lost of errno value in SemLock_impl,
- Reduce shared memory sise, used kern.posix.max.sem value
instaed of SC_SEM_NSEMS_MAX value for array size of shared memeory,
- Add a unit test that raises an error when the maximum opened sems is reached.

Last code reviews are relaized with Claude code
@YvesDup
Move 'int err;' decl
c9829f4
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
c50aaa3
@YvesDup YvesDup mentioned this pull request Jun 5, 2026
From CI, Tools/c-analyzer/check-c-globals.py failed #150948
Closed
@StanFromIreland StanFromIreland mentioned this pull request Jun 20, 2026
gh-150948: Exclude macOS semaphore header from c-analyzer #151777
Closed
@StanFromIreland
StanFromIreland requested a review from a team June 21, 2026 17:28
YvesDup added 2 commits June 22, 2026 10:29
@YvesDup
Refactor code, remove a static variable
05b0e0a
@YvesDup
Add semaphore_macosx.h to files to exclude with `./Tools/c-analyzer…
10b7a75
…/check-c-globals.py`` tool
@YvesDup
YvesDup requested a review from ericsnowcurrently as a code owner June 22, 2026 12:59
@YvesDup
Merge branch 'main' into sem-macosx-multiprocessing-module-C
d61ced1
@github-actions

github-actions Bot commented Oct 30, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 90 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review stale Stale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@YvesDup
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.