GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-126316: Make grp.getgrall() thread-safe: add a mutex by vstinner · Pull Request #127055 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Nov 20, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

grpmodule.c is no longer built with the limited C API, since PyMutex is excluded from the limited C API.

  • Issue: grp is not thread safe #126316

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
pythongh-126316: Make grp.getgrall() thread-safe: add a mutex
40744e4
grpmodule.c is no longer built with the limited C API, since PyMutex
is excluded from the limited C API.
@bedevere-app bedevere-app Bot mentioned this pull request Nov 20, 2024
grp is not thread safe #126316
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Nov 20, 2024
@vstinner

vstinner commented Nov 20, 2024

Copy link
Copy Markdown
Member Author

@ZeroIntensity @colesbury: Would you mind to review my change?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity reviewed Nov 20, 2024
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this is an interesting way to do it. Local static variables are notorious for being non-thread-safe, so I'm not sure this kind of approach will work. If it does, I don't think we're fixing any issues with re-entrancy.

I would just go with @critical_section on each function, and then switch the module's Py_mod_multiple_interpreters to Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED instead of Py_MOD_PER_INTERPRETER_GIL_SUPPORTED to denote shared-GIL for subinterpreters. That should fix races for both cases.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
colesbury
colesbury approved these changes Nov 20, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach looks good to me. I think you are right that we don't have to worry about deadlock here because the calls inside the lock won't call arbitrary Python code.

There's a failing CI related to generated files. I think the pycore_lock.h include in pycore_modsupport.h can be removed now. Also, maybe the arg clinic files need to be regenerated?

Local static variables are notorious for being non-thread-safe

A static PyMutex is fine because PyMutex is thread-safe. (It doesn't matter if it's a local or global static variable.)

The general problems with static variables (including static local variables) is that the data structures are not thread-safe or that they are lazily initialized pointers, which is not the case here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Nov 20, 2024
@ZeroIntensity

ZeroIntensity commented Nov 20, 2024

Copy link
Copy Markdown
Member

Thanks for the insight, Sam. How scalable is this approach (as in, will we have a static PyMutex everywhere)?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@colesbury

colesbury commented Nov 20, 2024

Copy link
Copy Markdown
Contributor

I don't think we'll have static PyMutex everywhere, but this is a good pattern for certain libc functions that are not thread-safe and do not have thread-safe alternatives.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added 2 commits November 21, 2024 10:37
@vstinner
Run make regen-all
f1caa08
@vstinner
Argument Clinic uses the internal C API
028a136
@vstinner

vstinner commented Nov 21, 2024

Copy link
Copy Markdown
Member Author

I think the pycore_lock.h include in pycore_modsupport.h can be removed now.

Correct: I wrote #127093 to remove it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity reviewed Nov 21, 2024
View reviewed changes
Comment thread Modules/grpmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
Fix make check-c-globals
95a12f5
@vstinner
vstinner requested a review from ericsnowcurrently as a code owner November 21, 2024 14:13
ZeroIntensity
ZeroIntensity approved these changes Nov 21, 2024
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ZeroIntensity ZeroIntensity mentioned this pull request Nov 21, 2024
Allow local use of static PyMutex in the C analyzer #127102
Merged
@vstinner
vstinner merged commit 3c2bd66 into python:main Nov 21, 2024
@vstinner
vstinner deleted the getgrall_mutex branch November 21, 2024 14:47
@bedevere-app bedevere-app Bot removed the awaiting merge label Nov 21, 2024
@vstinner

vstinner commented Nov 21, 2024

Copy link
Copy Markdown
Member Author

Merged, thanks for reviews @colesbury and @ZeroIntensity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner mentioned this pull request Nov 21, 2024
gh-126316: Use mutexes in the grp module #126504
Closed
@bedevere-app

bedevere-app Bot commented Nov 21, 2024

Copy link
Copy Markdown

GH-127104 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added a commit to vstinner/cpython that referenced this pull request Nov 21, 2024
@vstinner
pythongh-126316: Make grp.getgrall() thread-safe: add a mutex (python…
c36d77d
…#127055)

grpmodule.c is no longer built with the limited C API, since PyMutex
is excluded from the limited C API.

(cherry picked from commit 3c2bd66)
vstinner added a commit that referenced this pull request Nov 26, 2024
@vstinner
[3.13] gh-126316: Make grp.getgrall() thread-safe: add a mutex (#127055…
9187ac3
…) (#127104)

* gh-126316: Make grp.getgrall() thread-safe: add a mutex (#127055)

grpmodule.c is no longer built with the limited C API, since PyMutex
is excluded from the limited C API.

(cherry picked from commit 3c2bd66)

* Revert ABI changes

Don't use Argument Clinic for grp.getgrgid() to avoid changing the
ABI (change PyInterpreterState structure by adding an "id"
identifier).
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
@vstinner @ebonnal
pythongh-126316: Make grp.getgrall() thread-safe: add a mutex (python…
2629000
…#127055)

grpmodule.c is no longer built with the limited C API, since PyMutex
is excluded from the limited C API.
@yoney yoney mentioned this pull request Jul 15, 2025
gh-116738: Make pwd module thread-safe #136695
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury approved these changes
@ZeroIntensity ZeroIntensity ZeroIntensity approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@vstinner @ZeroIntensity @colesbury
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.