GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-116167: Allow disabling the GIL with `PYTHON_GIL=0` or `-X gil=0` by swtaarrs · Pull Request #116338 · python/cpython

Conversation

@swtaarrs

@swtaarrs swtaarrs commented Mar 4, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

In free-threaded builds, running with PYTHON_GIL=0 or -X gil=0 will now disable the GIL. #116322 and #116329 track follow-up work to re-enable the GIL when loading an incompatible extension, and to disable the GIL by default, respectively.

In order to support re-enabling the GIL at runtime, all GIL-related data structures are initialized as usual, and disabling the GIL simply sets a flag at startup that causes take_gil() and drop_gil() to return early.

With PYTHON_GIL=0 set, I spot-checked a few tests and small programs that don't use threads. They all seem to run fine, and very basic threaded programs work, sometimes. Trying to run the full test suite crashes pretty quickly, in test_asyncio.

  • Issue: Add a mechanism to disable the GIL #116167

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
colesbury, corona10, erlend-aasland, philippjfr, avinayak, tri-chu, tomwojcik, Josua996, jotamudo, alexfromapex, and 19 more reacted with hooray emoji itamaro, Eclips4, corona10, erlend-aasland, philippjfr, avinayak, tomwojcik, alexfromapex, jacoblearned, tizz98, and 15 more reacted with rocket emoji
All reactions
  • 🎉 29 reactions
  • 🚀 25 reactions
@swtaarrs swtaarrs added the topic-free-threading label Mar 4, 2024
@swtaarrs
swtaarrs requested a review from colesbury March 4, 2024 22:58
@bedevere-app bedevere-app Bot mentioned this pull request Mar 4, 2024
Add a mechanism to disable the GIL #116167
Closed
@swtaarrs swtaarrs added the skip news label Mar 4, 2024
@swtaarrs

swtaarrs commented Mar 4, 2024

Copy link
Copy Markdown
Member Author

!buildbot nogil
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Mar 4, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit ac24233 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • x86-64 MacOS Intel ASAN NoGIL PR
  • x86-64 MacOS Intel NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • ARM64 MacOS M1 NoGIL PR
  • AMD64 Ubuntu NoGIL Refleaks PR
  • AMD64 Ubuntu NoGIL PR
  • AMD64 Windows Server 2022 NoGIL PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@swtaarrs
Allow disabling the GIL with PYTHON_GIL=0
9e03999 
In free-threaded builds, running with `PYTHON_GIL=0` will now disable the
GIL. Follow-up issues track work to re-enable the GIL when loading an
incompatible extension, and to disable the GIL by default.

In order to support re-enabling the GIL at runtime, all GIL-related data
structures are initialized as usual, and disabling the GIL simply sets a flag
that causes `take_gil()` and `drop_gil()` to return early.

With `PYTHON_GIL=0` set, I spot-checked a few tests and small programs that
don't use threads. They all seem to run fine, and very basic threaded programs
work, sometimes. Trying to run the full test suite crashes pretty quickly, in
`test_asyncio`.
@swtaarrs
swtaarrs force-pushed the cpython-disable-gil branch from ac24233 to 9e03999 Compare March 5, 2024 20:38
@swtaarrs

swtaarrs commented Mar 5, 2024

Copy link
Copy Markdown
Member Author

!buildbot nogil
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Mar 5, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit 9e03999 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • x86-64 MacOS Intel ASAN NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • AMD64 Windows Server 2022 NoGIL PR
  • AMD64 Ubuntu NoGIL PR
  • x86-64 MacOS Intel NoGIL PR
  • AMD64 Ubuntu NoGIL Refleaks PR
  • ARM64 MacOS M1 NoGIL PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
corona10
corona10 reviewed Mar 5, 2024
View reviewed changes

@corona10 corona10 left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to add test case in this file too.

cpython/Lib/test/test_cmd_line.py

Line 896 in 72714c0

def test_int_max_str_digits(self):

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@corona10

corona10 commented Mar 5, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Ah ignore above comments, this PR only referencing the environment variable so not proper for this case.

You can just add the testcase for the environment variable, still worth to adding it.
swtaarrs reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10

corona10 commented Mar 5, 2024

Copy link
Copy Markdown
Member

At not for this PR, since this feature is not officially announced, but in some moment, you should write about the environment variable at https://github.com/python/cpython/blob/72714c0266ce6d39c7c7fb63f617573b8f5a3cb2/Doc/using/cmdline.rst to get user feedback.
swtaarrs reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
swtaarrs added 3 commits March 5, 2024 13:40
@swtaarrs
Don't add 'enable_gil' to test_embed in normal builds
a53ed2e
@swtaarrs
Set enable_gil properly when PYTHON_GIL=1
e0b8484
@swtaarrs
Add test for PYTHON_GIL in test_cmd_line
4f43a87
@swtaarrs

swtaarrs commented Mar 5, 2024

Copy link
Copy Markdown
Member Author

!buildbot nogil
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Mar 5, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit 4f43a87 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • x86-64 MacOS Intel ASAN NoGIL PR
  • ARM64 MacOS M1 NoGIL PR
  • AMD64 Ubuntu NoGIL PR
  • x86-64 MacOS Intel NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • AMD64 Ubuntu NoGIL Refleaks PR
  • AMD64 Windows Server 2022 NoGIL PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@swtaarrs
swtaarrs marked this pull request as ready for review March 5, 2024 22:56
@swtaarrs
swtaarrs requested a review from ericsnowcurrently as a code owner March 5, 2024 22:56
@bedevere-app bedevere-app Bot added the awaiting review label Mar 5, 2024
@swtaarrs
swtaarrs requested a review from vstinner March 5, 2024 23:16
@swtaarrs

swtaarrs commented Mar 5, 2024

Copy link
Copy Markdown
Member Author

you should write about the environment variable ... to get user feedback

What would be a good time to do that? Maybe once the GIL is disabled by default, so the description doesn't have to change?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
colesbury
colesbury reviewed Mar 5, 2024
View reviewed changes
Comment thread Python/initconfig.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@swtaarrs
Fix else brace formatting
9e9d925
@colesbury

colesbury commented Mar 6, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

I would document the environment variable now, when it's added. For reference, we already document the --disable-gil configure flag.

I don't think it needs to be extensive, but should cover:

  • Only available in free-threaded builds
  • 0 forces the GIL to be disabled
  • 1 forces the GIL to be enabled
  • New in version 3.13

The behavior when then environment variable is not set is what's going to change in the future. We can document that later.
swtaarrs, ambv, corona10, and erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 4 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@swtaarrs
Document PYTHON_GIL environment variable
5053661
@swtaarrs

swtaarrs commented Mar 7, 2024

Copy link
Copy Markdown
Member Author

After discussion in Discord, I'm also going to add a -X option to go with the environment variable.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@swtaarrs
swtaarrs marked this pull request as draft March 7, 2024 14:31
@bedevere-app bedevere-app Bot removed the awaiting review label Mar 7, 2024
swtaarrs added 2 commits March 7, 2024 12:25
@swtaarrs
Merge remote-tracking branch 'upstream/main' into HEAD
6d62fbb
@swtaarrs
Add -X gil option, add to sys.flags, modify test to cover env var…
971a9a1 
… and option
@swtaarrs swtaarrs changed the title gh-116167: Allow disabling the GIL with PYTHON_GIL=0 gh-116167: Allow disabling the GIL with PYTHON_GIL=0 or -X gil=0 Mar 7, 2024
swtaarrs added 2 commits March 7, 2024 12:49
@swtaarrs
Fix link to -X gil
89b1150
@swtaarrs
Fix PYTHON_GIL versionchanged line
16fbd44
@swtaarrs
swtaarrs marked this pull request as ready for review March 7, 2024 22:29
@swtaarrs
swtaarrs requested a review from gpshead as a code owner March 7, 2024 22:29
@bedevere-app bedevere-app Bot added the awaiting review label Mar 7, 2024
@colesbury
colesbury requested a review from erlend-aasland March 7, 2024 22:47
ericsnowcurrently
ericsnowcurrently approved these changes Mar 8, 2024
View reviewed changes

@ericsnowcurrently ericsnowcurrently left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I've left one minor comment about an unclear test. It really isn't a blocker, as the test is passing, but it would still be nice to have a response.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/_test_embed_set_config.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Mar 8, 2024
erlend-aasland
erlend-aasland approved these changes Mar 9, 2024
View reviewed changes
corona10
corona10 approved these changes Mar 10, 2024
View reviewed changes
@swtaarrs
Clarify test_flags in normal builds
8d39bbc
colesbury
colesbury approved these changes Mar 11, 2024
View reviewed changes
@colesbury
colesbury merged commit 2731913 into python:main Mar 11, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Mar 11, 2024
@vstinner

vstinner commented Mar 11, 2024

Copy link
Copy Markdown
Member

This is both exciting and very scary. Congrats @swtaarrs and @colesbury :-)
s373r and edpyt reacted with hooray emoji swtaarrs, s373r, matej-mnoucek-adsk, and edpyt reacted with heart emoji jezdez, cpvandehey, nkhitrov, edpyt, K0IN, and zaphodschmidt reacted with rocket emoji
All reactions
  • 🎉 2 reactions
  • ❤️ 4 reactions
  • 🚀 6 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@alikud

This comment was marked as off-topic.

Sign in to view
@edpyt

This comment was marked as off-topic.

Sign in to view
@fractalvision

This comment was marked as off-topic.

Sign in to view
@python python locked as spam and limited conversation to collaborators Mar 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@colesbury colesbury colesbury approved these changes
@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes
@corona10 corona10 corona10 approved these changes
@erlend-aasland erlend-aasland erlend-aasland approved these changes
@vstinner vstinner Awaiting requested review from vstinner
@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

10 participants

@swtaarrs @bedevere-bot @corona10 @colesbury @vstinner @alikud @edpyt @fractalvision @ericsnowcurrently @erlend-aasland
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.