GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-124984: Fix `ssl` thread safety by ZeroIntensity · Pull Request #124993 · python/cpython

Conversation

@ZeroIntensity

@ZeroIntensity ZeroIntensity commented Oct 5, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

As it turns out, OpenSSL doesn't like being called in multiple threads. This adds a per-socket (and per-context and per-session) lock for all OpenSSL calls.

  • Issue: Segfault on Python 3.13.0rc3 (free-threaded) with requests #124984

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ZeroIntensity added 15 commits October 4, 2024 22:21
@ZeroIntensity
Make SSL objects thread safe.
132a2dd
@ZeroIntensity
Fix silly syntax errors.
572e48f
@ZeroIntensity
Fix thread safety for set_alpn_protocols
b0cd005
@ZeroIntensity
Fix thread safety for configure_hostname
0629acd
@ZeroIntensity
Make some more operations atomic.
3712d05
@ZeroIntensity
Fix thread safety with hostnames.
eb06e05
@ZeroIntensity
Fix thread safety for BIO.
30fce36
@ZeroIntensity
Add blurb.
0d63bfb
@ZeroIntensity
Add test.
7268ebb
@ZeroIntensity
Fix deadlock.
28004f1
@ZeroIntensity
Remove useless locks in constructor.
caa205a
@ZeroIntensity
Zero-out locks upon initialization.
578d40a
@ZeroIntensity
Wrap lock initializations with #ifdef Py_GIL_DISABLED
c601906
@ZeroIntensity
Fix lock ordering deadlock.
014a95a
@ZeroIntensity
Remove lock for do_handshake()
ebb8b40
@bedevere-app bedevere-app Bot added the awaiting review label Oct 5, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Oct 5, 2024
Segfault on Python 3.13.0rc3 (free-threaded) with requests #124984
Closed
@ZeroIntensity
ZeroIntensity requested review from AA-Turner and colesbury October 5, 2024 04:25
@ZeroIntensity ZeroIntensity added topic-SSL topic-free-threading labels Oct 5, 2024
ZeroIntensity added 3 commits October 5, 2024 00:28
@ZeroIntensity
Fix typo.
a3ff154
@ZeroIntensity
Make tests better.
591535f
@ZeroIntensity
Fix tests.
53e6d59
JelleZijlstra
JelleZijlstra reviewed Oct 5, 2024
View reviewed changes
Comment thread Lib/test/test_ssl.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

JelleZijlstra
JelleZijlstra reviewed Oct 5, 2024
View reviewed changes

@JelleZijlstra JelleZijlstra left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that's a lot of locks.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity reacted with rocket emoji
All reactions
  • 🚀 1 reaction
Comment thread Modules/_ssl.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_ssl.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_ssl.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Oct 5, 2024
View reviewed changes
Comment thread Modules/_ssl.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_ssl.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_ssl.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_ssl.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity and others added 3 commits October 5, 2024 10:14
@ZeroIntensity
Fix missing global.
cd047ba
@ZeroIntensity
Update comment.
3f3715b
@ZeroIntensity @picnixz
Update Modules/_ssl.c
d808932
Co-authored-by: Bénédikt Tran <[email protected]>
@vstinner

vstinner commented Oct 19, 2024

Copy link
Copy Markdown
Member

Remaining question: should we backport this enhancement (bugfix?) to Python 3.13?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Oct 19, 2024

Copy link
Copy Markdown
Member Author

Remaining question: should we backport this enhancement (bugfix?) to Python 3.13?

Yeah, I had the same thought. I'll leave that decision to @Yhg1s

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Yhg1s

Yhg1s commented Oct 20, 2024

Copy link
Copy Markdown
Member

Backport this to 3.13? Uuhm...

image

No.

ZeroIntensity reacted with laugh emoji
All reactions
  • 😄 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Oct 20, 2024

Copy link
Copy Markdown
Member

Without this change, the ssl module is not usable on a Free Threaded build (it does easily crash). The change only affects the Free Threaded build: adding @critical_section does nothing in the regular build, and the added test is ignored on regular Python (only run on Free Threading). I would be fine with no backporting it, since Free Threading remains experimental.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@colesbury

colesbury commented Oct 20, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

I think that fixing the SSL module crash in the 3.13 free threading build is important -- lots of basic tasks around HTTP requests are likely to crash without it.

I don't think "lines of code changed" is a good measure of the complexity here -- the Modules/_ssl.c changes are mostly mechanical, and the only behavioral changes is additional locking in the free threading build.

If backporting this PR is a nonstarter than we should consider a smaller, more targeted change that only adds @critical_section to SSLContext methods (without changing getters and setters). That seems less ideal to me, but should fix the common SSL crash and be a pretty small code change.

JelleZijlstra, AA-Turner, dolfinus, and ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 4 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@colesbury

colesbury commented Oct 20, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Also, thank you @ZeroIntensity for fixing this bug and @vstinner, @corona10, and everyone else that reviewed the PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner added the needs backport to 3.13 only security fixes label Oct 21, 2024
@miss-islington-app

miss-islington-app Bot commented Oct 21, 2024

Copy link
Copy Markdown

Thanks @ZeroIntensity for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington-app

miss-islington-app Bot commented Oct 21, 2024

Copy link
Copy Markdown

Sorry, @ZeroIntensity and @vstinner, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 4c53b2577531c77193430cdcd66ad6385fcda81f 3.13
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington-app miss-islington-app Bot assigned vstinner Oct 21, 2024
@vstinner

vstinner commented Oct 21, 2024

Copy link
Copy Markdown
Member

@ZeroIntensity: Automated backport failed. Would you mind to backport the change manually?

With a backport, it might be easier to take a decision on fixing 3.13 or not.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Oct 21, 2024

Copy link
Copy Markdown
Member Author

Yeah, I can do it later today.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity ZeroIntensity assigned ZeroIntensity and unassigned vstinner Oct 21, 2024
@bedevere-app

bedevere-app Bot commented Oct 21, 2024

Copy link
Copy Markdown

GH-125780 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Oct 21, 2024
ZeroIntensity added a commit to ZeroIntensity/cpython that referenced this pull request Oct 21, 2024
@ZeroIntensity @picnixz
[3.13] pythongh-124984: Enhance ssl thread safety (pythonGH-124993)
1b4b996
Make SSL objects thread safe in Free Theaded build by
using critical sections.

(cherry picked from commit 4c53b25)

Co-authored-by: Peter Bierma <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
@vstinner

vstinner commented Oct 29, 2024

Copy link
Copy Markdown
Member

@Yhg1s: Are you still against the backport to 3.13 after @colesbury's comment?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Yhg1s

Yhg1s commented Nov 4, 2024

Copy link
Copy Markdown
Member

I'm okay with a backport of just the @critical_section changes (since those expand to nothing in the normal build, and the free-threaded build is experimental anyway). It's the larger refactorings that worry me.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 4, 2024

Copy link
Copy Markdown
Member

It's the larger refactorings that worry me.

Other changes are tests and changes to use the code declared with @critical_section.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Nov 4, 2024

Copy link
Copy Markdown
Member Author

There isn't any other refactoring going on here, I just had to switch the getters and setters over to AC for the critical section. Another issue is that not backporting this to 3.13 will also hurt any automatic backports for ssl in the future.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity ZeroIntensity mentioned this pull request Nov 5, 2024
Calling many methods on _ssl._SSLSocket() segfaults #126455
Closed
@ZeroIntensity ZeroIntensity mentioned this pull request Nov 15, 2024
Restore docstrings in ssl #126890
Closed
@ZeroIntensity ZeroIntensity mentioned this pull request Nov 26, 2024
gh-126890: Restore stripped ssl docstrings #127281
Merged
Yhg1s pushed a commit that referenced this pull request Dec 2, 2024
@ZeroIntensity @picnixz
[3.13] gh-124984: Enhance ssl thread safety (GH-124993) (#125780)
b105157
Make SSL objects thread safe in Free Theaded build by
using critical sections.

(cherry picked from commit 4c53b25)

Co-authored-by: Peter Bierma <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
@ZeroIntensity @picnixz @ebonnal
pythongh-124984: Enhance ssl thread safety (python#124993)
186f52f
Make SSL objects thread safe in Free Theaded build by
using critical sections.

Co-authored-by: Bénédikt Tran <[email protected]>
@ZeroIntensity ZeroIntensity mentioned this pull request Jan 17, 2025
gh-128942: make arraymodule.c free-thread safe #128943
Merged
youknowone
youknowone reviewed Oct 28, 2025
View reviewed changes
Comment thread Modules/clinic/_ssl.c.h
#endif
#if defined(_SSL__SSLCONTEXT_SECURITY_LEVEL_GETSETDEF)
# undef _SSL__SSLCONTEXT_SECURITY_LEVEL_GETSETDEF
# define _SSL__SSLCONTEXT_SECURITY_LEVEL_GETSETDEF {"security_level", (getter)_ssl__SSLContext_security_level_get, (setter)_ssl__SSLContext_security_level_set, _ssl__SSLContext_security_level_DOCSTR},

@youknowone youknowone Oct 28, 2025

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The symbol _ssl__SSLContext_security_level_set is used here, but nowhere added

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ZeroIntensity ZeroIntensity Oct 28, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generated clinic code. I'm pretty sure this route is never taken, and the macro right after this is used instead.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@youknowone youknowone Oct 28, 2025

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, thank you. I didn't know that

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@youknowone youknowone mentioned this pull request Oct 28, 2025
Unexisting function _ssl__SSLContext_security_level_set is referenced in source code #140692
Closed
@kumaraditya303 kumaraditya303 mentioned this pull request Oct 30, 2025
performance regression in ssl module under free-threading #140795
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead left review comments
@vstinner vstinner vstinner approved these changes
@JelleZijlstra JelleZijlstra JelleZijlstra approved these changes
@corona10 corona10 corona10 approved these changes
@picnixz picnixz picnixz approved these changes
@AA-Turner AA-Turner Awaiting requested review from AA-Turner
@colesbury colesbury Awaiting requested review from colesbury
+1 more reviewer
@youknowone youknowone youknowone left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@ZeroIntensity ZeroIntensity

Labels

topic-free-threading topic-SSL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

12 participants

@ZeroIntensity @picnixz @AA-Turner @efimov-mikhail @colesbury @vstinner @bedevere-bot @Yhg1s @gpshead @youknowone @JelleZijlstra @corona10
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.