GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-128384: Use a context variable for warnings.catch_warnings by nascheme · Pull Request #130010 · python/cpython

Conversation

@nascheme

@nascheme nascheme commented Feb 11, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Make warnings.catch_warnings() use a context variable for holding the warning filtering state if the sys.flags.context_aware_warnings flag is set to true. This makes using the context manager thread-safe in multi-threaded programs and safe for asyncio coroutines and tasks. The flag is true by default in free-threaded builds and is otherwise false. The value of the flag can be overridden by the the -X thread_safe_warnings command-line option or by the PYTHON_CONTEXT_AWARE_WARNINGS environment variable.

It is expected that one day the flag might default to true for all Python builds, not just the free-threaded one. However, I think it makes sense to not commit to a schedule to do that until we have a better idea of what code is affected by this change. Feedback from people using the free-threaded build should provide guidance.

This PR is on top of gh-128209.

A previous version of this PR used a single flag to control both behavior of Thread inheriting the context and also the catch_warnings context manager. However, based on some feedback, I've decided that two flags makes things more clear. Likely programs would like to set both flags to true to get the most intuitive behavior.

When the context_aware_warnings flag is true and a catch_warnings() context is active, added filters go into a list of filters stored in the contextvar, not the warnings.filters list. The filters in warnings.filters are still applied but they apply after the contextvar filters.

That difference with how warnings.filters works is probably the most likely thing to cause broken user code. In the unit tests, I had to change a number of references to warnings.filters to warnings._get_filters(). That function returns the list of filters from the current context or the global filters if there is no context active. Perhaps _get_filters() should become a public function? I think it would be better if examining and manipulating that list was discouraged and so that's why I made the function non-public, at least for now.

I created _py_warnings.py to contain the Python implementation for the warnings module. This matches other modules like decimal that have Python and C versions of the same module. In the case of warnings things are a bit more complicated since you can assign to module globals (the filters list or the showwarning() function are common to override). This is a cleaner organization, IMHO. Now warnings.py just imports the parts it needs and the unit tests are a bit simpler.

I cleaned up the unit tests for warnings a bit. Passing module to catch_warnings() actually serves no purpose. Since sys.modules['warnings'] is already swapped to the module under test, the code does the right thing even when that parameter is not given. I also replaced calls to original_warnings.catch_warnings() with self.module.catch_warnings(). Both those calls do the same thing but the second spelling seems less confusing to me.

  • Issue: warnings.catch_warnings is not thread safe or async safe #128384

pyperformance comparison


📚 Documentation preview 📚: https://cpython-previews--130010.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

colesbury, ngoldbaum, lysnikolaou, and Jackenmen reacted with hooray emoji
All reactions
  • 🎉 4 reactions
nascheme added 6 commits February 6, 2025 12:53
@nascheme
Make _contextvars a builtin module.
ba99f2e
@nascheme
Add 'context' parameter to Thread.
6d00c2a
* Add ``sys.flags.inherit_context``.
* Add ``-X inherit_context`` and :envvar:`PYTHON_INHERIT_CONTEXT`
@nascheme
Tweak blurb markup.
a868fe9
@nascheme
Doc markup fix.
16fa2c3
@nascheme
Use contextvar for catch_warnings().
f0ccc8d
@nascheme
Add blurb.
bcadd20
@bedevere-app bedevere-app Bot mentioned this pull request Feb 11, 2025
warnings.catch_warnings is not thread safe or async safe #128384
Closed
@nascheme
Add "_warnings_context" as identifier.
0cfe578
@nascheme nascheme mentioned this pull request Feb 11, 2025
gh-128384: Add thread-safe context manager to "warnings" module #128300
Closed
nascheme added 2 commits February 11, 2025 11:07
@nascheme
Fix test_support for context var filters.
928c2df
Since the unittest/runner.py establishes a new warnings context with
`catch_warnings`, these tests must use `warnings._get_filters()` to
retrieve the current list of filters, not the `warnings.filters` global.
@nascheme
Regenerate 'configure' script.
65751d9
@colesbury colesbury mentioned this pull request Feb 11, 2025
gh-129983: fix data race in compile_template in sre.c #130015
Merged
nascheme added 6 commits February 11, 2025 15:23
@nascheme
Rename flag to thread_inherit_context.
09e72b8
@nascheme
Merge branch 'main' into thread_inherit_context
75f1b38
@nascheme
Regenerate 'configure' script.
872920d
@nascheme
Merge branch 'thread_inherit_context' into pythongh-128384-warnings-c…
5b6b59e
…ontextvar
@nascheme
Use separate flag thread_safe_warnings.
9e955dc
@nascheme
Add doc for thread_safe_warnings flag.
3a56c64
@nascheme
nascheme marked this pull request as ready for review February 13, 2025 21:03
@nascheme
nascheme requested review from a team, 1st1, corona10, ericsnowcurrently and erlend-aasland as code owners February 13, 2025 21:03
@bedevere-app bedevere-app Bot added the awaiting core review label Feb 13, 2025
kumaraditya303
kumaraditya303 reviewed Feb 15, 2025
View reviewed changes
Comment thread Python/_warnings.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

nascheme added 4 commits February 18, 2025 12:05
@nascheme
Create _py_warnings.py for Python implementation.
daa3d52
@nascheme
Add _warnings_context to _warnings module.
83419e4
@nascheme
Remove '_warnings_context' global string.
15443e8
@nascheme
Rename flag to 'context_aware_warnings'.
983e7ee
nascheme added 7 commits March 17, 2025 10:25
@nascheme
Avoid unused-variable warning in _warnings.c.
47d0b6f
@nascheme
Minor code improvement to _warnings.c.
b880dd1
@nascheme
Merge 'origin/main' into pythongh-128384-warnings-contextvar
5a1115b
@nascheme
Merge branch 'origin/main' into pythongh-128384-warnings-contextvar
ae701e3
@nascheme
Merge 'origin/main' into pythongh-128384-warnings-contextvar
153c6b1
@nascheme
Add extra unit tests.
9220223
Check behaviour of the catch_warnings() context manager when used with
asyncio co-routines and threads.
@nascheme
Use asyncio events to get deterministic execution.
c2c90cb
@nascheme nascheme added the topic-free-threading label Apr 1, 2025
kumaraditya303
kumaraditya303 reviewed Apr 3, 2025
View reviewed changes
Comment thread Doc/howto/free-threading-python.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

nascheme and others added 3 commits April 9, 2025 12:32
@nascheme
Merge 'origin/main' into pythongh-128384-warnings-contextvar
c1def22
@nascheme
Add unit test for asyncio tasks.
4e461d9
@nascheme @kumaraditya303
Update Doc/howto/free-threading-python.rst
543927b
Co-authored-by: Kumar Aditya <[email protected]>
@nascheme
nascheme force-pushed the gh-128384-warnings-contextvar branch from 51da563 to 543927b Compare April 9, 2025 19:50
nascheme added 2 commits April 9, 2025 13:53
@nascheme
Use asyncio.gather() rather than create_task().
4f11910
This seems a bit simpler since gather() will create tasks as needed.
@nascheme
Call resetwarnings() in a few places.
42d157b
This is needed because unit tests are now run with "-W error" on the
command line in CI (pythonGH-128770).  We want to start with an empty list of
filters to avoid supurious errors.
@nascheme
nascheme merged commit d687900 into python:main Apr 9, 2025
@bedevere-app bedevere-app Bot removed the awaiting core review label Apr 9, 2025
This was referenced Apr 9, 2025
gh-128555: Add 'context' keyword parameter to Thread. #128209
Closed
warnings.catch_warnings is async-unsafe #91505
Closed
@hugovk

hugovk commented Apr 16, 2025

Copy link
Copy Markdown
Member

@nascheme @kumaraditya303 Please could you check the buildbots? https://buildbot.python.org/#/release_status

AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1: breaks need immediate fix/revert) and ARM64 MacOS M1 Refleaks NoGIL 3.x (tier-2: breaks need fix/revert within 24h) have been failing for 6 days, and this PR seems a likely cause?

For example:

test_threaded_context (test.test_warnings.CThreadTests.test_threaded_context) ... Warning -- Uncaught thread exception: TypeErrorException in thread Thread-5 (run_a):
Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/threading.py", line 1079, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/threading.py", line 1021, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/test_warnings/__init__.py", line 1650, in run_a
    self.module.warn('run_a warning', UserWarning)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/_py_warnings.py", line 227, in _showwarnmsg
    raise TypeError("warnings.showwarning() must be set to a "
                    "function or method")TypeError: warnings.showwarning() must be set to a function or method

https://buildbot.python.org/#/builders/1368/builds/2970/steps/6/logs/stdio

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk added a commit to hugovk/cpython that referenced this pull request Apr 16, 2025
@hugovk
Revert "pythongh-128384: Use a context variable for warnings.catch_wa…
32104a3
…rnings (pythongh-130010)"

This reverts commit d687900.
@nascheme

nascheme commented Apr 16, 2025

Copy link
Copy Markdown
Member Author

I'll look into it. My guess is it is a flaky test and the test could just be disabled as a stop-gap.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hugovk

hugovk commented Apr 16, 2025

Copy link
Copy Markdown
Member

Thanks, here's a draft revert PR has passing buildbots: #132601.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nascheme

nascheme commented Apr 16, 2025

Copy link
Copy Markdown
Member Author

I believe this should fix the test failure without reverting the whole change: GH-132611. I'm waiting for CI tests to complete.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nascheme nascheme mentioned this pull request Apr 28, 2025
gh-128384: Use contextvar for catch_warnings() #128463
Closed
@lesteve

lesteve commented Sep 24, 2025

Copy link
Copy Markdown
Contributor

Perhaps _get_filters() should become a public function? I think it would be better if examining and manipulating that list was discouraged and so that's why I made the function non-public, at least for now.

As a single data point, in scikit-learn we needed to use warnings._get_filters to propagate the main process warnings filters to workers subprocesses (created by joblib).

We were relying on warnings.filters previously for this. We now use warning._get_filters to get the warnings filters in the main process (see code) and then warnings.resetfilters followed by warnings.simplefilter/warnings.filterwarnings in the subprocesses (see code).

ogrisel reacted with thumbs up emoji gpshead reacted with heart emoji
All reactions
  • 👍 1 reaction
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@braingram braingram mentioned this pull request Sep 26, 2025
logging.captureWarnings and warnings.catch_warnings do not play well together #139355
Open
@lesteve lesteve mentioned this pull request Jan 27, 2026
Race condition with warnings issued once directly and once inside pytest.warns Quansight-Labs/pytest-run-parallel#169
Open
@skirpichev skirpichev mentioned this pull request Mar 22, 2026
data races in decimal module with global context #141148
Open
@coderabbitai coderabbitai Bot mentioned this pull request Apr 20, 2026
Add - and = keys to refine auto levels and change the way saving backups work AlanRockefeller/faststack#84
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kumaraditya303 kumaraditya303 kumaraditya303 left review comments
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@1st1 1st1 Awaiting requested review from 1st1 1st1 is a code owner
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@corona10 corona10 Awaiting requested review from corona10 corona10 is a code owner
@gpshead gpshead Awaiting requested review from gpshead

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@nascheme @gpshead @kumaraditya303 @hugovk @lesteve
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.