GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-106672: C API: Report indiscriminately ignored errors by serhiy-storchaka · Pull Request #106674 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Jul 12, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Functions PyDict_GetItem(), PyDict_GetItemString(),
PyMapping_HasKey(), PyMapping_HasKeyString(),
PyObject_HasAttr(), PyObject_HasAttrString(), and
PySys_GetObject(), which clear all errors occurred during calling
the function, report now them using sys.unraisablehook().

  • Issue: C API: Report ignored exception in C API functions which ignore all errors #106672

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
pythongh-106672: C API: Report indiscriminately ignored errors
e134bad
Functions which indiscriminately ignore all errors now report them as
unraisable errors.
@serhiy-storchaka
serhiy-storchaka requested a review from vstinner July 12, 2023 09:20
@serhiy-storchaka
serhiy-storchaka requested review from markshannon and methane as code owners July 12, 2023 09:20
@bedevere-bot bedevere-bot added the awaiting core review label Jul 12, 2023
@bedevere-bot bedevere-bot mentioned this pull request Jul 12, 2023
C API: Report ignored exception in C API functions which ignore all errors #106672
Closed
@vstinner

vstinner commented Jul 12, 2023

Copy link
Copy Markdown
Member

Can you please list affected functions?

Is there a way to mute these warnings if an application is affected by this change but the maintainer is not available in the short term to fix these issues? These APIs were ignoring silently errors for years.

Maybe we need a command line option and/or environment variable to set an unraisable hook which... Does nothing.

In code, I suppose that it's easy to do:

import sys
def silence_errors(args):
    pass
sys.unraisablehook=silence_errors
olife-png reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jul 12, 2023

Copy link
Copy Markdown
Member

I made a similar change in Python 3.13 io module: close() errors are now logged.

olife-png reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Jul 12, 2023

Copy link
Copy Markdown
Member Author

I listed them in the NEWS entry. Will add them to the commit message too.

There are no new reports in the CPython tests, so at least CPython code is almost clear. They can still occur if you press Ctrl-C or run program in memory constricted environment, but it is very unlikely.

I think that the problem of silencing unraisable exception messages is a separate issue. It may be even not too important issue if such messages are rare. In addition to "silence error" I would consider adding option for "abort immediately".

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jul 12, 2023

Copy link
Copy Markdown
Member

Oh sure, you can abort the process on the first unraisable exception:

import signal, sys

def abort_hook(unraisable):
    signal.raise_signal(signal.SIGABRT)

sys.unraisablehook = abort_hook

A better implementation may want to log the exception before 😁 Maybe by calling the old hook.

At the beginning, it was proposed to always crash the process.

  • https://bugs.python.org/issue36829
  • https://vstinner.github.io/sys-unraisablehook-python38.html

Are you suggesting command line and env var to get his behavior?

pytest now catchs these unraisable exceptions.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jul 12, 2023

Copy link
Copy Markdown
Member

I listed them in the NEWS entry

Oh sorry I miss it, thanks.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jul 12, 2023

Copy link
Copy Markdown
Member

Would it be possible to suggest a fix in the warning? Like using an API which report errors?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Jul 12, 2023

Copy link
Copy Markdown
Member Author

Are you suggesting command line and env var to get his behavior?

It is a different issue, but I would be interested in such feature. Not that I need it now, but if I need it, it would be useful.

Would it be possible to suggest a fix in the warning? Like using an API which report errors?

On one hand, it would help the author of the extension. On other hand, it will overwhelm the user of the extension with not useful information. Should I keep the current more general description?

Exception ignored on testing a mapping key:
Exception ignored in PyMapping_HasKey(); use PyMapping_GetOptionalItem() or PyObject_GetItem() instead:
Exception ignored on testing a mapping key in PyMapping_HasKey(); use PyMapping_GetOptionalItem() or PyObject_GetItem() instead:

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jul 12, 2023

Copy link
Copy Markdown
Member

I would prefer to suggest using a different function to avoid such warning. About the phrasing, you may write "...; consider using xxx()".

I'm also fine with the shorter error message.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Jul 12, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/C API/2023-07-12-12-14-52.gh-issue-106672.fkRjmi.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Jul 12, 2023
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated note: we should consider making _PyErr_WriteUnraisableMsg() a public function. But i'm not sure about its API. Is the error message format easy to get?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka

serhiy-storchaka commented Jul 12, 2023

Copy link
Copy Markdown
Member Author

Unrelated note: we should consider making _PyErr_WriteUnraisableMsg() a public function.

I would prefer richer API, similar to PyErr_Format().

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jul 12, 2023

Copy link
Copy Markdown
Member

I would prefer richer API, similar to PyErr_Format().

That sounds like a good idea :-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ye-abc ye-abc mentioned this pull request Jul 12, 2023
Powerfull #106681
Closed
serhiy-storchaka added 2 commits July 12, 2023 18:42
@serhiy-storchaka
Update output messsages.
00ce038
@serhiy-storchaka
Add an entry in What's New.
168f92a
serhiy-storchaka
serhiy-storchaka commented Jul 12, 2023
View reviewed changes
Comment thread Objects/dictobject.c Outdated
{
return dict_getitem(op, key,
"in PyDict_GetItem(); consider using "
"PyDict_GetItemWithError()");

@serhiy-storchaka serhiy-storchaka Jul 12, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be new PyDict_GetItemRef().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Objects/dictobject.c Outdated
if (kv == NULL) {
PyErr_Clear();
_PyErr_WriteUnraisableMsg(
"in PyDict_GetItemString()", NULL);

@serhiy-storchaka serhiy-storchaka Jul 12, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No replacement currently. There will be new PyDict_GetItemRefString().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/sysmodule.c Outdated
/* XXX Suppress a new exception if it was raised and restore
* the old one. */
if (_PyErr_Occurred(tstate)) {
_PyErr_WriteUnraisableMsg("in PySys_GetObject()", NULL);

@serhiy-storchaka serhiy-storchaka Jul 12, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No replacement currently.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jul 12, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm waiting until PyDict_GetItemRef() is accepted to consider proposing a variant for PySys_GetObject(). I'm tracking functions returning borrowed references and replacement at: https://pythoncapi.readthedocs.io/bad_api.html#functions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
vstinner
vstinner reviewed Jul 12, 2023
View reviewed changes
Comment thread Python/sysmodule.c Outdated
/* XXX Suppress a new exception if it was raised and restore
* the old one. */
if (_PyErr_Occurred(tstate)) {
_PyErr_WriteUnraisableMsg("in PySys_GetObject()", NULL);

@vstinner vstinner Jul 12, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm waiting until PyDict_GetItemRef() is accepted to consider proposing a variant for PySys_GetObject(). I'm tracking functions returning borrowed references and replacement at: https://pythoncapi.readthedocs.io/bad_api.html#functions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Objects/abstract.c Outdated
if (rc == 0 && PyErr_Occurred()) {
_PyErr_WriteUnraisableMsg(
"in PyMapping_HasKeyString(); consider using "
"PyMapping_GetOptionalItemString() or PyMapping_GetItemString()",

@vstinner vstinner Jul 12, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This case is very subtle and the error message is not heplful. It's non-obvious to me that the exception was already set before the function was called.

Maybe the error message should be something like: "Ignore exception set before calling in PyMapping_HasKeyString()".

Instead of "Exception ignored in PyMapping_HasKeyString()".

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jul 12, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately _PyErr_WriteUnraisableMsg() does not support this. ☹️

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
Merge branch 'main' into capi-write-unraisable-for-ignored-errors
884da66
vstinner
vstinner approved these changes Jul 12, 2023
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. It may annoy people but there is a way to silence these warnings, so i'm fine with it.

Error messages can be enhanced, but the current API to log unraisable exception is too limited. It's ok the revisit them later.

I dislike ignoring errors, it can silence important errors. So this change is a nice step forward.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Jul 12, 2023
serhiy-storchaka added 3 commits August 7, 2023 20:08
@serhiy-storchaka
Merge branch 'main' into capi-write-unraisable-for-ignored-errors
58dbac5
@serhiy-storchaka
Fix smelly symbol.
07c8250
@serhiy-storchaka
Remove PyDict_GetItem(dict, invalid_key) from test_dict_capi().
56e3a49
serhiy-storchaka added 3 commits August 7, 2023 22:33
@serhiy-storchaka
Merge branch 'main' into capi-write-unraisable-for-ignored-errors
4e13132
@serhiy-storchaka
Fix regressions in handling NULLs.
b5a661b
@serhiy-storchaka
Merge branch 'main' into capi-write-unraisable-for-ignored-errors
ab5eb8f
@vstinner vstinner mentioned this pull request Aug 17, 2023
[C API] Enhance PyErr_WriteUnraisable() API to pass an error message #108082
Closed
@vstinner

vstinner commented Aug 26, 2023

Copy link
Copy Markdown
Member

Do you need help to solve conflicts? I approved your PR but you didn't merge it yet. Do you plan further changes?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Aug 26, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I could merge this PR long time ago, but I wanted to do some things before to make this PR better:

  • Add tests for PyObject_HasAttr(), PyMapping_HasKey(), etc. (Add the C API tests for Mapping Protocol and Sequence Protocol #107178)
  • Add tests for PySys_GetObject(). (Add C API tests for PySys_GetObject() and PySys_SetObject() #107735)
  • Enhance PyErr_WriteUnraisable() API. ([C API] Enhance PyErr_WriteUnraisable() API to pass an error message #108082)
  • Add replacements for PyObject_HasAttr(), PyMapping_HasKey(), etc. (C API: Add replacements for PyObject_HasAttr() etc #108511)
  • Add a replacement for PySys_GetObject(). (C API: Add a replacement for PySys_GetObject #108512)

Of course it can be merged before implementing all steps above, but then we will need to return to this code to improve it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka added 5 commits August 26, 2023 09:00
@serhiy-storchaka
Merge branch 'main' into capi-write-unraisable-for-ignored-errors
c681ce2
@serhiy-storchaka
Fix merging error.
82efd1c
@serhiy-storchaka
Merge branch 'main' into capi-write-unraisable-for-ignored-errors
c687ff8
@serhiy-storchaka
Add more suggestions.
310ae51
@serhiy-storchaka
Improve error messages for ignoring errors set before the calls.
241576a
@serhiy-storchaka

serhiy-storchaka commented Nov 5, 2023

Copy link
Copy Markdown
Member Author

Updated, improved warning messages.

Adding a replacement for PySys_GetObject() is more controversial, so it is better to merge this PR first.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Nov 6, 2023
View reviewed changes
Comment thread Objects/abstract.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/abstract.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/abstract.c Outdated
if (v) {
Py_DECREF(v);
return 1;
PyObject *dummy;

@vstinner vstinner Nov 6, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe rename to 'item' or 'value'.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Nov 6, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner

vstinner commented Nov 6, 2023

Copy link
Copy Markdown
Member

If these warnings become a performance issue later, maybe we can compute some fixed strings as static strings, like _Py_STR() API.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 6, 2023

Copy link
Copy Markdown
Member

In the What's New, would it be possible to mention recommended replacement function(s), maybe as a list (func: replace)?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka and others added 2 commits November 6, 2023 17:44
@serhiy-storchaka @vstinner
Apply suggestions from code review
fd50ea9
Co-authored-by: Victor Stinner <[email protected]>
@serhiy-storchaka
Address review comments.
587b3cc
@serhiy-storchaka

serhiy-storchaka commented Nov 6, 2023

Copy link
Copy Markdown
Member Author

In the What's New, would it be possible to mention recommended replacement function(s), maybe as a list (func: replace)?

They are already mentioned in the documentation for corresponding functions.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Nov 6, 2023
View reviewed changes
Comment thread Objects/abstract.c Outdated
PyErr_FormatUnraisable(
"Ignore exception set before calling in PyMapping_HasKeyString(); "
"consider using PyMapping_HasKeyStringWithError(), "
"PyMapping_GetOptionalItemString() or PyMapping_GetItemString()");

@vstinner vstinner Nov 6, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these 3 functions be called with an exception set? They don't override the exception? That sounds surprising. I would prefer suggesting to not call the function with an exception set. What do you think?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Nov 6, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got catch. Indeed, PyMapping_GetOptionalItemString() can only return 0 if no exception is set, so this condition is always false. Also, the alternative functions also can clear exceptions.

I think that we should classify the C API by classes:

  1. Function that can be called when an exception is set, and they do not change it.
  2. Function that can be called when an exception is set, and they do not change it unless they fail.
  3. Function that can be called when an exception is set, but they can change it even at success.
  4. Function that can be called when an exception is set, but the result is ambiguous in some cases (you cannot distinguish some successful results from failure).
  5. Function that should never be called when an exception is set.

There may be more classes.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Nov 7, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the general case, to write safe code handling a raised exception, I think the safest option is to keep the exception aside using PyErr_GetRaisedException(). Maybe today some functions are perfectly fine and never override the currently raised exception. But what if tomorrow their implementation changes, and they may start to clear the currently raised exception?

In Python, in an except: block, there is no "currently raised exception" in the C level, even if sys.exc_info() returns the exception. The difference between PyThreadState.exc_info and PyThreadState.current_exception is subtle.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Nov 7, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is why it should be clearly documented.

Obviously you can call PyErr_Occurred(), PyErr_GetRaisedException() or PyErr_WriteUnraisable() when an exception is set.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
Remove non-existing cases.
bfb206a
@serhiy-storchaka
serhiy-storchaka merged commit f55cb44 into python:main Nov 7, 2023
@bedevere-app bedevere-app Bot removed the awaiting merge label Nov 7, 2023
@serhiy-storchaka
serhiy-storchaka deleted the capi-write-unraisable-for-ignored-errors branch November 7, 2023 13:58
hugovk pushed a commit to hugovk/cpython that referenced this pull request Nov 8, 2023
@serhiy-storchaka @hugovk
pythongh-106672: C API: Report indiscriminately ignored errors (pytho…
c1001c5
…nGH-106674)

Functions which indiscriminately ignore all errors now report them as
unraisable errors.
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@serhiy-storchaka @aisk
pythongh-106672: C API: Report indiscriminately ignored errors (pytho…
643f0b6
…nGH-106674)

Functions which indiscriminately ignore all errors now report them as
unraisable errors.
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@serhiy-storchaka @Glyphack
pythongh-106672: C API: Report indiscriminately ignored errors (pytho…
42bf264
…nGH-106674)

Functions which indiscriminately ignore all errors now report them as
unraisable errors.
@sklam sklam mentioned this pull request Oct 23, 2024
Outdated PyObject_HasAttr documentation (Exception behavior) #125887
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@methane methane Awaiting requested review from methane methane is a code owner
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@serhiy-storchaka @vstinner @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.