GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-123378: fix some corner cases of `start` and `end` values in `PyUnicodeErrorObject` by picnixz · Pull Request #123380 · python/cpython

Conversation

@picnixz

@picnixz picnixz commented Aug 27, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Incorrect handling of negative start values on PyUnicodeErrorObject #123378

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz added 3 commits August 27, 2024 09:11
@picnixz
Fix PyUnicode{Encode,Decode}Error_GetStart.
7a1574d
This fixes a bug when the `start` value of a `UnicodeError` is 0
and the underlying object is an empty string or bytes.
@picnixz
blurb
6ef0c6d
@picnixz
add tests
60ab0bb
@picnixz
picnixz requested a review from iritkatriel as a code owner August 27, 2024 07:51
@bedevere-app bedevere-app Bot mentioned this pull request Aug 27, 2024
Incorrect handling of negative start values on PyUnicodeErrorObject #123378
Closed
@bedevere-app bedevere-app Bot added the awaiting review label Aug 27, 2024
@picnixz
fix NEWS
67b3d8e
skirpichev
skirpichev previously approved these changes Aug 27, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting core review and removed awaiting review labels Aug 27, 2024
picnixz added 7 commits August 27, 2024 10:29
@picnixz
remove a duplicated normal case
78fff57
@picnixz
handle start < 0
a6e6f80
@picnixz
add C tests
20c47ba
@picnixz
add test coverage
51bc77e
@picnixz
update docs
b290e58
@picnixz
fixup
75398a9
@picnixz
update blurb
546be87
@picnixz picnixz changed the title gh-123378: Fix out-of-bounds results in PyUnicode{Encode,Decode}Error_GetStart gh-123378: reject negative start values on PyUnicodeError objects Aug 28, 2024
@picnixz picnixz changed the title gh-123378: reject negative start values on PyUnicodeError objects gh-123378: reject negative start values on PyUnicodeErrorObject Aug 28, 2024
vstinner
vstinner reviewed Aug 28, 2024
View reviewed changes
Comment thread Include/cpython/pyerrors.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/exceptions.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_exceptions.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_exceptions.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 2 commits August 29, 2024 10:13
@picnixz
address Victor's review
cded571
@picnixz
refactor name
1900d9a
@picnixz
picnixz requested a review from vstinner August 29, 2024 08:15
@picnixz

picnixz commented Aug 29, 2024

Copy link
Copy Markdown
Member Author

Thank you Victor for your review!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 2 commits August 29, 2024 10:39
@picnixz
fix refcounts
8acc563
@picnixz
remove debugging code
0538c83
picnixz
picnixz commented Aug 29, 2024
View reviewed changes
Comment thread Objects/exceptions.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Aug 29, 2024
View reviewed changes
Comment thread Lib/test/test_capi/test_exceptions.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/exceptions.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/exceptions.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
address Victor's review (round 2)
d5ea357
vstinner
vstinner approved these changes Aug 29, 2024
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@serhiy-storchaka: Do you want to double check the fix?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz added 4 commits October 27, 2024 09:19
@picnixz
Merge remote-tracking branch 'upstream/main' into fix/c-api-unicode-e…
6caf5b6
…rror-get-start-123378
@picnixz
post-merge cleanup
fcde448
@picnixz
Merge branch 'main' into fix/capi/unicode-error-start-end-123378
2e66302
@picnixz
Merge branch 'main' into fix/capi/unicode-error-start-end-123378
dc6b37b
@picnixz picnixz mentioned this pull request Nov 29, 2024
Incorrect handling of start and end values in codecs error handlers #126004
Closed
encukou
encukou reviewed Dec 2, 2024
View reviewed changes

@encukou encukou left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit sad that we can't agree to fix the setters, but, if we're fixing the C getters, this is the way to go.
A few nitpicks:

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Objects/exceptions.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/exceptions.c
static inline int
unicode_error_set_end_impl(PyObject *exc, Py_ssize_t end)
{
((PyUnicodeErrorObject *)exc)->end = end;

@encukou encukou Dec 2, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While you're here, could you add a type check before each cast?

    assert(PyObject_TypeCheck(exc, (PyTypeObject*)&PyExc_UnicodeError));

Ideally, public API would raise TypeError on wrong types, but that change might be too big.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@picnixz picnixz Dec 2, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make a follow-up PR for adding assertions on the getters as well?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Dec 3, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be great! Thank you for volunteering :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Misc/NEWS.d/next/C_API/2024-08-27-09-07-56.gh-issue-123378.JJ6n_u.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 5 commits December 2, 2024 16:05
@picnixz
fix typo
baa5cb2
@picnixz
update NEWS and docs
4c4808e
@picnixz
add some assertion checks
efbdff1
@picnixz
add some assertion checks
180f3c2
@picnixz
Merge branch 'main' into fix/c-api-unicode-error-get-start
4bd6e20
encukou
encukou previously approved these changes Dec 3, 2024
View reviewed changes
Comment thread Objects/exceptions.c
static inline int
unicode_error_set_end_impl(PyObject *exc, Py_ssize_t end)
{
((PyUnicodeErrorObject *)exc)->end = end;

@encukou encukou Dec 3, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be great! Thank you for volunteering :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Dec 3, 2024
@encukou
encukou dismissed their stale review December 3, 2024 12:36

Oops, didn't look at the CI :/

@encukou

encukou commented Dec 3, 2024

Copy link
Copy Markdown
Member

Feel free to leave all the asserts to a follow-up PR.

picnixz reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 2 commits December 3, 2024 13:42
@picnixz
remove failing assertions for now
5759a70
@picnixz
fix docs
8c12171
@picnixz
picnixz requested a review from encukou December 3, 2024 13:29
@encukou
encukou merged commit bc0f2e9 into python:main Dec 4, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Dec 4, 2024
@encukou

encukou commented Dec 4, 2024

Copy link
Copy Markdown
Member

Thank you!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
picnixz deleted the fix/c-api-unicode-error-get-start branch December 5, 2024 12:56
@picnixz

picnixz commented Dec 5, 2024

Copy link
Copy Markdown
Member Author

Question: should we backport this to 3.12 and 3.13? it's both a feature and a bugfix in some sense but I feel it's more a bug fix even though it could change the behaviour of some code (hopefully, not too much...). The start getter was buggy (it could return -1) but the end getter was not.

WDYT? By the way, I spotted some typos here and there. I'll correct them as part of a follow-up PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz picnixz mentioned this pull request Dec 6, 2024
gh-126004: fix positions handling in codecs.xmlcharrefreplace_errors #127672
Closed
@encukou

encukou commented Dec 6, 2024

Copy link
Copy Markdown
Member

If I understand correctly, it could only return -1 with an empty string, which isn't something one would typically get encoding errors for. Also, at this level you get a wrong value rather than a crash.
I don't think it's worth backporting.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Dec 6, 2024

Copy link
Copy Markdown
Member Author

it could only return -1 with an empty string

Yup. (to be clear, it's the size that is being stored in start, not the function return value which returns -1 on error and 0 on success).

In this case, subsequent calls usually lead to crashes because the C API assumes non-relative offsets in general.

I don't think it's worth backporting.

Thanks, it's essentially to know whether the patches for codecs handlers will be backported or not (without the backport of this PR, the handlers would still be broken I think).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz picnixz mentioned this pull request Dec 6, 2024
Check for type consistency for PyUnicodeError API #127691
Closed
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this pull request Jan 8, 2025
@picnixz @skirpichev @srinivasreddy
pythongh-123378: Ensure results of PyUnicode*Error_Get{Start,End} a…
92e9165
…re clamped (pythonGH-123380)


Co-authored-by: Sergey B Kirpichev <[email protected]>
@skirpichev
skirpichev removed their request for review January 28, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@skirpichev skirpichev skirpichev left review comments
@iritkatriel iritkatriel Awaiting requested review from iritkatriel iritkatriel is a code owner
@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka
@vstinner vstinner Awaiting requested review from vstinner
@encukou encukou Awaiting requested review from encukou

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@picnixz @encukou @vstinner @skirpichev @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.