GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-126004: fix positions handling in `codecs.xmlcharrefreplace_errors` by picnixz · Pull Request #127675 · python/cpython

Conversation

@picnixz

@picnixz picnixz commented Dec 6, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Incorrect handling of start and end values in codecs error handlers #126004

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz added 2 commits December 6, 2024 11:22
@picnixz
fix codecs.xmlcharrefreplace_errors handler
8c4d75f
@picnixz
blurb
4bee635
@bedevere-app bedevere-app Bot mentioned this pull request Dec 6, 2024
Incorrect handling of start and end values in codecs error handlers #126004
Closed
@bedevere-app bedevere-app Bot added the awaiting review label Dec 6, 2024
@picnixz
cosmetic changes
3820eca
encukou
encukou reviewed Dec 6, 2024
View reviewed changes
Comment thread Python/codecs.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/codecs.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 3 commits December 6, 2024 17:12
@picnixz
address Petr's review
e178ddf
@picnixz
Merge remote-tracking branch 'upstream/main' into fix/codecs/xmlcharr…
442a938
…efreplace-errors-126004
@picnixz
use internal _PyUnicodeError_GetParams helper
e3cec4f
@picnixz
picnixz requested a review from encukou January 3, 2025 13:42
picnixz added 5 commits January 14, 2025 11:19
@picnixz
Merge branch 'main' into fix/codecs/xmlcharrefreplace-errors-126004
5261772
@picnixz
Merge branch 'main' into fix/codecs/xmlcharrefreplace-errors-126004
e12f17f
@picnixz
update usages of _PyUnicodeError_GetParams
b1ba109
@picnixz
amend some cosmetic changes to be consistent
debf20f
@picnixz
fix bounds
2ac9464
vstinner
vstinner reviewed Jan 22, 2025
View reviewed changes
Comment thread Python/codecs.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/codecs.c
/* object is guaranteed to be "ready" */
Py_UCS4 ch = PyUnicode_READ_CHAR(obj, i);
if (ch < 10) {
ressize += 2 + 1 + 1;

@vstinner vstinner Jan 22, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative to end = start + PY_SSIZE_T_MAX / (2 + 7 + 1); would be to set digits = 1; and after the serie of ifs, check that ressize += (2 + digits + 1); doesn't overflow. If it does overflow, return MemoryError. Something like:

if (resize > PY_SSIZE_T_MAX - (2 + digits + 1)) {
    return PyErr_MemoryError();
}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jan 22, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's what PyCodec_NameReplaceErrors does but I don't know how performances would be affected. Hitting PY_SSIZE_T_MAX / (2 + 7 + 1) means that we're handling something that is quite large. So doing the check on resize at each loop iteration might slow down the handler a bit.

Now, it took me a while to convince myself that it won't be slowing down the handlers by much. Namely, I don't think it would dramatically slow it down because if our characters are > 10^3, we would do < 10, < 100, < 1000 and < 10000 checks (this last one is needed to know that it's > 10^3 but not > 10^4) already. So instead of 4 we have 5 checks which is not that annoying.

Why can we assume that we will have at least 2 checks and not less? Well... everything < 100 is actually ASCII, so and unless someone is using a special codec for which those characters are not supported or for artificially created exceptions that indicate their start/end positions incorrectly, we're likely to have at least 2 checks in the loop (namely < 10 and < 100) because the bad characters are likely do be something outside the ASCII range.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jan 22, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would look like:

    Py_ssize_t ressize = 0;
    for (Py_ssize_t i = start; i < end; ++i) {
        // The number of characters that each character 'ch' contributes
        // in the result is 2 + k + 1, where k = min{t >= 1 | 10^t > ch}
        // and will be formatted as "&#" + DIGITS + ";". Since the Unicode
        // range is below 10^7, each "block" requires at most 2 + 7 + 1
        // characters.
        int replsize;
        /* object is guaranteed to be "ready" */
        Py_UCS4 ch = PyUnicode_READ_CHAR(obj, i);
        if (ch < 10) {
            replsize = 2 + 1 + 1;
        }
        else if (ch < 100) {
            replsize = 2 + 2 + 1;
        }
        else if (ch < 1000) {
            replsize = 2 + 3 + 1;
        }
        else if (ch < 10000) {
            replsize = 2 + 4 + 1;
        }
        else if (ch < 100000) {
            replsize = 2 + 5 + 1;
        }
        else if (ch < 1000000) {
            replsize = 2 + 6 + 1;
        }
        else {
            assert(ch < 10000000);
            replsize = 2 + 7 + 1;
        }
        if (ressize > PY_SSIZE_T_MAX - replsize) {
            Py_DECREF(obj);
            return PyErr_NoMemory();
        }
        ressize += replsize;

But I'm not very fond of this. I think it's still nicer to have the check outside the loop.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jan 22, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we decide in a follow-up PR whether those special checks (for instance, we have similar checks for PyCodec_NameReplaceErrors and PyCodec_BackslashReplaceErrors) need to be kept. For the 'namereplace' handler, we purely break actually:

        for (i = start, ressize = 0; i < end; ++i) {
            /* object is guaranteed to be "ready" */
            c = PyUnicode_READ_CHAR(object, i);
            if (ucnhash_capi->getname(c, buffer, sizeof(buffer), 1)) {
                replsize = 1+1+1+(int)strlen(buffer)+1;
            }
            else if (c >= 0x10000) {
                replsize = 1+1+8;
            }
            else if (c >= 0x100) {
                replsize = 1+1+4;
            }
            else
                replsize = 1+1+2;
            if (ressize > PY_SSIZE_T_MAX - replsize)
                break;
            ressize += replsize;
        }

and just don't care anymore :') (the reason is that cannot determine in advance how much it would take unless we call getname before...)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@picnixz
add assertion per Victor's suggestion
5046358
vstinner
vstinner approved these changes Jan 22, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I'm fine with keeping the code as it is, or changing how PY_SSIZE_T_MAX limit is handled in a separated PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Jan 22, 2025
@vstinner

vstinner commented Jan 22, 2025

Copy link
Copy Markdown
Member

Strange, the Lint job reports an unrelated issue:

diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py
index a053db8..341e3e9 100644
--- a/Lib/test/test_asyncio/test_subprocess.py
+++ b/Lib/test/test_asyncio/test_subprocess.py
@@ -905,7 +905,7 @@ def setUp(self):
             # Force the use of the threaded child watcher
             unix_events.can_use_pidfd = mock.Mock(return_value=False)
             super().setUp()
-        
+
         def tearDown(self):
             unix_events.can_use_pidfd = self._original_can_use_pidfd
             return super().tearDown()
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Jan 22, 2025

Copy link
Copy Markdown
Member Author

Strange, the Lint job reports an unrelated issue:

It might be because I merged main and that main is failing?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Jan 22, 2025

Copy link
Copy Markdown
Member Author

Yes: https://github.com/python/cpython/pull/129160/files#r1925168965

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

This comment was marked as off-topic.

Sign in to view
@picnixz
Merge branch 'main' into fix/codecs/xmlcharrefreplace-errors-126004
9df878e
@picnixz

picnixz commented Jan 22, 2025

Copy link
Copy Markdown
Member Author

I'll merge this one and the other PR you approved tomorrow (I've reverted the bad commit).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
picnixz merged commit 70dcc84 into python:main Jan 23, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label Jan 23, 2025
@picnixz
picnixz deleted the fix/codecs/xmlcharrefreplace-errors-126004 branch January 23, 2025 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@encukou encukou Awaiting requested review from encukou

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@picnixz @vstinner @encukou
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.