Description

picnixz

opened on Oct 26, 2024

Issue body actions

Crash report

What happened?

./python -c "import codecs; codecs.xmlcharrefreplace_errors(UnicodeEncodeError('bad', '', 0, 1, 'reason'))"
python: ./Include/cpython/unicodeobject.h:339: PyUnicode_READ_CHAR: Assertion `index >= 0' failed.
Aborted (core dumped)

./python -c "import codecs; codecs.backslashreplace_errors(UnicodeDecodeError('utf-8', b'00000', 9, 2, 'reason'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
SystemError: Negative size passed to PyUnicode_New

./python -c "import codecs; codecs.replace_errors(UnicodeTranslateError('000', 1, -7, 'reason'))"
python: Python/codecs.c:743: PyCodec_ReplaceErrors: Assertion `PyUnicode_KIND(res) == PyUnicode_2BYTE_KIND' failed.
Aborted (core dumped)

See #123378 for the root cause. Since we are still wondering how to fix the getters and setters, I suggest we first fix the crash by adding the checks inside at the handler's level (for now). I'm not sure if the handler itself is handling corner cases correctly as well.

Linked PRs

• gh-126004: remove redundant safeguards for codecs handlers tests #127680
• gh-126004: fix positions handling in codecs.replace_errors #127674
• gh-126004: fix positions handling in codecs.xmlcharrefreplace_errors #127675
• gh-126004: fix positions handling in codecs.backslashreplace_errors #127676

Activity

1. picnixz

   added

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   topic-C-API

   on Oct 26, 2024
2. picnixz

   self-assigned this

   on Oct 26, 2024
3. picnixz

   added this to Codecs and encodings issueson Oct 26, 2024
4. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Oct 26, 2024
5. picnixz

   changed the title [-]Incorrect handling of `start` and `end` values in `codecs.xmlcharrefreplace_errors` handler[/-] [+]Incorrect handling of `start` and `end` values in `codecs` error handlers[/+] on Nov 29, 2024
6. picnixz commented on Nov 29, 2024

   

   picnixz

   on Nov 29, 2024

   MemberAuthor

   More actions

   FTR: I prepared three branches that fix (and cleanup) the error handlers.

   Those branches include #123380 and thus I'll wait for the latter to be merged since otherwise there will be a crash even before the actual "bad" code happens.

   The fixes are actually quite straightforward: there is actually no check on the consistency of start and end after they've been retrieved via *GetStart and *GetEnd, and this leads to OOB issues or other kind of issues. So, even if we fix the getters, we still need to handle the consistency of the bounds.

   cc @encukou
7. picnixz

   moved this to In Progress in Codecs and encodings issueson Nov 29, 2024
8. picnixz commented on Dec 6, 2024

   

   picnixz

   on Dec 6, 2024

   MemberAuthor

   More actions

   Since #123378 is only 3.14+ (not backported), the fixes for those handlers will also be 3.14+ only.
9. picnixz

   added

   3.14bugs and security fixesbugs and security fixes

   on Dec 6, 2024
10. picnixz

    added 4 commits that reference this issue on Jan 23, 2025

    gh-126004: Fix positions handling in `codecs.xmlcharrefreplace_errors` (

    70dcc84

    gh-126004: Fix positions handling in `codecs.replace_errors` (#127674)

    225296c

    gh-126004: Fix positions handling in `codecs.backslashreplace_errors` (…

    25a614a

    gh-126004: Remove redundant safeguards for codecs handlers tests (#12…

    cf0b2da
11. picnixz

    closed this as completedon Jan 23, 2025
12. github-project-automation

    moved this from In Progress to Done in Codecs and encodings issueson Jan 23, 2025

Issue actions