Description

vstinner

opened on Sep 29, 2026

Issue body actions

Currently, PyUnicode_FromKindAndData() and PyUnicodeWriter_WriteUCS4() create an invalid string if a character is not in the [U+0000; U+10ffff] range. Instead, I propose to raise an exception in this case.

Example creating an invalid string:

$ python3.15
>>> import _testcapi, sys
>>> invalid_ucs4 = (0x11_0000).to_bytes(4, byteorder=sys.byteorder)
>>> s = _testcapi.unicode_fromkindanddata(4, invalid_ucs4*5)

>>> list(s)
SystemError: invalid maximum character passed to PyUnicode_New

>>> s[0]
SystemError: invalid maximum character passed to PyUnicode_New

Linked PRs

• gh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData() #158447
• gh-158445: Detect invalid UCS4 characters in debug mode #158502
• gh-158445: Add C API tests on build information functions #158603
• gh-158445: Skip invalid UCS-4 tests if Python is built with assertions #158604
• gh-158445: Allocate memory in the heap in Py_GetVersion() #158608
• [3.15] gh-158445: Add C API tests on build information functions #159515
• Revert "gh-158445: Allocate memory in the heap in Py_GetVersion() (#158608)" #159519
• [3.15] gh-158445: Detect invalid UCS4 characters in debug mode (#158502) #159524
• [3.14] gh-158445: Add C API tests on build information functions (#159515) #159525
• [3.14] gh-158445: Detect invalid UCS4 characters in debug mode (#158502) (#159524) #159540

Activity

1. vstinner

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   topic-C-API

   on Sep 29, 2026
2. vstinner commented on Sep 29, 2026

   

   vstinner

   on Sep 29, 2026

   MemberAuthor

   More actions

   With 32-bit wchar_t (ex: Unix), PyUnicode_FromWideChar() and PyUnicodeWriter_WriteWideChar() already raise a ValueError if a character is outside the [U+0000; U+10ffff] range.
3. vstinner

   added 2 commits that reference this issue on Sep 29, 2026

   pythongh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData()

   ae17c69

   pythongh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData()

   da5cb33
4. vstinner commented on Sep 29, 2026

   

   vstinner

   on Sep 29, 2026

   MemberAuthor

   More actions

   Other Python and C functions raising an exception if the code pointer is negative or is greater than 0x10ffff:

   • Python chr() raises ValueError.
   • PyUnicode_FromOrdinal(), PyUnicode_Fill() and PyUnicodeWriter_WriteChar() raise ValueError.
   • PyUnicode_FromFormat("%c") raises OverflowError.
   • PyUnicode_New() raises SystemError.

   PyUnicode_WriteChar() raises ValueError if the character is greater than the string maximum character.
5. vstinner

   added 3 commits that reference this issue on Sep 30, 2026

   pythongh-158445: Detect invalid UCS4 characters in debug mode

   5286181

   pythongh-158445: Detect invalid UCS4 characters in debug mode

   4a8e756

   gh-158445: Detect invalid UCS4 characters in debug mode (#158502)

   a14a4a6
6. vstinner

   changed the title [-]C API: Reject invalid UCS-4 string in PyUnicode_FromKindAndData[/-] [+]C API: Detect invalid UCS4 characters in debug mode[/+] on Sep 30, 2026
7. vstinner commented on Sep 30, 2026

   

   vstinner

   on Sep 30, 2026

   MemberAuthor

   More actions

   Since checking for invalid characters at runtime was rejected, I repurpose the issue to "Detect invalid UCS4 characters in debug mode".
8. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Oct 1, 2026
9. encukou commented on Oct 2, 2026

   

   encukou

   on Oct 2, 2026

   Member

   More actions

   This broke several buildbots.
10. vstinner

    added a commit that references this issue on Oct 2, 2026

    gh-158445: Add C API tests on build information functions (#158603)

    a831abb
11. vstinner commented on Oct 2, 2026

    

    vstinner

    on Oct 2, 2026

    MemberAuthor

    More actions

    This broke several buildbots.

    Oh, I didn't get buildbot failures report on my PR, so I didn't notice. Thanks for the notification.

    So test_capi fails on 2 buildbots: Python is built in release mode with assertion using ./configure (...) CFLAGS=-UNDEBUG

    • ARM64 Raspbian 3.x: https://buildbot.python.org/#/builders/1678/builds/3888
    • x86 Debian Non-Debug with X 3.x: https://buildbot.python.org/#/builders/1245/builds/9676

    I wrote a first PR #158603 to fix support.built_with_c_assertions() (merged). Previously, -UNDEBUG compiler flag was not handled properly.

    I wrote a second PR #158604 to fix test_capi.test_unicode.
12. vstinner

    added a commit that references this issue on Oct 2, 2026

    gh-158445: Skip invalid UCS-4 tests if Python is built with assertions (

    35bdbca
13. vstinner commented on Oct 2, 2026

    

    vstinner

    on Oct 2, 2026

    MemberAuthor

    More actions

    My latest change should fix the 2 buildbots.
14. vstinner

    closed this as completedon Oct 2, 2026
15. encukou commented on Oct 2, 2026

    

    encukou

    on Oct 2, 2026

    Member

    More actions

    Thank you!

    Reacted by Victor Stinner
16. vstinner

    added a commit that references this issue on Oct 3, 2026

    gh-158445: Allocate memory in the heap in Py_GetVersion() (#158608)

    f5e913c
17. vstinner commented on Oct 3, 2026

    

    vstinner

    on Oct 3, 2026

    MemberAuthor

    More actions

    ARM64 Raspbian 3.x: https://buildbot.python.org/#/builders/1678/builds/3888
    x86 Debian Non-Debug with X 3.x: https://buildbot.python.org/#/builders/1245/builds/9676

    Both workers are back to green.
18. vstinner

    added 5 commits that reference this issue on Oct 16, 2026

    Revert "gh-158445: Allocate memory in the heap in Py_GetVersion() (#1…

    35a0009

    [3.15] gh-158445: Add C API tests on build information functions (#15…

    cb7a629

    [3.14] gh-158445: Add C API tests on build information functions (#15…

    1de946c

    [3.15] gh-158445: Detect invalid UCS4 characters in debug mode (#158502…

    2c23505

    [3.14] gh-158445: Detect invalid UCS4 characters in debug mode (#158502…

    09a7956

Issue actions