Description

vstinner

opened on Oct 19, 2023

Issue body actions

I propose to change the PyUnicode_AsUTF8() API to raise an exception and return NULL if the string contains embedded null characters.

If the string contains an embedded null character, the UTF-8 encoded string can be truncated if used with C functions using char* since a null byte is treated as the terminator: marker of the string end. Truncating a string silently is a bad practice and can lead to different bugs including security vulnerabilities.

In practice, the minority of impacted C extensions and impacted users should benefit of such backward incompatible change, since truncating a string silently is a bad practice. Impacted users can use PyUnicode_AsUTF8AndSize(obj, NULL) and just ignore the size if they want to truncate on purpose.

It would address the following "hidden" comment on PyUnicode_AsUTF8():

Use of this API is DEPRECATED since no size information can be
extracted from the returned data.

PyUnicode_AsUTF8String() is part of the limited C API, whereas PyUnicode_AsUTF8() is not.

In the recently added PyUnicode_EqualToUTF8(obj, str), str is treated as not equal if obj contains embedded null characters.

The folllowing functions already raise an exception if the string contains embedded null characters or bytes:

• PyUnicode_AsWideCharString()
• PyUnicode_EncodeLocale()
• PyUnicode_EncodeFSDefault()
• PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
• PyUnicode_DecodeFSDefaultAndSize()
• PyUnicode_FSConverter()
• PyUnicode_FSDecoder()

PyUnicode_AsUTF8String() returns a bytes object and so the length, so it doesn't raise the exception.

PyUnicode_AsUTF8AndSize() also returns the size and so don't raise on embedded null characters.

Linked PRs

• gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL #111091
• gh-111089: PyUnicode_AsUTF8AndSize() sets size on error #111106
• gh-111089: Add PyUnicode_AsUTF8() to the limited C API #111121
• gh-111089: Use PyUnicode_AsUTF8() in sqlite3 #111122
• gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic #111585
• gh-111089: Add cache to PyUnicode_AsUTF8() for embedded NUL #111587
• gh-111089: Use PyUnicode_AsUTF8() in getargs.c #111620
• gh-111089: Add PyUnicode_AsUTF8Unsafe() function #111672
• gh-111089: Add PyUnicode_AsUTF8NoNUL() function #111688
• gh-111089: Revert PyUnicode_AsUTF8() changes #111833

Activity

1. vstinner

   added

   topic-C-API

   on Oct 19, 2023
2. vstinner

   added 2 commits that reference this issue on Oct 19, 2023

   pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL

   d62aab6

   pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL

   eba2f84
3. vstinner

   self-assigned this

   on Oct 19, 2023
4. vstinner

   added 9 commits that reference this issue on Oct 20, 2023

   pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error

   585158d

   pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL

   4e0b3d3

   pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error

   ae7734d

   gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (#111091)

   d731579

   pythongh-111089: Add PyUnicode_AsUTF8() to the limited C API

   e20557c

   pythongh-111089: Use PyUnicode_AsUTF8() in sqlite3

   714c652

   gh-111089: Add PyUnicode_AsUTF8() to the limited C API (#111121)

   d8f32be

   gh-111089: PyUnicode_AsUTF8AndSize() sets size on error (#111106)

   f1e751e

   gh-111089: Use PyUnicode_AsUTF8() in sqlite3 (#111122)

   37e4e20
5. vstinner commented on Oct 20, 2023

   

   vstinner

   on Oct 20, 2023

   MemberAuthor

   More actions

   I added PyUnicode_AsUTF8() to the limited C API and it now raises an exception if the string contains embedded null characters.
6. vstinner

   closed this as completedon Oct 20, 2023
7. 48 remaining items

8. encukou commented on Nov 8, 2023

   

   encukou

   on Nov 8, 2023

   Member

   More actions

   Thank you for the revert. Sorry that the issues were only discovered after merge, and after such heated discussion. Maybe there's some process improvement to be made.

   I filed this issue for the C-API-WG-to-be: capi-workgroup/api-evolution#39
   Feel free to re-frame the issue or propose other solutions.

   Reacted by Gregory P. Smith
9. aisk

   added 7 commits that reference this issue on Feb 11, 2024

   pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (pytho…

   6f3024f

   pythongh-111089: Add PyUnicode_AsUTF8() to the limited C API (python#…

   b87be3e

   pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error (python…

   0df9740

   pythongh-111089: Use PyUnicode_AsUTF8() in sqlite3 (python#111122)

   9e62c3a

   pythongh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (python#11…

   5c79eca

   pythongh-111089: Use PyUnicode_AsUTF8() in getargs.c (python#111620)

   6c7245d

   pythongh-111089: Revert PyUnicode_AsUTF8() changes (python#111833)

   fce855e
10. Glyphack

    added 7 commits that reference this issue on Sep 2, 2024

    pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (pytho…

    af7fb2a

    pythongh-111089: Add PyUnicode_AsUTF8() to the limited C API (python#…

    2599c32

    pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error (python…

    cb8a69a

    pythongh-111089: Use PyUnicode_AsUTF8() in sqlite3 (python#111122)

    ff94692

    pythongh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (python#11…

    ef2e01b

    pythongh-111089: Use PyUnicode_AsUTF8() in getargs.c (python#111620)

    02db58a

    pythongh-111089: Revert PyUnicode_AsUTF8() changes (python#111833)

    455f5c7

Issue actions