Description

vstinner

opened on Sep 4, 2026

Issue body actions

PyBytesWriter gives a directy access to its content with PyBytesWriter_GetData(). The caller is responsible to handle correctly the buffer size and write the correct amount of bytes. But programming is hard, and mistakes are easy to do. I propose adding a mechanism to detect buffer overflow to help developers detecting bugs in in their code.

Example of bug:

    PyBytesWriter *writer = PyBytesWriter_Create(0);
    if (writer == NULL) {
        goto error;
    }
    if (PyBytesWriter_Grow(writer, 3) < 0) {
        goto error;
    }
    char *data = PyBytesWriter_GetData(writer);
    strcpy(data, "abc");

    return PyBytesWriter_Finish(writer);

This code allocates 3 bytes but writes 4 bytes because strcat() also writes a trailing NUL byte.

Linked PRs

• gh-156939: Detect buffer overflow in PyBytesWriter in debug mode #156943
• gh-156939: Fix struct.pack('0p', bytes) #157071
• gh-156939: Fix memory corruption check in fcntl #157103
• gh-156939: Fix xmlcharrefreplace() buffer overflow #157109
• [3.15] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) #157129
• [3.14] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) #157130
• [3.13] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) #157131
• [3.15] gh-156939: Fix xmlcharrefreplace() buffer overflow (GH-157109) #157232
• [3.14] gh-156939: Fix xmlcharrefreplace() buffer overflow (GH-157109) #157233
• [3.13] gh-156939: Fix xmlcharrefreplace() buffer overflow (GH-157109) #157234
• gh-156939: Document that PyBytesObject ends with a NUL byte #157236
• gh-156939: Detect buffer overflow in bytearray #157382
• gh-156939: Update UTF-8 encoder for PyBytesWriter changes #157383
• gh-156939: Detect PyBytesWriter buffer overflow earlier #157385
• gh-156939: Clear newly allocated bytes in PyBytesWriter_Resize() #157455
• gh-156939: Detect buffer overflow in bytes and bytearray #157529
• [3.15] gh-156939: Document that PyBytesObject ends with a NUL byte (GH-157236) #157550
• [3.14] gh-156939: Document that PyBytesObject ends with a NUL byte (GH-157236) #157551
• [3.13] gh-156939: Document that PyBytesObject ends with a NUL byte (GH-157236) #157552
• gh-156939: Document that PyASCIIObject ends with null byte #157708
• gh-156939: Add tests on the PyUnicode C API #157798
• gh-156939: Fix Modules/_testcapi/bytes.c: remove double Py_DECREF #158400
• gh-156939: Fix two bugs in PyBytesWriter #158404
• [3.15] gh-156939: Fix two bugs in PyBytesWriter (#158404) #158409
• [3.14] gh-156939: Document that PyUnicodeObject ends with null character (GH-157708) #159530
• [3.15] gh-156939: Document that PyUnicodeObject ends with null character (GH-157708) #159531
• [3.15] gh-156939: Add tests on the PyUnicode C API #159533
• [3.14] gh-156939: Add tests on the PyUnicode C API (#159533) #159539

Activity

1. vstinner

   added

   topic-C-API

   on Sep 4, 2026
2. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   type-featureA feature request or enhancementA feature request or enhancement

   on Sep 4, 2026
3. vstinner

   added 5 commits that reference this issue on Sep 4, 2026

   pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode

   bf9beb8

   pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode

   62da3be

   pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode

   97c29ff

   pythongh-156939: Fix xmlcharrefreplace()

   017d597

   pythongh-156939: Fix xmlcharrefreplace() buffer overflow

   d1adc33
4. vstinner commented on Sep 7, 2026

   

   vstinner

   on Sep 7, 2026

   MemberAuthor

   More actions

   Oh, I found multiple functions which write an extra (NUL) byte (buffer overflow!) or read the byte after the end of the buffer (read uninitialized memory):

   • struct.pack('0p', bytes) writes a trailing NUL byte.
   • str.encode('latin1', 'xmlcharrefreplace') writes a trailing NUL byte.
   • fcntl.fcntl() and fcntl.ioctl() reads data[len]: read one byte after the end of the buffer (and expect it to be NUL).

   These functions rely on the fact that in C, bytes objects always end with a trailing NUL byte: assert(PyBytes_AS_STRING(obj)[PyBytes_GET_SIZE(obj)] == 0); is true (and doesn't read uninitialized memory, it's well defined). This bytes property is not documented in bytes C API documentation.
5. vstinner commented on Sep 7, 2026

   

   vstinner

   on Sep 7, 2026

   MemberAuthor

   More actions

   But programming is hard, and mistakes are easy to do.

   See issue gh-155702 in sqlite3 where a bytes object created by PyBytesWriter was modified, whereas bytes objects are immutable. In this case, it was worse since the bytes object was a singleton object.

   See also issue gh-155742 "Treat bytes objects as immutable in C code".
6. vstinner

   added 5 commits that reference this issue on Sep 7, 2026

   gh-156939: Fix struct.pack('0p', bytes) (#157071)

   23525c9

   [3.13] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) (#157131)

   1c5df82

   [3.14] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) (#157130)

   3a27a35

   pythongh-156939: Fix xmlcharrefreplace() buffer overflow

   3bdaf2c

   pythongh-156939: Fix xmlcharrefreplace() buffer overflow

   aada0ed
7. 36 remaining items
8. vstinner

   added 6 commits that reference this issue on Sep 17, 2026

   pythongh-156939: Document that PyASCIIObject ends with null byte

   dcc4075

   pythongh-156939: Document that PyASCIIObject ends with null byte

   ee34138

   pythongh-156939: Add tests on the PyUnicode C API

   aa13241

   gh-156939: Document that PyUnicodeObject ends with null character (#1…

   4d3b405

   pythongh-156939: Add tests on the PyUnicode C API

   e0d8fd5

   gh-156939: Add tests on the PyUnicode C API (#157798)

   e8f6448
9. vstinner

   added 3 commits that reference this issue on Sep 29, 2026

   pythongh-156939: Fix two bugs in PyBytesWriter

   ed1785f

   gh-156939: Fix Modules/_testcapi/bytes.c: remove double Py_DECREF (#1…

   90d4002

   gh-156939: Fix two bugs in PyBytesWriter (#158404)

   24ea85a
10. hugovk

    added a commit that references this issue on Sep 30, 2026

    [3.15] gh-156939: Fix two bugs in PyBytesWriter (#158404) (#158409)

    3740f74
11. serhiy-storchaka

    added a commit that references this issue on Oct 10, 2026

    [3.15] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) (GH-157129)

    61d94ed
12. vstinner

    added 4 commits that reference this issue on Oct 16, 2026

    [3.14] gh-156939: Document that PyUnicodeObject ends with null charac…

    823f781

    [3.15] gh-156939: Document that PyUnicodeObject ends with null charac…

    5a6369f

    [3.15] gh-156939: Add tests on the PyUnicode C API (#159533)

    bcbe9bc

    [3.14] gh-156939: Add tests on the PyUnicode C API (#159533) (#159539)

    21aae35

Issue actions