Description

ngoldbaum

opened on Sep 10, 2026

Issue body actions

Summary

The C API documents that the buffers returned by PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize() to contain an extra trailing NUL byte. RustPython instead returns pointers to the objects' exact-length Rust buffers. Reading the documented byte at data[len] is therefore a heap-buffer-overflow.

The relevant implementations return raw pointers without adding a terminator: PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize().

The bytearray case was raised during review of PR #8009, but dismissed as intentional. However, I'm not sure whether this being a soundness issue was raised at that time.

Reproducer

Save as capi_trailing_null.c:

#include <Python.h>
#include <stdio.h>

int main(void)
{
    PyObject *bytes, *bytearray, *unicode;
    char *bytes_data, *bytearray_data;
    const char *unicode_data;
    Py_ssize_t bytes_len, bytearray_len, unicode_len;
    unsigned char bytes_trailing, bytearray_trailing, unicode_trailing;

    Py_Initialize();

    bytes = PyBytes_FromStringAndSize("abc", 3);
    bytearray = PyByteArray_FromStringAndSize("abc", 3);
    unicode = PyUnicode_FromStringAndSize("abc", 3);
    if (!bytes || !bytearray || !unicode)
        return 2;

    bytes_data = PyBytes_AsString(bytes);
    bytearray_data = PyByteArray_AsString(bytearray);
    unicode_data = PyUnicode_AsUTF8AndSize(unicode, &unicode_len);
    bytes_len = PyBytes_Size(bytes);
    bytearray_len = PyByteArray_Size(bytearray);
    if (!bytes_data || !bytearray_data || !unicode_data ||
        bytes_len < 0 || bytearray_len < 0)
        return 2;

    bytes_trailing = (unsigned char)bytes_data[bytes_len];
    bytearray_trailing = (unsigned char)bytearray_data[bytearray_len];
    unicode_trailing = (unsigned char)unicode_data[unicode_len];
    printf("bytes: data[len] = 0x%02x\n", bytes_trailing);
    printf("bytearray: data[len] = 0x%02x\n", bytearray_trailing);
    printf("unicode UTF-8: data[len] = 0x%02x\n", unicode_trailing);

    Py_Finalize();
    return bytes_trailing || bytearray_trailing || unicode_trailing;
}

Expected

With CPython:

$ cc -fsanitize=address -fno-omit-frame-pointer $(python3-config --cflags) capi_trailing_null.c $(python3-config --embed --ldflags) -o trailing-null-cpython
$ ASAN_OPTIONS=detect_leaks=0 ./trailing-null-cpython
bytes: data[len] = 0x00
bytearray: data[len] = 0x00
unicode UTF-8: data[len] = 0x00

Actual

From a RustPython checkout on macOS (use .so instead of .dylib on Linux):

$ cargo build -p rustpython-capi
$ cc -fsanitize=address -fno-omit-frame-pointer $(python3-config --includes) capi_trailing_null.c target/debug/librustpython_capi.dylib -o trailing-null-rustpython
$ ASAN_OPTIONS=detect_leaks=0 ./trailing-null-rustpython
==...==ERROR: AddressSanitizer: heap-buffer-overflow
...
0x... is located 0 bytes after 3-byte region [0x...,0x...)

Python Documentation

CPython documents the trailing byte for PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize().

Reacted by Jeong, YunWon

Activity

1. ngoldbaum

   added

   C-bugSomething isn't workingSomething isn't working

   on Sep 10, 2026
2. ngoldbaum commented on Sep 10, 2026

   

   ngoldbaum

   on Sep 10, 2026

   Author

   More actions

   Also you might be interested in python/cpython#157236. Thinking about that PR is how I ran into this.
3. bschoenmaeckers commented on Sep 13, 2026

   

   bschoenmaeckers

   on Sep 13, 2026

   Contributor

   More actions

   Thanks @ngoldbaum, this is indeed a bug. I will pick this up in the near future.
4. bschoenmaeckers commented on Sep 15, 2026

   

   bschoenmaeckers

   on Sep 15, 2026

   Contributor

   More actions

   If I understand python/cpython#157236 correctly, this will probably be documented as an implementation detail of CPython. So we might not be required to allocate a extra null byte here.
5. ngoldbaum commented on Sep 15, 2026

   

   ngoldbaum

   on Sep 15, 2026

   Author

   More actions

   The C API functions returning the extra NULL byte isn't an implementation detail though. That's been documented for a long time.
6. vstinner commented on Sep 15, 2026

   

   vstinner

   on Sep 15, 2026

   More actions

   If I understand python/cpython#157236 correctly, this will probably be documented as an implementation detail of CPython. So we might not be required to allocate a extra null byte here.

   Well, Petr Viktorin has a strong opinion on what should qualify as an "implementation detail". His point is that it's possible to have a byte object without extra trailing null byte, but allocate a copy with a trailing null byte when PyBytes_AsString() is called.

   PyBytes_AsString() requires to have this trailing null byte:

   The last byte in the buffer is always null, regardless of whether there are any other null bytes.

   Well, for me it would be more efficient to create directly the bytes object with the trailing null copy. Having to create a copy for PyBytes_AsString() does double the memory usage and make memory management more complicated.

   CPython always allocates the extra null byte when it creates bytes and bytearray objects.

   PyByteArray_AsString() also says:

   The returned array always has an extra null byte appended.
7. bschoenmaeckers commented on Sep 15, 2026

   

   bschoenmaeckers

   on Sep 15, 2026

   Contributor

   More actions

   Thanks for the explanation! My understanding was wrong 😅

   Reacted by Victor Stinner
8. youknowone commented on Sep 16, 2026

   

   youknowone

   on Sep 16, 2026

   Member

   More actions

   Thank you so much everyone. We tried not to add trailing nul, but regarding c api, adding one will be more easy for everything. That will also help to resolve remaining ctypes imcompatibility.

   Reacted by Victor Stinner

Issue actions