Description

vstinner

opened on Jul 13, 2024

Issue body actions

Feature or enhancement

I propose adding a new PyBytesWriter API to create a Python bytes object.

API:

typedef struct PyBytesWriter PyBytesWriter;

// Create a bytes writer instance.
// On success, set *str and return a new writer.
// On error, set an exception and return NULLL.
PyAPI_FUNC(PyBytesWriter*) PyBytesWriter_Create(
    Py_ssize_t size,
    char **str);

// Return the final Python bytes object and destroy the writer instance.
// On success, return a bytes object.
// On error, set an exception and return NULL.
PyAPI_FUNC(PyObject *) PyBytesWriter_Finish(
    PyBytesWriter *writer,
    char *str);

// Disard a writer: deallocate its memory.
PyAPI_FUNC(void) PyBytesWriter_Discard(PyBytesWriter *writer);

// Allocate 'size' bytes to prepare writing into writer.
// On success, return 0.
// On error, set an exception and return -1.
PyAPI_FUNC(int) PyBytesWriter_Prepare(
    PyBytesWriter *writer,
    char **str,
    Py_ssize_t size);

// Write a bytes string into writer.
// On success, return 0.
// On error, set an exception and return -1.
PyAPI_FUNC(int) PyBytesWriter_WriteBytes(
    PyBytesWriter *writer,
    char **str,
    const void *bytes,
    Py_ssize_t size);

The PyBytesWriter writer is responsible to manage memor and uses overallocation to be more efficient, str is a cursor to write into the bytes string.

The implementation is based on the existing private _PyBytesWriter API which exists for many years and is used by many functions such as ASCII and Latin1 encoders, binascii module, _pickle module, _struct module, bytes methods, etc.

Linked PRs

• gh-121710: Add PyBytesWriter API #121726

Reacted by Tomas R. and Nadeshiko Manju

Activity

1. vstinner

   added

   type-featureA feature request or enhancementA feature request or enhancement

   topic-C-API

   on Jul 13, 2024
2. vstinner commented on Jul 13, 2024

   

   vstinner

   on Jul 13, 2024

   MemberAuthor

   More actions

   Example creating the bytes string b"abcdef" with PyBytesWriter_WriteBytes():

   static PyObject* create_string(void)
   {
       char *str;
       PyBytesWriter *writer = PyBytesWriter_Create(6, &str);
       if (writer == NULL) {
           return NULL;
       }
   
       if (PyBytesWriter_WriteBytes(writer, &str, "abc", 3) < 0) {
           goto error;
       }
       if (PyBytesWriter_WriteBytes(writer, &str, "def", 3) < 0) {
           goto error;
       }
   
       return PyBytesWriter_Finish(writer, str);
   
   error:
       PyBytesWriter_Discard(writer);
       return NULL;
   }

   Example creating the bytes string b"Hello." using PyBytesWriter_Prepare():

   static PyObject *
   test_byteswriter_prepare(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args))
   {
       char *str;
       PyBytesWriter *writer = PyBytesWriter_Create(0, &str);
       if (writer == NULL) {
           return NULL;
       }
   
       if (PyBytesWriter_Prepare(writer, &str, 6) < 0) {
           PyBytesWriter_Discard(writer);
           return NULL;
       }
   
       memcpy(str, "Hello", 5);
       str += 5;
   
       *str++ = '.';
   
       return PyBytesWriter_Finish(writer, str);
   }
3. vstinner

   added a commit that references this issue on Jul 13, 2024

   pythongh-121710: Add PyBytesWriter API

   3405f3a
4. vstinner commented on Jul 13, 2024

   

   vstinner

   on Jul 13, 2024

   MemberAuthor

   More actions

   The implementation is based on the existing private _PyBytesWriter API which exists for many years

   The proposed public API is different than the private API: it allows to still use the writer when an error occurs, whereas the private API requires the function to discard the writer instance. The public API is also simpler to be less error-prone.

   Reacted by Erlend E. Aasland
5. vstinner

   added a commit that references this issue on Jul 13, 2024

   pythongh-121710: Add PyBytesWriter API

   7862808
6. vstinner

   added a commit that references this issue on Jul 27, 2024

   pythongh-121710: Add PyBytesWriter API

   fe9667d
7. vstinner commented on Oct 7, 2024

   

   vstinner

   on Oct 7, 2024

   MemberAuthor

   More actions

   I decided to reject this API for now. It's too low-level and too error prone: capi-workgroup/decisions#39 (comment)
8. vstinner

   closed this as not plannedon Oct 7, 2024

Issue actions