GeezerVison is showing github.com — [C API] PEP 782: Add PyBytesWriter API · Issue #129813 · python/cpython
[C API] PEP 782: Add PyBytesWriter API #129813DescriptionFeature or enhancement
Proposal:
I propose adding a PyBytesWriter API to create bytes objects.
API:
typedef struct PyBytesWriter PyBytesWriter;
PyAPI_FUNC(void*) PyBytesWriter_Create(
PyBytesWriter **writer,
Py_ssize_t alloc);
PyAPI_FUNC(void) PyBytesWriter_Discard(
PyBytesWriter *writer);
PyAPI_FUNC(PyObject*) PyBytesWriter_Finish(
PyBytesWriter *writer,
void *buf);
PyAPI_FUNC(Py_ssize_t) PyBytesWriter_GetRemaining(
PyBytesWriter *writer,
void *buf);
PyAPI_FUNC(void*) PyBytesWriter_Extend(
PyBytesWriter *writer,
void *buf,
Py_ssize_t extend);
PyAPI_FUNC(void*) PyBytesWriter_WriteBytes(
PyBytesWriter *writer,
void *buf,
const void *bytes,
Py_ssize_t size);
PyAPI_FUNC(void*) PyBytesWriter_Format(
PyBytesWriter *writer,
void *buf,
const char *format,
...);Simple example creating the string b"abc":
PyObject* create_abc(void)
{
PyBytesWriter *writer;
char *str = PyBytesWriter_Create(&writer, 3);
if (writer == NULL) return NULL;
memcpy(str, "abc", 3);
str += 3;
return PyBytesWriter_Finish(writer, str);
}Example formatting an integer in decimal, the size is not known in advance::
PyObject* format_int(int value)
{
PyBytesWriter *writer;
char *str = PyBytesWriter_Create(&writer, 20);
if (writer == NULL) return NULL;
str += PyOS_snprintf(str, 20, "%i", value);
return PyBytesWriter_Finish(writer, str);
}Note: using PyBytesWriter_Format() would make this code simpler.
Example using PyBytesWriter_Extend(),smilar to bytes.center() with a different API: spaces are number of whitespaces added to the left and to the right:
static PyObject *
byteswriter_center_example(Py_ssize_t spaces, char *str, Py_ssize_t str_size)
{
PyBytesWriter *writer;
char *buf = PyBytesWriter_Create(&writer, spaces * 2);
if (buf == NULL) {
goto error;
}
assert(PyBytesWriter_GetRemaining(writer, buf) == spaces * 2);
// Add left spaces
memset(buf, ' ', spaces);
buf += spaces;
assert(PyBytesWriter_GetRemaining(writer, buf) == spaces);
// Copy string
buf = PyBytesWriter_Extend(writer, buf, str_size);
if (buf == NULL) {
goto error;
}
assert(PyBytesWriter_GetRemaining(writer, buf) == spaces + str_size);
memcpy(buf, str, str_size);
buf += str_size;
assert(PyBytesWriter_GetRemaining(writer, buf) == spaces);
// Add right spaces
memset(buf, ' ', spaces);
buf += spaces;
assert(PyBytesWriter_GetRemaining(writer, buf) == 0);
return PyBytesWriter_Finish(writer, buf);
error:
PyBytesWriter_Discard(writer);
return NULL;
}Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
My previous attempt in July/August 2024:
Linked PRs
- [WIP] gh-129813: Add PyBytesWriter C API #129814
- [WIP] gh-129813: Add PyBytesWriter C API (version 2) #131520
- [WIP] gh-129813, PEP 782: Add PyBytesWriter C API #131681
- gh-129813, PEP 782: Add PyBytesWriter C API #138822
- gh-129813, PEP 782: Add PyBytesWriter_Format() #138824
- gh-129813, PEP 782: Use PyBytesWriter in binascii #138825
- gh-129813, PEP 782: Use PyBytesWriter in posix extension #138829
- gh-129813, PEP 782: Use Py_GetConstant(Py_CONSTANT_EMPTY_BYTES) #138830
- gh-129813, PEP 782: Use PyBytesWriter in socket and mmap #138831
- gh-129813, PEP 782: Use PyBytesWriter in lzma and zlib #138832
- gh-129813, PEP 782: Use PyBytesWriter in pickle and struct #138833
- gh-129813, PEP 782: Use PyBytesWriter in _hashopenssl #138835
- gh-129813, PEP 782: Use PyBytesWriter in memoryview #138836
- gh-129813, PEP 782: Use PyBytesWriter in _PyBytes_FromList() #138837
- gh-129813, PEP 782: Use PyBytesWriter in _PyBytes_DecodeEscape2() #138838
- gh-129813, PEP 782: Use PyBytesWriter in _PyBytes_FormatEx() #138839
- gh-129813, PEP 782: Use PyBytesWriter in bytes_concat() #138840
- gh-129813, PEP 782: Use PyBytesWriter in utf8_encoder() #138874
- gh-129813, PEP 782: Use PyBytesWriter in _io.FileIO.readall #138901
- gh-129813, PEP 782: Use PyBytesWriter in _codecs.escape_decode() #138919
- gh-129813, PEP 782: Use PyBytesWriter in _curses #138920
- gh-129813, PEP 782: Use PyBytesWriter in fcntl #138921
- gh-129813, PEP 782: Use PyBytesWriter in _hashopenssl #138922
- gh-129813, PEP 782: Use PyBytesWriter in _sha3 #138923
- gh-129813, PEP 782: Init small_buffer in PyBytesWriter_Create() #138924
- gh-129813, PEP 782: Use PyBytesWriter in _ssl #138929
- gh-129813, PEP 782: Use PyBytesWriter in _winapi.PeekNamedPipe() #138930
- gh-129813, PEP 782: Add PyBytesWriter.overallocate #138941
- gh-129813, PEP 782: Use PyBytesWriter in bufferedio.c #138954
- gh-129813, PEP 782: Use PyBytesWriter in FileIO.read() #138955
- gh-129813: Use
PyBytesWriter in _json:_match_number_unicode #138957
- gh-129813, PEP 782: Soft deprecate _PyBytes_Resize() #138964
- gh-129813, PEP 782: Add doc reference link #138986
- gh-129813, PEP 782: Use PyBytesWriter in _Py_bytes_maketrans() #139044
- gh-129813, PEP 782: Use PyBytesWriter in _multiprocessing #139047
- gh-129813, PEP 782: Use PyBytesWriter in _testclinic #139048
- gh-129813, PEP 782: Set invalid bytes in PyBytesWriter #139054
- gh-129813, PEP 782: Use PyBytesWriter in _socket #139097
- gh-129813, PEP 782: Optimize byteswriter_resize() #139101
- gh-129813, PEP 782: Use PyBytesWriter in ssl.MemoryBIO #139113
- gh-129813: PEP 782: Use PyBytesWriter in _sqlite #138956
- gh-129813, PEP 782: Use PyBytesWriter in bufferedio.c #139121
- gh-129813, PEP 782: Use PyBytesWriter in socket recvmsg() #139131
- gh-129813, PEP 782: Fix refleak in sock_recvfrom() #139151
- gh-129813, PEP 782: Remove the private _PyBytesWriter API #139264
- gh-129813: Fix PyBytesWriter tests #139892
- gh-129813: Enhance PyBytesWriter documentation #140152
- gh-129813: Document that PyBytesWriter_GetData() cannot fail #145900
- gh-129813: Fix typo in byteswriter_resize() #156938
- [3.15] gh-129813: Fix typo in byteswriter_resize() (GH-156938) #156979
- gh-129813: Check size in PyBytesWriter_FinishWithSize() #157226
- [3.15] gh-129813: Check size in PyBytesWriter_FinishWithSize() (#157226) #159261
Reacted by Andrew Svetlov, Cody Maloney and Danny McClanahan Activityvstinner commented on Feb 7, 2025
My previous attempt in July/August 2024: #121726
The new API is similar to the old one, but I renamed functions, changed arguments order, added PyBytesWriter_Format() and PyBytesWriter_GetAllocated() functions, etc. encukou commented on Feb 7, 2025 Thank you! This looks great :)
I have a few suggestions for your consideration, purely from the API-surface point of view:
PyAPI_FUNC(void*) PyBytesWriter_WriteBytes(
PyBytesWriter *writer,
void *buf,
const char *bytes,
Py_ssize_t size);
Consider void *bytes (or uint8_t *bytes), per the WIP guidelines.
PyAPI_FUNC(void*) PyBytesWriter_Create(
PyBytesWriter **writer,
Py_ssize_t alloc);
Consider returning the writer, and having void **buf as an output argument, for consistency with other *_Create/*_Finish/*_Discard families.
PyAPI_FUNC(void*) PyBytesWriter_WriteBytes(
PyBytesWriter *writer,
void *buf,
const char *bytes,
Py_ssize_t size);
This is meant to be called as buf = PyBytesWriter_WriteBytes(..., buf, ...), right?
Could it take a void **buf and update it?
(Same for PyBytesWriter_Format, PyBytesWriter_Extend)
PyAPI_FUNC(Py_ssize_t) PyBytesWriter_GetAllocated(
PyBytesWriter *writer);
It seems something like PyBytesWriter_HowMuchMoreCanIWrite(PyBytesWriter *writer, void *buf) would be more useful -- i.e. get the space that's available after buf, rather than the total.
PyAPI_FUNC(void*) PyBytesWriter_Extend(
PyBytesWriter *writer,
void *buf,
Py_ssize_t extend);
It might be more useful to add PyBytesWriter_Reserve, with the same signature but ensuring extend bytes after buf are writable.
(If you use the PyBytesWriter_Create & str += PyOS_snprintf pattern you show, you don't really care about any previous overallocation.) vstinner commented on Feb 11, 2025 For PyBytesWriter_Extend(), you usually don't want to forget previous PyBytesWriter_Create() and PyBytesWriter_Extend() calls.
// allocate 100 bytes
buf = PyBytesWriter_Create(&writer, 100);
(... write a few bytes ...)
// make room to write 5 bytes at the current position
// allocate 100+5 bytes (+ overallocation)
buf = PyBytesWriter_Extend(writer, buf, 5);
(... write more bytes ...)
For example, PyBytes_FromFormatV(format) uses such pattern: it allocates strlen(format), and then call PyBytesWriter_Extend() if it's not enough. Copying each format byte is safe since you have at least strlen(format) bytes, but some formats need extra space and here comes PyBytesWriter_Extend().
Another full example from _pickle.c:
static PyObject *
raw_unicode_escape(PyObject *obj)
{
Py_ssize_t size = PyUnicode_GET_LENGTH(obj);
const void *data = PyUnicode_DATA(obj);
int kind = PyUnicode_KIND(obj);
PyBytesWriter *writer;
char *p = PyBytesWriter_Create(&writer, size);
if (p == NULL) {
return NULL;
}
for (Py_ssize_t i=0; i < size; i++) {
Py_UCS4 ch = PyUnicode_READ(kind, data, i);
/* Map 32-bit characters to '\Uxxxxxxxx' */
if (ch >= 0x10000) {
/* -1: subtract 1 preallocated byte */
p = PyBytesWriter_Extend(writer, p, 10-1);
if (p == NULL)
goto error;
*p++ = '\\';
*p++ = 'U';
*p++ = Py_hexdigits[(ch >> 28) & 0xf];
*p++ = Py_hexdigits[(ch >> 24) & 0xf];
*p++ = Py_hexdigits[(ch >> 20) & 0xf];
*p++ = Py_hexdigits[(ch >> 16) & 0xf];
*p++ = Py_hexdigits[(ch >> 12) & 0xf];
*p++ = Py_hexdigits[(ch >> 8) & 0xf];
*p++ = Py_hexdigits[(ch >> 4) & 0xf];
*p++ = Py_hexdigits[ch & 15];
}
/* Map 16-bit characters, '\\' and '\n' to '\uxxxx' */
else if (ch >= 256 ||
ch == '\\' || ch == 0 || ch == '\n' || ch == '\r' ||
ch == 0x1a)
{
/* -1: subtract 1 preallocated byte */
p = PyBytesWriter_Extend(writer, p, 6-1);
if (p == NULL)
goto error;
*p++ = '\\';
*p++ = 'u';
*p++ = Py_hexdigits[(ch >> 12) & 0xf];
*p++ = Py_hexdigits[(ch >> 8) & 0xf];
*p++ = Py_hexdigits[(ch >> 4) & 0xf];
*p++ = Py_hexdigits[ch & 15];
}
/* Copy everything else as-is */
else
*p++ = (char) ch;
}
return PyBytesWriter_Finish(writer, p);
error:
PyBytesWriter_Discard(writer);
return NULL;
}Sometimes, you don't know how many bytes were allocated and you don't care. Just call PyBytesWriter_Extend(size) to write size bytes. Example: pylong_int_to_decimal_string() in longobject.c, it gets a writer and just need to extend to write a few bytes.
I'm not sure in which case a PyBytesWriter_Reserve() function would be useful. At least, I don't think that it's currently needed in the Python code base (with my PR). At the same time, I'm not opposed to add it if it's useful :-) vstinner commented on Feb 11, 2025
Consider returning the writer, and having void **buf as an output argument
I'm afraid of strict aliasing issues.
For example, if Extend() signature is changed to:
PyAPI_FUNC(int) PyBytesWriter_Extend(
PyBytesWriter *writer,
void **buf,
Py_ssize_t extend);I'm afraid of compiler issues with the cast:
char *buf;
(...)
if (PyBytesWriter_Extend(writer, &buf, 100) < 0) { /* handle error */ }char* is cast to void*, maybe the compiler will consider that buf cannot be modified by PyBytesWriter_Extend().
I had issues similar to that in the past, sadly I forgot details. encukou commented on Feb 12, 2025
For PyBytesWriter_Extend(), you usually don't want to forget previous PyBytesWriter_Create() and PyBytesWriter_Extend() calls.
Ah, I see, PyBytesWriter_Extend is useful and PyBytesWriter_Reserve can be added later.
As long as the documentation clearly says what the function does, that sounds good.
I'm afraid of strict aliasing issues.
I don't see strict aliasing issues here. Were the issues you saw related to casting between char* and void*? vstinner commented on Feb 14, 2025
PyAPI_FUNC(Py_ssize_t) PyBytesWriter_GetAllocated(
PyBytesWriter *writer);
It seems something like PyBytesWriter_HowMuchMoreCanIWrite(PyBytesWriter *writer, void *buf) would be more useful -- i.e. get the space that's available after buf, rather than the total.
I removed PyBytesWriter_GetAllocated() and I added PyBytesWriter_GetRemaining(PyBytesWriter *writer, void *buf) instead. vstinner commented on Feb 14, 2025
I don't see strict aliasing issues here. Were the issues you saw related to casting between char* and void*?
A "recent" example of type punning / strict aliasing issue with Py_CLEAR()/Py_SETREF(): #99701 These macros take a pointer to a pointer to any object (ex: PyLongObject**) and cast it to a pointer to a pointer to a PyObject (PyObject**). vstinner commented on Feb 16, 2025
Consider returning the writer, and having void **buf as an output argument, for consistency with other _Create/_Finish/*_Discard families.
It would be inconvenient. If &buf is passed as void** with char *buf, it fails with a compiler error:
error: passing argument 1 of 'PyBytesWriter_Create' from incompatible pointer type [-Wincompatible-pointer-types]
You cannot pass char** as void** without an explicit cast to void**.
Also, as written before, I'm afraid of type punning / strict aliasing issues if void** is used. vstinner commented on Feb 16, 2025
PyAPI_FUNC(void*) PyBytesWriter_WriteBytes(..., const char *bytes, ...)
Consider void *bytes (or uint8_t *bytes), per the WIP guidelines.
Ok, I changed bytes parameter type to void*. vstinner commented on Feb 18, 2025 vstinner commented on Mar 12, 2025
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|