GeezerVison is showing github.com — Add `PyListWriter` C API to the limited C API · Issue #118 · capi-workgroup/decisions
Add PyListWriter C API to the limited C API #118DescriptionOn Free Threading using abi3t stable ABI, PyList_SetItem() function is between 4.6x and 6.7x slower than the PyList_SET_ITEM() macro (@ngoldbaum measured 2.7x slower). The main cost comes from the fact that PyList_SetItem() requires locking on Free Threading to make the operation atomic, whereas PyList_SET_ITEM() avoids locks. The problem is that the PyList_SET_ITEM() macro is not available in the limited C API, only the "slow" (but safe!) PyList_SetItem() function.
@ngoldbaum proposed adding a new API to make the list creation fast also on Free Threading when using the abi3t stable ABI. With @ngoldbaum help, I designed the following API:
PyListWriter* PyListWriter_Create(Py_ssize_t size);
PyObject* PyListWriter_Finish(PyListWriter *writer);
void PyListWriter_Discard(PyListWriter *writer);
int PyListWriter_Append(PyListWriter *writer, PyObject *item);
int PyListWriter_AppendSteal(PyListWriter *writer, PyObject *item);
int PyListWriter_Extend(PyListWriter *writer, PyObject *iterable);
int PyListWriter_ExtendArray(PyListWriter *writer, PyObject *const *items, Py_ssize_t nitem);
int PyListWriter_ExtendArraySteal(PyListWriter *writer, PyObject *const *items, Py_ssize_t nitem); UPDATE: I added PyListWriter_FromFormat() and PyListWriter_FromBuffer() functions.
UPDATE 2: I removed PyListWriter_FromFormat() and PyListWriter_FromBuffer() functions again. If this API is accepted, I will open a follow-up issue to add PyListWriter_FromBuffer(), as proposed by @encukou. The PyListWriter_FromFormat() function is not really worth it (if PyListWriter_FromBuffer() is added).
Since this API is designed first to solve an abi3t performance issue, I propose adding this PyListWriter API to the limited C API.
The main idea of the design is that the Python list object is not exposed before PyListWriter_Finish(), it allows multiple optimizations like avoiding locking (see below).
Notes on the API:
- As other "Writer "APIs, the API is not thread safe: using it in multiple threads requires external locking (or at least make sure that two threads cannot use the API at the same time).
PyListWriter_Append() and PyListWriter_Extend() are similar to PyList_Append() and PyList_Extend()PyListWriter_ExtendArray() allows passing Python objects as a C array: avoid the cost of creating a temporary object to call PyList_Extend().- The
Steal variants are convenient variants to consume (steal) a reference in all cases (success or error). It's convenient since it's common to create new strong references when building a container (like a frozenset). Also, on Free Threading, PyListWriter_ExtendArraySteal() is 1.28x faster than PyListWriter_ExtendArray() in average.
- See the pull request for the documentation.
Benchmarks:
PyListWriter_Append() is between 2.9x and 3.9x faster than PyList_SetItem() function (and between 1.6x and 1.7x slower than the PyList_SET_ITEM() macro).PyListWriter_Extend() is up to 1.9x faster than PyList_SET_ITEM() macro! (it's between 1.04x and 2.3x faster than PyListWriter_Append().)PyListWriter_ExtendArray() is between 1.3x and 2.0x faster than PyList_SET_ITEM() macro.
Notes on the implementation:
PyListWriter_Create() doesn't allocate a PyListWriter* structure on the heap memory: it's just a PyListObject* hidden as a different pointer type, so PyListWriter_Create() and PyListWriter_Finish() are fast. PyLongWriter_Create() has a similar design.- The API avoids exposing the Python list object to other C functions or to Python, since
PyListWriter* is an opaque pointer (cannot be dereferenced).
- The list is not tracked by the GC and list items are not initialized for best performance. In the best case,
PyListWriter (especially PyListWriter_Extend()) can be faster than PyList_New() + PyList_SET_ITEM() because the memory is not initialized.
It's a little bit unfortunate to have do "duplicate" the PyList API (PyList_New() and PyList_SetItem() already exists), but it's needed to get good performance on abi3t stable ABI.
@encukou would prefer separating Steal suffix with an underscore: PyListWriter_Append_Steal(). We already have Ref suffix like PyDict_GetItemRef() which doesn't use an understand separator.
@encukou considers that PyListWriter_Append() is unnecesary: "you can call PyListWriter_ExtendArray with a size of 1." I would prefer to add PyListWriter_Append() to have an API easier and more convenient to use. For example, that's why we have PyObject_CallNoArgs() and PyObject_CallOneArg() even if we have PyObject_Vectorcall(). It may also be more efficient.
An alternative would be to have an API closer to PyBytesWriter: give a direct access to a PyObject** array (array of PyObject* objects). But such API is more error prone (the caller has to manage reference counting and check array bounds). It also makes the assumption that a Python list is always implemented as an array of Python objects, which is not the case in PyPy where list of integers are stored as an integer array. I would prefer the PyListWriter API with Append/Extend methods. Activityvstinner commented on Oct 27, 2026 changed the title [-]Add `PyListWriter` C API[/-] [+]Add `PyListWriter` C API to the limited C API[/+] on Oct 27, 2026vstinner commented on Nov 2, 2026 zooba commented on Nov 2, 2026 How do you feel about AppendAndDecref rather than AppendSteal?
Extend (taking a Python iterable) feels complicated, but as long as the behaviour is very clearly specified for cases where the iterable raises (i.e. is the builder still valid, and which values does it contain; can the builder be freed without replacing an existing exception, etc.).
Taking a native array is also not ideal, but something has to support it, and this seems like the right API for it. I'd rather see an array of native values and a format code, so that it's our code that handles failures along the way, but I can imagine that being too limiting for some users and if we don't add the API then we'll just have to argue about it every year.
Other than that, seems fine to me. I see no problem with Append being its own function here - there's always a possibility that it could be implemented more efficiently than Extend one day, and so keeping it separate (as a real API) now keeps that possibility open. pitrou commented on Nov 2, 2026
Taking a native array is also not ideal, but something has to support it, and this seems like the right API for it. I'd rather see an array of native values and a format code, so that it's our code that handles failures along the way,
For extra flexibility I think the two concerns should be decoupled:
- populate a native array of
PyObjects from an array of native values and a format code
- populate a list writer by stealing references from a native array of
PyObjects
For example:
int Py_BuildValues(const char *format, PyObject **items, Py_ssize_t nitem, const void *values);
int PyListWriter_ExtendArraySteal(PyListWriter *writer, PyObject *const *items, Py_ssize_t nitem); zooba commented on Nov 2, 2026
For extra flexibility I think the two concerns should be decoupled
They both make some amount of sense, but the advantage of the combined one is that the user never has to deal with a native array containing refcounted objects, and our API doesn't have to deal with potentially corrupt/mis-sized/nulled-out native arrays.
You basically have to do the same handling to initialize/free an array of PyObject * using a for loop as with a Py_BuildValues, but you don't have to do it with a safe PyListWriter, which can simply be released and all its contents will be safely released.
PyListWriter_Create with a size and then _Append is also a safer and just as easy way to do the equivalent of filling a native array with arbitrary PyObject *, but I would expect it's way more common to want to convert a fully native array into a PyListObject of objects, and that's not so easy to get right. It's worth us having the API for that case.
Unless we start accepting PyObject *const * as a common type in the API (which I think would be a mistake), Py_BuildValues is only a very minor convenience.
Extend (taking a Python iterable) feels complicated
A second thought on this - taking something that must be a list (i.e. it has an internal native array of references we can duplicate, rather than having to call __next__) would be fine and useful. It's the arbitrary protocol-based iteration that feels risky, and probably better to either leave it to the original list constructor or make the caller implement it so they can do error handling at the right place. Reacted by Erlend E. Aasland pitrou commented on Nov 2, 2026
7. They both make some amount of sense, but the advantage of the combined one is that the user never has to deal with a native array containing refcounted objects, and our API doesn't have to deal with potentially corrupt/mis-sized/nulled-out native arrays.
That's true, but it's unrealistic for CPython to support all possible kinds of native inputs that might be converted to Python objects (let's say std::optional<double> for a simple example), so there has to be an escape hatch where only the caller, not CPython, knows how to do the conversion.
Of course, the escape hatch can simply be the format code O or N as supported by Py_BuildValue 😁
A second thought on this - taking something that must be a list (i.e. it has an internal native array of references we can duplicate, rather than having to call __next__) would be fine and useful. It's the arbitrary protocol-based iteration that feels risky, and probably better to either leave it to the original list constructor or make the caller implement it so they can do error handling at the right place.
That sounds fine to me. vstinner commented on Nov 3, 2026
Extend (taking a Python iterable) feels complicated, but as long as the behaviour is very clearly specified for cases where the iterable raises (i.e. is the builder still valid, and which values does it contain; can the builder be freed without replacing an existing exception, etc.).
If the iterator raises an exception, Extend() fails with the exception and return -1.
In my current implementation, in this case, items already added are kept in the writer. Maybe this behavior is not great since the caller cannot know exactly how many items were added and where the iterable raises an exception. The only safe option for the caller is to discard the whole writer.
In the PyUnicodeWriter, I made functions atomic. On error, characters already written are discarded. I can change my Extend() implementation to do the same there: discard added items on error, to reset the writer to its previous state. vstinner commented on Nov 3, 2026
How do you feel about AppendAndDecref rather than AppendSteal?
In the internal C API, we already use the Steal suffix for functions stealing references. Examples: _PyTuple_FromArraySteal() and PyStackRef_FromPyObjectSteal().
Steal is now a coined term in the Python glossary.
Last year, when the term was discussed in the C API Working Group, Steal suffix got the majority of votes. Reacted by Petr Viktorin and Erlend E. Aasland vstinner commented on Nov 3, 2026
our API doesn't have to deal with potentially corrupt/mis-sized/nulled-out native arrays
Arrays of Python objects (PyObject*) are not completely new in the existing Python C API.
- There was the
METH_FASTCALL calling convention using an array of objects to pass positional and keyword arguments to function calls, later standardized as PEP 590 "vectorcall".
- For vectorcall,
PyArg_ParseArray() and PyArg_ParseArrayAndKeywords() were added to Python 3.15 to parse an array of objects (the opposite of Py_BuildValue() discussed above).
PyTuple_FromArray() added to Python 3.15 takes an array of objects (decision issue). The caller must pass a valid array, otherwise the function does crash (adding more checks would make the function slower, whereas it's used in hot code).
Note: even if PyTuple_FromArray() was added, no helper function to clear an array of object (call Py_DECREF() on each item) was added yet.
I agree with @pitrou that it's better to decouple functions to produce an array of objects and functions to consume an array of objects. It's more flexible. Later, we can add N producer and P consumer functions (N + P functions) this way, instead of having to add N x P combined functions (larger API, more code to maintain). There are various ways to create such array. For vectorcall, a common (and efficient) way is to allocate the array on the stack and pass borrowed references. Example:
PyObject *stack[] = {(PyObject *)self, limit};
return PyObject_Vectorcall(state->asyncio_task_get_stack_func,
stack, 2, NULL);Another example, more compact:
result = PyTuple_FromArray((PyObject*[]){ traverse, clear, free }, 3);Counter-example: There is a "combined" function PyObject_CallFunction() which builds a temporary array from a format string, and then discards the array (call Py_DECREF() on each item). This function existed before FASTCALL/vectorcall, and it's convenient to use!
I'm not a fan of such format string since it's not very efficient to build an array this way. To parse function arguments, argument Clinic produces faster code (which doesn't use format strings). encukou commented on Nov 3, 2026 Reacted by Erlend E. Aasland vstinner commented on Nov 3, 2026
I can change my Extend() implementation to do the same there: discard added items on error, to reset the writer to its previous state.
I modified my PR to make Extend() atomic: discard added iterable items on error. IMO it's a safer behavior than keeping added items on error.
Also, the writer remains usable after PyListWriter_Extend() error (there are tests on this case).
But list.extend(iterable) behaves differently on this corner case: added items are kept when iterable raises an exception:
class IterError:
def __init__(self):
self.index = 1
def __iter__(self):
return self
def __next__(self):
value = self.index
self.index += 1
if value >= 3:
raise ValueError("iter error")
return value
iterable = IterError()
lst = []
try:
lst.extend(iterable)
except ValueError:
pass
else:
raise AssertionError
assert iterable.index == 4
assert lst == [1, 2], lstlst is non-empty on list.extend() error.
This corner case is only for the generic code path calling PyObject_GetIter() and type->tp_iternext(): list_extend_iter_lock_held() function. For the following types, list.extend() is atomic (either add all items or add nothing):
list, tupleset, frozensetdict, dict.values(), dict.keys(), dict.items()
Note: list.extend(list), adding a list to it-self, is implemented as list *= 2 which is also atomic. vstinner commented on Nov 3, 2026 I added int PyListWriter_FromFormat(PyListWriter *writer, const char *format, ...) to my PR. It uses https://docs.python.org/dev/c-api/arg.html#building-values format string. For example, PyListWriter_FromFormat(writer, "iii", 1, 2, 3) adds 3 integers.
IMO we should still provide PyListWriter_ExtendArray() which is the most generic (and most efficient) way to add items to a list.
So, it's similar to PyUnicodeWriter API: PyUnicodeWriter_Format() function has const char *format, ... API, and there are more specialized functions such as PyUnicodeWriter_WriteUTF8() and PyUnicodeWriter_WriteStr(), whereas PyUnicodeWriter_Format() could be used with "%s" or "%S" formats. pitrou commented on Nov 3, 2026
For example, PyListWriter_FromFormat(writer, "iii", 1, 2, 3) adds 3 integers.
Hmm, I think I'd rather write:
const auto ints = std::array{1, 2, 3};
PyListWriter_FromFormat(writer, "i", ints.data(), ints.size());i.e.:
- pass a single format code for all values (homogenous unpack)
- pass a pointer and size into the native values, rather than through the varargs calling convention (which might also be annoying for non-C languages)
vstinner commented on Nov 3, 2026 I made an experiment locally on an array producer/consumer API, I added Py_BuildArray(). But I'm not fully satisfied of the API for the simple case. Example adding 3 integers to a list writer:
int
add_3_integers(PyListWriter *writer, int a, int b, int c)
{
Py_ssize_t size;
PyObject **array;
if (Py_BuildArray(&array, &size, "iii", a, b, c) < 0) {
return NULL;
}
int res = PyListWriter_ExtendArraySteal(array, size);
PyMem_Free(array);
return res;
}Using PyListWriter_FromFormat(), the code becomes a single function call, error handling is way simpler:
int
add_3_integers(PyListWriter *writer, int a, int b, int c)
{
return PyListWriter_FromFormat(array, "iii", a, b, c);
}I'm not satisfied by Py_BuildArray() because for small arrays, you really want to avoid a "slow" allocation on the heap, and allocate it on the stack instead. But then,the API becomes even worse:
int
add_3_integers(PyListWriter *writer, int a, int b, int c)
{
PyObject* small_array[5];
Py_ssize_t size;
PyObject **array;
if (Py_BuildArray(&array, &size,
small_array, Py_ARRAY_LENGTH(small_array),
"iii", a, b, c) < 0) {
return NULL;
}
int res = PyListWriter_ExtendArraySteal(array, size);
if (array != small_array) {
PyMem_Free(array);
}
return res;
}pitrou commented on Nov 3, 2026
My thinking was that this escape hatch is to append one value at a time to the list writer, so that there's never an unmanaged array of PyObject * involved.
Right, but that also leaves performance on the table.
For the sophisticated use cases where the objects are nested values (such as dicts or lists), I don't think it would make a tangible difference (because of the cost of building the objects themselves), but when wanting to append e.g. a hundred times None, the function call overhead of each append becomes important. encukou commented on Nov 4, 2026 Yup. I don't think optimizing Append on our side can ever make things faster than the caller switching to Extend (except for one-element lists that aren't worth optimizing).
This is an API for creating lists efficiently. For ergonomics, we have PyList_Append.
In any case, I don't think _ExtendArray(list, item, 1) is as discoverable as _Append(list, item), and so you'd have to clearly document the equivalence and it would likely be considered a wart for all time.
I fully agree with documenting it. But the 1 is not a wart, it's a hint about preferred usage (i.e. avoid repeated Append).
Show me third-party C developers raving about getting to use native arrays and I might believe you, but I'm pretty sure they're not.
Show me ones raving about C :)
We're building an interface to native code. Yes, native arrays have all the problems with refcounting, multiplied by all the problems with C arrays, but they're the basic building block at this level. Yes, you want PyO3 or nanobind to wrap them for you -- but the easiest way to allow that is to expose the raw stuff.
How would you feel about splitting PyListWriter_FromBuffer into a separate proposal? There's a bunch of edge cases and ambiguities around converting arbitrary buffers to Python objects. If the aim is “do what memoryview does”, I think for the initial implementation we can ask users to create a temporary memoryview explicitly. pitrou commented on Nov 4, 2026
If the aim is “do what memoryview does”, I think for the initial implementation we can ask users to create a temporary memoryview explicitly.
... and then? What step follows? Is it:
- creating a temporary memoryview using
PyMemoryView_FromBuffer (that's two memory allocations, because of the memoryview's internal "managed buffer" object)
- call
PyListWriter_Extend with the memoryview as an iterable (with a third memory allocation to create the iterator?)
- destroy the temporary memoryview
That's more involved, and probably less efficient, than a direct PyListWriter_FromBuffer method.
(still, this is a nice suggestion in case we go with a minimal API for PyListWriter)
How would you feel about splitting PyListWriter_FromBuffer into a separate proposal? There's a bunch of edge cases and ambiguities around converting arbitrary buffers to Python objects.
The implementation already exists in the memoryview object and its tolist method, so it mostly has to be factored out, right? vstinner commented on Nov 4, 2026 @encukou:
Yup. I don't think optimizing Append on our side can ever make things faster than the caller switching to Extend (except for one-element lists that aren't worth optimizing).
So I ran a benchmark to compare calling PyListWriter_Append(writer, item) versus calling PyListWriter_ExtendArray(writer, &item, 1):
| Benchmark |
writer_append |
writer_append_extend |
| size=10 |
50.3 ns |
55.6 ns: 1.10x slower |
| size=100 |
383 ns |
400 ns: 1.04x slower |
| size=1,000 |
3.07 us |
3.71 us: 1.21x slower |
| size=100,000 |
465 us |
546 us: 1.17x slower |
| Geometric mean |
(ref) |
1.10x slower |
Benchmark hidden because not significant (1): size=0
PyListWriter_ExtendArray(writer, &item, 1) is currently 1.10x slower than PyListWriter_Append(writer, item) in average. I'm not surprised: PyListWriter_ExtendArray() code is more complex than PyListWriter_Append(writer, item) since it has to deal with an array of objects, instead of having to deal with a single object.
I still consider that it's worth it to add Append() and AppendSteal() functions.
If you look at the current usage of the PyList_New() API, you can see that it's a very common pattern to use PyList_New() with PyList_Append(), and it would be complicated to refactor the code to call PyListWriter_ExtendArray() with add multiple items at once. Using Append() is more natural (easier to write). Just one (simplified) example from _decimal.c:
list = PyList_New(0);
if (list == NULL) {
return NULL;
}
for (cm = state->cond_map; cm->name != NULL; cm++) {
if (flags&cm->flag) {
if (PyList_Append(list, cm->ex) < 0) {
goto error;
}
}
}
return list;Also, having a PyListWriter_Append() function makes it easier to convert using PyList_New() code to the PyListWriter API. encukou commented on Nov 4, 2026 @pitrou: Yes, I'd prefer the minimal API now. Before reusing memoryview.tolist, we should document or fix its quirks (AFAIK there are some; I'd prefer not doing the research right now), and then decide if it's good for a low-level API. I don't want to derail this discussion.
@vstinner
So I ran a benchmark
Well, you ran it for a single item, right?
Could you compare PyListWriter_ExtendArray(writer, items, 2) to calling PyListWriter_Append twice?
As I said, I don't think that building a one-element list is a use case to optimize for.
If you look at the current usage of the PyList_New() API, you can see that it's a very common pattern to use PyList_New() with PyList_Append(), and it would be complicated to refactor the code to call PyListWriter_ExtendArray() with add multiple items at once. Using Append() is more natural (easier to write).
For performance, the caller should switch to batching. I don't want a solution that saves 10% but avoids even larger wins.
(For your _decimal.c example: in current code, the size of cond_map is known at compile time.) vstinner commented on Nov 4, 2026 @pitrou:
That's more involved, and probably less efficient, than a direct PyListWriter_FromBuffer method.
I ran a benchmark to compare 3 functions:
- writer_extend_array_steal: create an array of Python objects using
PyLong_FromLong() and call PyListWriter_ExtendArraySteal() -- require more complex error handling code: if PyLong_FromLong() fails, the partially initialized array should be deallocated
- writer_frombuffer: call
PyListWriter_FromBuffer() on an array of C int -- simplest code
- writer_memoryview_extend: call
PyMemoryView_FromBuffer() on an array of C int and call PyListWriter_Extend(writer, memoryview)
Results:
| Benchmark |
writer_extend_array_steal |
writer_frombuffer |
writer_memoryview_extend |
| size=0 |
15.4 ns |
18.2 ns: 1.18x slower |
116 ns: 7.53x slower |
| size=10 |
45.0 ns |
55.7 ns: 1.24x slower |
142 ns: 3.14x slower |
| size=100 |
287 ns |
368 ns: 1.28x slower |
308 ns: 1.07x slower |
| size=1,000 |
2.58 us |
3.41 us: 1.32x slower |
1.49 us: 1.74x faster |
| size=100,000 |
1.19 ms |
1.23 ms: 1.03x slower |
396 us: 3.00x faster |
| Geometric mean |
(ref) |
1.21x slower |
1.37x slower |
So PyListWriter_FromBuffer() is 1.21x slower in average than PyLong_FromLong() loop + PyListWriter_ExtendArraySteal(): that's the cost of providing a convenient API to add an array of native C types.
PyMemoryView_FromBuffer() + PyListWriter_Extend() is between 1.07x and 3.14x slower (up to 7.53x slower for an empty array) than PyLong_FromLong() loop + PyListWriter_ExtendArraySteal(). As @pitrou explained, this function has to allocate multiple objects on the heap memory, use the slower iterator protocol (PyObject_GetIter() + loops on Py_TYPE(it)->tp_iternext), and it uses less optimized code than PyListWriter_FromBuffer().
PyMemoryView_FromBuffer()+PyListWriter_Extend() compared to PyListWriter_FromBuffer():
| Benchmark |
writer_frombuffer |
writer_memoryview_extend |
| size=0 |
18.2 ns |
116 ns: 6.38x slower |
| size=10 |
55.7 ns |
142 ns: 2.54x slower |
| size=100 |
368 ns |
308 ns: 1.19x faster |
| size=1,000 |
3.41 us |
1.49 us: 2.30x faster |
| size=100,000 |
1.23 ms |
396 us: 3.10x faster |
| Geometric mean |
(ref) |
1.14x slower |
pitrou commented on Nov 4, 2026 Thanks for the numbers.
It seems unexpected that writer_memoryview_extend becomes faster than writer_frombuffer above a certain threshold. I can't think of a reason why that would be the case (is writer_frombuffer allocating an unbounded array of objects?). vstinner commented on Nov 4, 2026 @encukou:
How would you feel about splitting PyListWriter_FromBuffer into a separate proposal? There's a bunch of edge cases and ambiguities around converting arbitrary buffers to Python objects. If the aim is “do what memoryview does”, I think for the initial implementation we can ask users to create a temporary memoryview explicitly.
Would you mind to elaborate on these "ambiguities"? My implementation reuses memoryobject.c code and so PyListWriter_FromBuffer() should behave the same as PyMemoryView_FromBuffer()+PyListWriter_Extend(). It supports the same format strings. It supports shape, strides and suboffsets parameters. But it requires a Py_buffer with 1 dimension (other dimensions are not supported on purpose). Or does someone think that supporting other dimensions would be useful/more sense?
It should give the same output than memorview(view).tolist() or arrray.array(...).tolist() in Python. I just updated tests to check exactly that. vstinner commented on Nov 4, 2026 @encukou:
Note that behaviour-wise, PyListWriter_Append(list, item) is equivalent to PyListWriter_ExtendArray(list, item, 1).
The correct code is PyListWriter_ExtendArray(list, &item, 1): & was missing. It's a little bit unusual to have to pass an address to a Python object in the Python C API.
Or you might create explicitly an array: PyObject* array[1] = {item}; PyListWriter_ExtendArray(list, array, 1). Reacted by Petr Viktorin and Steve Dower encukou commented on Nov 5, 2026
Would you mind to elaborate on these "ambiguities"?
Sure, but ideally in a follow-up, to not derail this discussion.
Mostly, memoryview.tolist supports its own subset of the buffer format syntax, with differences from other modules that use it (array, ctypes, NumPy). Do we use that? Is the plan to eventually support PEP 3118 additions, or keep PyListWriter_FromBuffer's initial limitations?
Format strings have a footgun: struct has native (variable) or "standard" (fixed) size. People will will assume the wrong one, and get memory corruption on some platforms.
But it requires a Py_buffer with 1 dimension (other dimensions are not supported on purpose). Or does someone think that supporting other dimensions would be useful/more sense?
I'm not sure, but, should this really be different from tolist? vstinner commented on Nov 7, 2026 @encukou:
How would you feel about splitting PyListWriter_FromBuffer into a separate proposal?
Ok, I removed it again from the issue.
I also removed PyListWriter_FromFormat(): it's is not really worth it (especially if PyListWriter_FromBuffer() is added). vstinner commented on Nov 7, 2026 Ok, I think that this API has been discussed in length, we can now vote on adding this API to the limited C API: add Create/Finish/Discard, add 2 Append and 3 Extend.
Vote to add PyListWriter C API to the limited C API:
Note: if this API is approved, I will open a follow-up issue to propose adding PyListWriter_FromBuffer(). vstinner commented on Nov 8, 2026 I propose adding PyListWriter_Extend() which accepts any iterable, not only list and tuple.
I agree with @zooba that the iterable form of PyListWriter_Extend is not necessary, a simpler form that only accepts a list (or tuple) is IMHO sufficient
In my current implementation, PyListWriter_Extend() is basically the same as PyList_Extend(). I would prefer that PyListWriter_Extend() accepts any iterable object, rather than only list and tuple, so it's easier to migrate existing code calling PyList_Extend() to PyListWriter.
PyListWriter can call other internal/private functions to provide an efficient implementation. It's complicated to consume an iterator: it takes around 74 lines of C code (list_extend_iter_lock_held()) to call PyObject_GetIter() and loop on Py_TYPE(it)->tp_iternext. I prefer to provide a simple function for that, rather than forcing users to write their own implementation (which may be have more bugs or be less efficient).
I don't understand well why you don't want to support types other than list and tuple? encukou commented on Nov 9, 2026 I'm still -1 on Append, but +1 for the rest of the API. zooba commented on Nov 9, 2026
the 1 is not a wart, it's a hint about preferred usage (i.e. avoid repeated Append).
This swung me over to agreeing with you, but then we both missed the & and passed PyObject * as PyObject ** 😉 So I'm around +0 on Append, which basically means I'm reserving the right to say "I told you so" later on when leaving it out becomes an issue. I won't block the proposal over it.
I don't understand well why you don't want to support types other than list and tuple?
Because I don't want _Extend to re-enter Python code. I want these C APIs to be fully native below them, and never enter into the interpreter loop again (with the exception of a decref operation that has to call finalizers/etc. for elements that have no remaining references). Iteration in the general case is hard, but also has overheads that won't be improved by using this API, so there's very little benefit.
Meanwhile, we can very cheaply copy the entire contents of any of our own types where we can directly access the underlying array, which will be much better than what users can do.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|