GeezerVison is showing github.com — [C API] Add `PyTupleWriter` API · Issue #139888 · python/cpython
[C API] Add PyTupleWriter API #139888DescriptionFeature or enhancement
Hi,
Creating a tuple with the current C API has multiple issues:
PyTuple_SetItem() and PyTuple_SET_ITEM() modify an immutable tuple.PyTuple_New() creates an incomplete object: items are set to NULL. This is bad:
PyTuple_New() tracks directly the tuple in the garbage collector. For example, gc.get_objects() gives access to the incomplete tuple. Using the tuple, like calling repr(tuple), can crash Python._PyTuple_Resize() is private which is surprising for a documented API. It stays private because it has issues._PyTuple_Resize() modifies an immutable tuple._PyTuple_Resize() must not be used of the refcount is greater than 1: the API is fragile.
I propose adding a new efficient PyTupleWriter API: work on a temporary "writer" object, and then call Finish() on it to get the tuple.
I already proposed a similar API in 2023: _PyTupleBuilder. Since that, Python C API got the PyUnicodeWriter API and the PyBytesWriter API which are efficient writers for str and bytes objects, and the C API Working Group was created. The proposed API is now public and allocates the structure on the heap memory to hide the implementation details (the structure).
Mark Shannon asked if it would be possible to work on a list and then convert the list to a tuple, but it's less efficient. Tuples are commonly used in Python, and so creating a tuple should be efficient.
Mark Shannon also tried to initialize tuple items to None instead of NULL in PyTuple_New(). His attempt failed because of implementation issues. Also, this change only fix some of the issues that I listed, not all of them.
An alternative is to fill a C array of Python objects, call the new PyTuple_FromArray(), and then call Py_DECREF() on the array items. It requires to allocate and deallocate an array, and call Py_DECREF() on items. It can be less efficient.
Linked PRs
Activityzooba commented on Oct 10, 2025 ... but why? We're adding PyTuple_FromArray already, and it's going to be just as easy for a user to fill out a native array and turn it into a tuple in one go. The majority of tuples aren't variable length, they're known at compile time (because that's what a tuple represents - a list is for dynamically sized objects).
Where's the real world benchmarks showing that this API is worth any performance or correctness benefit? We can still deprecate the bad APIs in favour of "just use a native array until you're ready". vstinner commented on Oct 10, 2025
The majority of tuples aren't variable length
This API is mostly a replacement to _PyTuple_Resize(): function when the input size is not known in advance. In that case, you should allocate a few items, fill these items, allocate more items, etc.
Having to manage manually the buffer/array is non trivial and so PyTupleWriter offers a high-level API for that.
When the input size is known, there are other existing safe functions: PyTuple_FromArray() (new! I just added it), PyTuple_Pack(), Py_BuildValue(), etc.
Where's the real world benchmarks showing that this API is worth any performance or correctness benefit?
I ran a micro-benchmark: #139891 (comment)
About correctness, this API should fix bugs when it is possible to discover an incomplete tuple through the GC:
PySequence_Tuple() was fixed recently (Python 3.14) against such bug: commit 5a23994. sergey-miryanov commented on Oct 12, 2025 What do you think about adding a PyTuple_FromSingle and PyTuple_FromPair (or PyTuple_MakeSingle and PyTuple_MakePair) (according to my measurements for pyperformance 1-size and 2-size tuples have about 80% of occurrences)?
Following is a data of occurrences of calls that can be replaced with PyTuple_FromSingle and PyTuple_FromPair:
| file |
function |
count |
_asyncmodule.c |
PyTuple_New(2) |
2 |
_collectionsmodule.c |
PyTuple_Pack(1) |
2 |
_csv.c |
PyTuple_Pack(1) |
1 |
_datetimemodule.c |
PyTuple_Pack(2) |
4 |
|
PyTuple_Pack(1) |
3 |
_elementtree.c |
PyTuple_Pack(2) |
4 |
_functoolsmodule.c |
PyTuple_New(2) |
2 |
_interpretersmodule.c |
PyTuple_Pack(2) |
1 |
_json.c |
PyTuple_New(2) |
1 |
|
PyTuple_Pack(2) |
1 |
|
PyTuple_Pack(1) |
1 |
_operator.c |
PyTuple_Pack(2) |
1 |
_pickle.c |
PyTuple_Pack(2) |
3 |
|
PyTuple_New(2) |
2 |
|
PyTuple_New(1) |
2 |
_ssl.c |
PyTuple_New(2) |
6 |
|
PyTuple_Pack(2) |
1 |
_threadmodule.c |
PyTuple_New(2) |
1 |
_tkinter.c |
PyTuple_Pack(1) |
|
arraymodule.c |
PyTuple_New(2) |
2 |
itertoolsmodule.c |
PyTuple_Pack(2) |
2 |
|
PyTuple_New(2) |
1 |
main.c |
PyTuple_Pack(2) |
1 |
overlapped.c |
PyTuple_New(2) |
2 |
posixmodule.c |
PyTuple_Pack(2) |
1 |
pyexpat.c |
PyTuple_New(1) |
1 |
selectmodule.c |
PyTuple_Pack(2) |
1 |
|
PyTuple_New(2) |
1 |
signal_module.c |
PyTuple_New(2) |
1 |
socket_module.c |
PyTuple_Pack(2) |
3 |
termios.c |
PyTuple_New(2) |
2 |
_ctypes.c |
PyTuple_Pack(2) |
2 |
stgdict.c |
PyTuple_Pack(2) |
1 |
decimal.c |
PyTuple_Pack(2) |
7 |
|
PyTuple_Pack(1) |
2 |
microprotocol.c |
PyTuple_Pack(2) |
2 |
_sre.c |
PyTuple_New(2) |
1 |
datetime.c |
PyTuple_Pack(1) |
2 |
|
PyTuple_Pack(2) |
1 |
getargs.c |
PyTuple_Pack(1) |
1 |
heaptype.c |
PyTuple_Pack(2) |
1 |
|
PyTuple_Pack(1) |
2 |
|
PyTuple_New(2) |
1 |
vectorcall_limited.c |
PyTuple_New(1) |
2 |
multibytecodec.c |
PyTuple_New(2) |
1 |
codeobject.c |
PyTuple_Pack(2) |
7 |
dictobject.c |
PyTuple_Pack(2) |
2 |
|
PyTuple_New(2) |
4 |
enumobject.c |
PyTuple_Pack(2) |
1 |
|
PyTuple_New(2) |
2 |
exceptions.c |
PyTuple_Pack(2) |
7 |
floatobject.c |
PyTuple_Pack(2) |
1 |
frameobject.c |
PyTuple_Pack(2) |
2 |
|
PyTuple_Pack(1) |
1 |
genericaliasobject.c |
PyTuple_Pack(1) |
2 |
listobject.c |
PyTuple_Pack(2) |
1 |
longobject.c |
PyTuple_Pack(2) |
1 |
|
PyTuple_New(2) |
2 |
odictobject.c |
PyTuple_Pack(2) |
2 |
|
PyTuple_New(2) |
1 |
setobject.c |
PyTuple_Pack(1) |
1 |
typeobject.c |
PyTuple_Pack(2) |
2 |
|
PyTuple_Pack(1) |
5 |
typevarobject.c |
PyTuple_Pack(2) |
1 |
|
PyTuple_Pack(1) |
2 |
unicode_format.h |
PyTuple_Pack(2) |
2 |
pegen_errors.c |
PyTuple_Pack(2) |
2 |
_warnings.c |
PyTuple_Pack(2) |
1 |
bltnmodule.c |
PyTuple_Pack(2) |
1 |
ceval.c |
PyTuple_Pack(1) |
1 |
_codegen.c |
PyTuple_Pack(1) |
2 |
compile.c |
PyTuple_Pack(2) |
1 |
crossinterp.c |
PyTuple_Pack(1) |
1 |
errors.c |
PyTuple_Pack(1) |
1 |
hamt.c |
PyTuple_Pack(2) |
1 |
marshal.c |
PyTuple_Pack(2) |
1 |
pylifecycle.c |
PyTuple_Pack(2) |
1 |
Python-tokenize.c |
PyTuple_Pack(2) |
1 |
sysmodule.c |
PyTuple_Pack(1) |
1 |
tracemalloc.c |
PyTuple_New(2) |
1 |
Is it worth to implement those methods, replace in the codebase and benchmark it?
I believe that my question interleaves with #140009 a bit. eendebakpt commented on Oct 12, 2025
What do you think about adding a PyTuple_FromSingle and PyTuple_FromPair (or PyTuple_MakeSingle and PyTuple_MakePair) (according to my measurements for pyperformance 1-size and 2-size tuples have about 80% of occurrences)?
This was suggested in #118222 Reacted by Sergey Miryanov vstinner commented on Oct 12, 2025
What do you think about adding a PyTuple_FromSingle and PyTuple_FromPair (or PyTuple_MakeSingle and PyTuple_MakePair) (according to my measurements for pyperformance 1-size and 2-size tuples have about 80% of occurrences)?
Would you mind to open a separated issue for that? sergey-miryanov commented on Oct 13, 2025 Reacted by Victor Stinner vstinner commented on Dec 2, 2025 I wrote this API to get rid of _PyTuple_Resize() which treats an immutable tuple as mutable. I would like to avoid that: do not provide any function to mutate an immutable tuple.
I failed to convince other core developers that this PyTupleWriter API is useful. Moreover, it's 1.4x slower than the current API based on macros (which mutate an immutable tuple!).
I prefer to give up on PyTupleWriter. It seems like _PyTuple_Resize() has to stay for a few more years. In the meanwhile, one option is to create a list and then call PyList_AsTuple() to convert it to a tuple.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|