GeezerVison is showing github.com — C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions · Issue #111140 · python/cpython
C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions #111140DescriptionFeature or enhancement
The private _PyLong_AsByteArray() and _PyLong_FromByteArray() functions were removed in Python 3.13: see PR #108429.
@scoder asked what is the intended replacement for _PyLong_FromByteArray().
The replacement for _PyLong_FromByteArray() is PyObject_CallMethod((PyObject*)&PyList_Type, "from_bytes", "s#s", str, len, "big") but I'm not sure what is the easy way to set the signed parameter to True (default: signed=False).
The replacement for _PyLong_AsByteArray() is PyObject_CallMethod(my_int, "to_bytes", "ns", length, "big"). Same, I'm not sure how to easy set the signed parameter to True (default: signed=False).
I propose to add public PyLong_AsByteArray() and PyLong_FromByteArray() functions to the C API.
Python 3.12 modified PyLongObject: it's no longer a simple array of digits, but it's now a more less straightforward _PyLongValue structure which requires using unstable functions to access small "compact" values:
- PyUnstable_Long_IsCompact()
- PyUnstable_Long_CompactValue()
So having a reliable and simple way to import/export a Python int object as bytes became even more important.
A code search for _PyLong_AsByteArray in PyPI top 5,000 projects found 12 projects using it:
- Cython (0.29.36)
- blspy (2.0.2)
- catboost (1.2)
- fastobo (0.12.2)
- gevent (22.10.2)
- guppy3 (3.1.3)
- line_profiler (4.0.3)
- msgspec (0.16.0)
- orjson (3.9.1)
- pickle5 (0.0.12)
- pyodbc (4.0.39)
- rlp (3.0.0)
Linked PRs
Activitychanged the title [-] C API: Consider adding a public PyLong_AsByteArray() and PyLong_FromByteArray() functions[/-] [+] C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions[/+] on Oct 20, 2023scoder commented on Oct 21, 2023 Thanks for creating the issue. I agree that the functions should be added. The current replacements seem awful for this kind of basic functionality. Going through an expensive Python call like this for converting between PyLong and large C integer types (int128_t) seems excessive.
Note that at least a couple of projects that you list use Cython implemented parts and thus probably just mention the function in there. I'm sure something like line_profiler would never end up calling it. serhiy-storchaka commented on Oct 21, 2023 It was already discussed several times. This API lacks some features which would be needed for general use. You need to know the size of the resulting bytes array. Calculating it is not trivial, especially for negative integers. Also, it would be core convenient to support "native" ending, not just "big"/"littel".
I have been thinking about implementing a similar API for mpz_import/mpz_export in GMP (https://gmplib.org/manual/Integer-Import-and-Export) or mp_unpack/mp_unpack in libtommath (https://github.com/libtom/libtommath). It is powerful and a kind of standard. It is general enough to support internal PyLong represenation and the marshal format used for long integers (15 bits packed in 16-bit words). But it is only for unsigned integers, you are supposed to store the sign separately. It should be extended to support negative integers in several formats, for convenience and for performance. vstinner commented on Oct 21, 2023 I'm not sure that passing the endian as a string is efficient if this function is part of hot code. serhiy-storchaka commented on Oct 21, 2023 Not as a string. Just 3-variant value native/little/big instead of boolean little/big. vstinner commented on Oct 21, 2023
Not as a string. Just 3-variant value native/little/big instead of boolean little/big.
Sorry, I was confused between C API (int for the endian) and the Python API (string for the endian). scoder commented on Oct 21, 2023
It was already discussed several times. This API lacks some features which would be needed for general use.
Ok, then please put the existing function back in until there is a public replacement. scoder commented on Oct 21, 2023 vstinner commented on Oct 22, 2023
This API lacks some features which would be needed for general use.
Which features are missing?
Do you have links to previous discussions if it was discussed previously? vstinner commented on Oct 22, 2023
You need to know the size of the resulting bytes array.
wcstombs() can be called with NULL buffer to compute the buffer size. It avoids to have to provide a second API "calculate the buffet size".
I suppose that a common use case is also to convert a Python int object to C int type for which there is no C API. Like int128_t scoder commented on Oct 23, 2023 I suppose that a common use case is also to convert a Python int object to C int type for which there is no C API. Like int128_t
Or something like 256 bytes for a crypto key, hash value or similar data type. Probably a known size, so that users can live with an exception if it doesn't fit (because it's an error or can't-handle-that situation).
That said, a function to ask for the bit length of the integer value could be helpful in order to find a suitable integer/storage size. And also more generally to estimate the size of an integer value.
Both together would probably cover a lot of use cases. serhiy-storchaka commented on Oct 23, 2023 encukou commented on Oct 23, 2023 I'm partial to API like mpz_export that accepts a buffer and length, and can:
- Successfully fill the buffer, and output the length
- Not fill the buffer, but output the necessary length
This allows handling the common cases with a minimum of function calls:
- If the size is already known, you only need one function call
- If the caller can pre-allocate a buffer, and the value happens to fit, this is again one function call
- If the size is unknown, and a pre-allocated buffer can't be used or turns out to be too small, it's two function calls
But, IMO we also need general API for exporting/importing Python objects to/from buffers (see e.g. #15 in capi-workgroup/problems), and it would be good to make this consistent.
I'd prefer adding the original functions back until we design a proper replacement. scoder commented on May 5, 2024 The interface seems complete and usable now. Is this done now or is there anything left for this ticket to stay open? gpshead commented on May 5, 2024 Looking things over I like the C API that what was settled upon. It seems to address all of the needs from our earlier discussions.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|