GeezerVison is showing github.com — PyType_GetModuleByDef family for binary functions performance · Issue #117578 · python/cpython
PyType_GetModuleByDef family for binary functions performance #117578DescriptionFeature or enhancement
Proposal:
When implementing an extension module with a module state enabled,
binary tp slot functions (e.g. nb_add) need two PyType_GetModuleByDef() to be used like the following to compare the given types with a heap-type object in the module state, which can be slow:
static PyObject *
foo_add(PyObject *left, PyObject *right)
{
...
PyObject *module = PyType_GetModuleByDef(Py_TYPE(left), &module_def);
if (module == NULL) {
PyErr_Clear();
module = PyType_GetModuleByDef(Py_TYPE(right), &module_def);
}
...
}
- 3.13.0 alpha5
_decimal (module state ver.)
from timeit import timeit
f = lambda s: timeit(s, 'import _decimal; d = _decimal.Decimal(1)')
a = f('d + 1')
b = f('1 + d')
print('d + 1:', a)
print('1 + d:', b, b / a)d + 1: 0.11202071857405826
1 + d: 0.49533294743326095 4.421797625818569
The difference mainly comes from a TypeError emission from PyType_GetModuleByDef(), so it would be nice if we could have a new function which receives two types and emits an error after checking them.
cc @encukou
Related issue:
Linked PRs
Reacted by An Long, Petr Viktorin and Erlend E. Aasland Activityencukou commented on Apr 8, 2024 Make it private at first, then perhaps expose it in another issue. Reacted by Erlend E. Aasland and neonene gvanrossum commented on Apr 8, 2024 @neonene Are you mainly interested in fixing the specific case for _decimal? Other stdlib modules? 3rd party modules? Limited API? I agree with @encukou that if you want this mainly for _decimal, a private function would be best, and we should start there. neonene commented on Apr 8, 2024 ContributorAuthor More actions It seems that extension modules such as _decimal and _ctypes cannot use any private extern functions, so my proposal (a private function) might be related to built-in modules. I'm thinking I should take back if not so beneficial. gvanrossum commented on Apr 8, 2024
It seems that extension modules such as _decimal and _ctypes cannot use any private extern functions
Isn't this a recent side effect caused by a desire to use only the Limited API? @vstinner should know more. I think the slowdown you've observed is rather significant, so maybe we need to make an exception that restriction (if that's what's causing your observation).
In modules not restricted to the Limited API, private APIs can be used as long as they use the PyAPI_FUNC() macro. neonene commented on Apr 9, 2024 ContributorAuthor More actions Tested _decimal with #117661, where the private function was temporarily exposed:
d + 1: 0.1184351708673473
1 + d: 0.1197695422950866 1.0112666821685414
vstinner commented on Apr 9, 2024
Make it private at first, then perhaps expose it in another issue.
Yep, I agree. Also, there is maybe no need to expose it in public right now :-)
It seems that extension modules such as _decimal and _ctypes cannot use any private extern functions
Python has multiple stdlib extensions built as shared extensions: you should export internal functions using PyAPI_FUNC() for them.
Example in Include/internal/pycore_object.h:
// Export for 'math' shared extension
PyAPI_FUNC(PyObject*) _PyObject_LookupSpecial(PyObject *, PyObject *);
Isn't this a recent side effect caused by a desire to use only the Limited API?
_decimal and _ctypes are not built with the limited C API. If tomorrow there is a great advantage to use the internal C API in a stdlib extension built with the limited C API, just remove the few lines around Py_LIMITED_API and that's it.
neonene commented on Apr 9, 2024 ContributorAuthor More actions Thanks. I've been very confused around the PyAPI_FUNC() macro. picnixz commented on Aug 17, 2024 Re-opening until #123100 is merged (just for tracking purposes). neonene commented on Sep 26, 2024 ContributorAuthor More actions Superseded by PyType_GetBaseByToken(): d7248cd
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|