GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — gh-139772: Add PyDict_NewPresized() function by vstinner · Pull Request #139773 · python/cpython Conversation
|
|
I convert this PR to a draft for now since it seems like the API is misused by 3rd party projects, and I proposed |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
eb555c6 to
8bb9715
Compare
8bb9715 to
8a61f5a
Compare
|
I rewrote the PR to add unicode_keys parameters: |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
There are two news entries. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
Benchmark on PyDict_New() vs PyDict_NewPresized() with Unicode keys:
Benchmark hidden because not significant (1): dict-1 UPDATE: Benchmark rerun to fix a refleak in benchmarks (DECREF key and value after SetItem). Code: Detailsdiff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index c14f925b4e7..d2044b55f76 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2595,6 +2595,80 @@ create_managed_weakref_nogc_type(PyObject *self, PyObject *Py_UNUSED(args)) } +static PyObject * +bench_dict_new(PyObject *ob, PyObject *args) +{ + Py_ssize_t size, loops; + if (!PyArg_ParseTuple(args, "nn", &size, &loops)) { + return NULL; + } + + PyTime_t t1, t2; + PyTime_PerfCounterRaw(&t1); + for (Py_ssize_t loop=0; loop < loops; loop++) { + PyObject *d = PyDict_New(); + if (d == NULL) { + return NULL; + } + + for (Py_ssize_t i=0; i < size; i++) { + PyObject *key = PyUnicode_FromFormat("%zi", i); + assert(key != NULL); + + PyObject *value = PyLong_FromLong(i); + assert(value != NULL); + + assert(PyDict_SetItem(d, key, value) == 0); + Py_DECREF(key); + Py_DECREF(value); + } + + assert(PyDict_Size(d) == size); + Py_DECREF(d); + } + PyTime_PerfCounterRaw(&t2); + + return PyFloat_FromDouble(PyTime_AsSecondsDouble(t2 - t1)); +} + + +static PyObject * +bench_dict_presized(PyObject *ob, PyObject *args) +{ + Py_ssize_t size, loops; + if (!PyArg_ParseTuple(args, "nn", &size, &loops)) { + return NULL; + } + + PyTime_t t1, t2; + PyTime_PerfCounterRaw(&t1); + for (Py_ssize_t loop=0; loop < loops; loop++) { + PyObject *d = PyDict_NewPresized(size, 1); + if (d == NULL) { + return NULL; + } + + for (Py_ssize_t i=0; i < size; i++) { + PyObject *key = PyUnicode_FromFormat("%zi", i); + assert(key != NULL); + + PyObject *value = PyLong_FromLong(i); + assert(value != NULL); + + assert(PyDict_SetItem(d, key, value) == 0); + Py_DECREF(key); + Py_DECREF(value); + } + + assert(PyDict_Size(d) == size); + Py_DECREF(d); + } + PyTime_PerfCounterRaw(&t2); + + return PyFloat_FromDouble(PyTime_AsSecondsDouble(t2 - t1)); +} + + static PyMethodDef TestMethods[] = { {"set_errno", set_errno, METH_VARARGS}, {"test_config", test_config, METH_NOARGS}, @@ -2691,6 +2765,8 @@ static PyMethodDef TestMethods[] = { {"toggle_reftrace_printer", toggle_reftrace_printer, METH_O}, {"create_managed_weakref_nogc_type", create_managed_weakref_nogc_type, METH_NOARGS}, + {"bench_dict_new", bench_dict_new, METH_VARARGS}, + {"bench_dict_presized", bench_dict_presized, METH_VARARGS}, {NULL, NULL} /* sentinel */ };
import pyperf import _testcapi runner = pyperf.Runner() for size in (1, 5, 10, 25, 100, 500, 1_000): runner.bench_time_func(f'dict-{size:,}', _testcapi.bench_dict_new, size)
import pyperf import _testcapi runner = pyperf.Runner() for size in (1, 5, 10, 25, 100, 500, 1_000): runner.bench_time_func(f'dict-{size:,}', _testcapi.bench_dict_presized, size) |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
I created capi-workgroup/decisions#80 to the C API Working Group for this API. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
Benchmark on
Benchmark hidden because not significant (1): dict-1 UPDATE: Benchmark re-run to fix refleak in the benchmark. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
This seems useful to me for PyO3 👍 I am unsure how reliably we will be able to use the |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Correct. If you know your input data, you can set the unicode_keys hint in advance, before consuming the iterator. You can use If you don't know your input data, you might need to consume the iterator and store keys and values in a temporary array, and then call |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
I think this seems the wrong way around for me as a user; if I don't know my input data I'd rather not collect it to a temporary array, it could be a large dataset which would be a big temporary allocation. If I know the input data, I was thinking I would even be able to allocate the items in stack memory before calling |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Or are you saying that it is more efficient to use |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Oh, I don't know which function is faster. So I ran benchmarks: #139963 (comment). |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
👍 that matches what I was thinking then:
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
@vstinner You forgot Excluding PyUnicode_FromFormat(), the performance difference between PyDict_NewPresized + PyDict_SetItem and PyDict_FromItems was negligible.
Benchmark hidden because not significant (2): dict-10, dict-10,000 Details |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
So performance difference should be very small. In case of free-threaded build, |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Ooops :-( I fixed the benchmarks and re-run them. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
@vstinner |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
I close this PR in favor of #139963. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.