GeezerVison

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

All commit authors signed the Contributor License Agreement.

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Thanks @scopreon for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15.
🐍🍒⛏🤖

Sorry, @scopreon and @JelleZijlstra, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 57a0e570d36f41b953a91bbaf4262a5d05d0391b 3.14

GH-150039 is a backport of this pull request to the 3.15 branch.

GH-150062 is a backport of this pull request to the 3.14 branch.

I propose to revert these changes, since they cause Out of Memory on the main branch when running test_free_threading.test_pickle. Because of that, the GitHub Action CI jobs fail with exit code 143 (issue gh-150114).

I don't think that this change is correct: sometimes (race condition), test_pickle_dumps_with_concurrent_dict_mutation() of test_free_threading.test_pickle enters an unlimited loop and exhaust the system memory. See the issue gh-150257.

The following code was added to _pickle to fix the crash gh-146452:

/* gh-146452: Wrap the dict iteration in a critical section to prevent
   concurrent mutation from invalidating PyDict_Next() iteration state. */
static int
batch_dict_exact(PickleState *state, PicklerObject *self, PyObject *obj)
{
    int ret;
    Py_BEGIN_CRITICAL_SECTION(obj);
    ret = batch_dict_exact_impl(state, self, obj);
    Py_END_CRITICAL_SECTION();
    return ret;
}

This PR replaces this function with "fine-grained sections".

Problem: Py_BEGIN_CRITICAL_SECTION(obj); and Py_END_CRITICAL_SECTION(); is put around a single PyDict_Next() call, whereas the PyDict_Next() documentation recommends putting the whole loop in the critical section.

By the way, the 3.15 backport of this PR is different since there was not batch_dict_exact_impl() function. Maybe a 3.15 backport of the previous fix was missed?

I prepared 3 reverts:

• main: Revert "gh-146452: Improve locking granularity in pickle's batch_dict_exact and fix race condition (#150025)" #150261
• 3.15: [3.15] Revert "[3.15] gh-146452: Improve locking granularity in pickle's batch_dict_exact and fix race condition (GH-150025) (#150039)" #150262
• 3.14: [3.14] Revert "[3.14] gh-146452: Improve locking granularity in pickle's batch_dict_… (#150062)" #150263

Once the 3.15 revert will be merged, the commit e62a611 should be backported to the 3.15 branch.

Thanks for spotting this regression @vstinner , and for the explanation. Sorry for overlooking that.