GeezerVison is showing github.com — gh-161850: Copy a deque into a list under a single critical section by 113xiaoji · Pull Request #161854 · python/cpython
Conversation
113xiaoji
commented
Nov 12, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Summary
Depends on #161853. This is a draft follow-up. The branch currently contains the commits from #161853, so the diff against main includes that prerequisite. The PR2-only changes are deque/list bulk materialization and its tests/NEWS entry.
Closes #161850 once both changes are merged.
Changes
- Add a fast path for copying a deque or the remaining items of a forward deque iterator into a list while holding the list and deque critical sections once.
- Register the internal fast-path hooks from
_collections, where the deque types are defined. Store the hook pointer in per-interpreter state, using release/acquire publication for concurrent free-threaded access.
- Recognize the supported built-in slots and fall back for subclasses overriding iteration or length behavior.
- Preserve iterator mutation detection, exhaustion/reduction state and generic-path list allocation behavior in the covered cases.
- Add block-boundary, partially consumed iterator, subclass, allocation/refcount and concurrent copy/iteration tests.
The atomic bulk copy can change which outcomes are observed under concurrent mutation; this is not a claim that it reproduces every possible interleaving of repeated next() calls. The supported built-in-slot paths avoid user-defined iteration/length callbacks.
Test Plan
- Source review and
git diff --check pass for the PR2-only delta and the combined branch.
- Earlier byte-for-byte Mako HTML/API checks and performance measurements used a CPython 3.14.6 backport, not this main-based (3.16 development) port.
- Main-based pydebug builds of ordinary and free-threaded CPython pass
test_deque, test_list, test_collections, test_free_threading.test_collections, test_embed and test_interpreters (the GIL-specific suites are skipped in the opposite configuration).
- The new-head generated-files job passes the full C globals check, exported symbols, limited ABI and regeneration checks. The complete CI matrix passed all 48 jobs on head
b83e4241979bfbf5036008b0b3675ac50ca884c5, including ordinary/free-threaded TSan, ASan, MSan, UBSan, fuzz and platform tests.
- This PR stays draft while the prerequisite and hook design are reviewed.
For context, the combined 3.14.6 backport reduced unmodified Mako's FT render time by approximately 33% on Kunpeng 950 and 9% on EPYC 9654 in the issue's measurements. These are single-thread workload results and do not isolate this PR from its prerequisite.
AI tools assisted preparation of the change and submission text.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
In the free-threaded build, dequeiter_next(), dequereviter_next() and
dequeiter_reduce() entered a two-object critical section on the iterator
and its deque. All writes to the iterator's state already happen while
holding the deque's lock, so the deque's critical section alone is enough
to serialize concurrent next() calls on a shared iterator. This halves
the atomic read-modify-write operations per item.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
list(d), [*d], list.extend(d) and PySequence_Fast(d) -- and hence
str.join(d), which hands the deque's iterator to list.extend() --
fetched one item per tp_iternext call, each entering a critical section.
Add a fast path that copies all remaining items of a deque or a forward
deque iterator while holding the list's and the deque's critical sections
once. The _collections module registers it with listobject.c, since the
deque type lives in that module. The result is identical to running the
iterator to exhaustion: no user code runs, the mutation check matches
next(), and the list's allocation matches the generic path.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
All commit authors signed the Contributor License Agreement.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Keep the mutable deque hook pointer in PyInterpreterState instead of
a C global so check-c-globals accepts it. Preserve release/acquire
publication for concurrent free-threaded registration.
Validated with ordinary and free-threaded pydebug builds, the focused
C globals check, and deque/list/collections/embed/interpreter tests.
pythongh-161850
Reviewers
rhettinger
Awaiting requested review from rhettinger
rhettinger will be requested when the pull request is marked ready for review
rhettinger is a code owner
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|