GeezerVison is showing github.com — Make `list` objects thread-safe in `--disable-gil` builds · Issue #112087 · python/cpython
Make list objects thread-safe in --disable-gil builds #112087DescriptionFeature or enhancement
I expect this to be implemented across multiple PRs.
For context, here is the change from the nogil-3.12 fork, but things might be done a bit differently in CPython 3.13: colesbury/nogil-3.12@df4c51f82b.
- Most operations should acquire the list object lock using the critical section API.
- Accessing a single element should optimistically avoid locking for performance
- Iterators need some special handling
- list.sort
Iterators
For performance reasons, we don't want to acquire locks while iterating over lists. This means that list iterators have a lesser notion of thread-safety: multiple threads concurrently using the same iterator should not crash but may revisit elements. We should make clear in the documentation that iterators are not generally thread-safe. From an implementation point of view, we should use relaxed atomics to update the it_index variable.
Additionally, we don't want to clear the it_seq reference to the list when the iterator is exhausted in --disable-gil builds. Doing so would pose thread-safety issues (in the "may segfault sense").
I don't expect this to pose any issues for real code. While it's fairly common to share lists between threads, it's rare to share the same iterator object between threads.
Linked PRs
- gh-112087: Update list.{pop,clear,reverse,remove} to use CS #113764
- gh-112087: Update list impl to be thread-safe with manual CS #113863
- gh-112087: Remove duplicated critical_section #114268
- gh-112087: Make list_repr and list_length to be thread-safe #114582
- gh-112087: Make PyList_{Append,Size,GetSlice} to be thread-safe #114651
- gh-112087: Make __sizeof__ and listiter_{len, next} to be threadsafe #114843
- gh-112087: Make list_{count, index, contains} to be thread-safe. #114916
- [3.12] gh-112087: Fix reduce logic for the empty reverse iterator for list #115471
- [3.11] gh-112087: Fix reduce logic for the empty reverse iterator for list #115472
- gh-112087: Make list_{concat, repeat, inplace_repeat, ass_item) to be thread-safe #115605
- gh-112087: Make list_{subscript, ass_slice, slice} to be thread-safe #115854
- gh-112087: Use QSBR technique for list_new/clear for free-thread build #115875
- gh-112087: Make list_{slice, ass_slice, subscript} to be threadsafe #116233
- gh-112087: Test fix for test_sched leak #116237
- gh-112087: Update list_get_item_ref to optimistically avoid locking #116353
- gh-112087: Store memory allocation information into _PyListArray #116529
- gh-112087: Make list.sort to be thread-safe for PEP 703. #116553
- gh-112087: Make
list.extend(dict) behave atomically #117438
ActivityLincoln-developer commented on Nov 16, 2023 Hey, I would love to work on this issue if you don't mind, thanks. colesbury commented on Nov 16, 2023 ContributorAuthor More actions Thanks @Lincoln-developer! I'd suggest starting by transforming some of the functions to use the critical section API. Some functions that use Argument Clinic can just use the @critical_section directive. Other functions will need manual Py_BEGIN_CRITICAL_SECTION() and Py_END_CRITICAL_SECTION() calls. Feel free to tag me on a draft PR if you would like early feedback. Reacted by Angufibo Lincoln corona10 commented on Jan 5, 2024 @colesbury Since this work has not be updated from last year November, can I assign this issue to myself? Reacted by Erlend E. Aasland colesbury commented on Jan 5, 2024 ContributorAuthor More actions Lincoln-developer commented on Jan 5, 2024 I think he can take it on because l have taken longer than expected ,its okay Lincoln-developer commented on Jan 5, 2024 My apologies for not delivering as expected, corona10 commented on Jan 6, 2024 TODO based on diff
- PyList_Insert
- PyList_SetItem
- PyList_Append
- list_repr
- list_length
- list_contains
- list_item
- list_slice
- PyList_GetSlice
- list_concat
- list_repeat
- _list_clear
- list_ass_slice
- list_inplace_repeat
- list_ass_item
- list_insert_impl
- list_clear_impl
- list_remove_impl
- list_copy_impl
- list_append
- list_extend
- list_pop_impl
- list_reverse_impl
- PyList_Reverse
- PyList_AsTuple
- list_index_impl
- list_count
- list_remove
- list_richcompare
- list___sizeof___impl
- list_subscript
- list_ass_subscript
- listiter_next
- listiter_len
- listreviter_next
- listreviter_len
- listiter_reduce_general
Reacted by Erlend E. Aasland corona10 commented on Jan 13, 2024
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|