GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-116738: Make _csv module thread-safe by aisk · Pull Request #118344 · python/cpython

Conversation

@aisk

@aisk aisk commented Apr 27, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This extension module has some module state values which may be read or written in different threads:

  • field_limit: A simple integer value to store the maximum field size for the parser;
  • dialects: A dict value to store all the dialects by string names.

field_limit

There is a public function field_size_limit that can read and write it, using a critical section to protect it. For other internal usages, _Py_atomic_load_long and _Py_atomic_store_long are used to make it thread safe.

dialects

There are three public functions get_dialect, unregister_dialect and list_dialects that can access this field. For the first two functions, a critical section is used to protect it.

And the list_dialects function just returns the dict's keys with PyDict_Keys. I think its result is readonly and PyDict_Keys itself is thread safe, so we don't need to do anything.
Other internal usages of dialects involve accessing its value with get_dialect_from_registry, which simply calls PyDict_GetItemRef and returns the result. I think we don't need to change it too.

Update: I think these APIs is just call dict APIs which is already thread-safe, and there are no other state changes inside these APIs, and no intermediate variables need to protect in these functions, so we don't need add locks on them.

  • Issue: Audit all built-in modules for thread safety #116738

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
aisk added 2 commits April 27, 2024 17:57
@aisk
Make the filed_limit thread safe
eeba1c2
@aisk
Make dialects fields on module state thread safe
40e71a8
@bedevere-app bedevere-app Bot added the awaiting review label Apr 27, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Apr 27, 2024
Audit all built-in modules for thread safety #116738
Open
@aisk aisk added skip news and removed awaiting review labels Apr 27, 2024
corona10
corona10 reviewed Apr 27, 2024
View reviewed changes
Comment thread Modules/_csv.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

corona10
corona10 reviewed Apr 27, 2024
View reviewed changes
Comment thread Modules/_csv.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@Eclips4 Eclips4 added the topic-free-threading label Apr 27, 2024
aisk added 2 commits April 27, 2024 20:00
@aisk
Utilize FT_ATOMIC_LOAD_INT32
c77750f
@aisk
Remove critical section on dialects field
6fbcb9f
corona10
corona10 reviewed Apr 28, 2024
View reviewed changes
Comment thread Modules/_csv.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

corona10
corona10 reviewed Apr 28, 2024
View reviewed changes
Comment thread Modules/_csv.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

aisk and others added 7 commits April 29, 2024 23:06
@aisk
Totally revert the change to csv_register_dialect
60753f1
@aisk
Revert field_limit to long, and add atomic api for it
64b7d20
@aisk
Add _Py_atomic_load_long for other platforms
1de333e
@aisk
Merge branch 'main' into csv-thread-safe
7bbfd36
@aisk
Revert the type change to old_limit
0a41927
@aisk
Using atomic_store instead of critical_section
f413092
@aisk
Update clinic generated files
6c40b59
@aisk
aisk requested a review from colesbury May 17, 2024 09:46
aisk
aisk commented May 17, 2024
View reviewed changes
Comment thread Include/cpython/pyatomic_msc.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_csv.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@aisk
Apply suggestions from code review
a79cf88
corona10
corona10 approved these changes Jun 19, 2024
View reviewed changes

@corona10 corona10 left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Sorry for the late.

@colesbury Do you have any comments about this?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

aisk reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added the awaiting merge label Jun 19, 2024
colesbury
colesbury reviewed Jul 8, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the long delay in reviewing this. Two requests:

  • Let's change field_limit to a Py_ssize_t. That seems more appropriate here and avoids the need for adding new atomic bindings. I'd prefer to avoid adding additional atomic bindings because its more code to maintain and its easy to add subtle bugs in the atomic bindings. Additionally, field_limit is compared to field_len, which is already a Py_ssize_t.
  • Let's use relaxed atomics for reading and writing field_limit, e.g., FT_ATOMIC_STORE_SSIZE_RELAXED. Relaxed ordering is fine for these sorts of global settings.

For the data type change:

  • %ld -> %zd
  • PyLong_AsLong -> PyLong_AsSsize_t

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

aisk reacted with heart emoji
All reactions
  • ❤️ 1 reaction
aisk and others added 4 commits July 10, 2024 20:55
@aisk
Merge branch 'main' into csv-thread-safe
1f03de1
@aisk
Change field_limit to Py_ssize_t
3739c46
@aisk
Remove atomic functions for long type
850bf2a
@aisk
Fix types
bfbfc56
@aisk

aisk commented Jul 10, 2024

Copy link
Copy Markdown
Member Author

Sorry for the long delay in reviewing this. Two requests:

* Let's change `field_limit` to a `Py_ssize_t`. That seems more appropriate here and avoids the need for adding new atomic bindings. I'd prefer to avoid adding additional atomic bindings because its more code to maintain and its easy to add subtle bugs in the atomic bindings. Additionally, `field_limit` is compared to `field_len`, which is already a `Py_ssize_t`.

* Let's use relaxed atomics for reading and writing `field_limit`, e.g., `FT_ATOMIC_STORE_SSIZE_RELAXED`. Relaxed ordering is fine for these sorts of global settings.

For the data type change:

* `%ld` -> `%zd`

* `PyLong_AsLong` -> `PyLong_AsSsize_t`

Thanks for the review, updated!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Fidget-Spinner
Fidget-Spinner reviewed Jul 14, 2024
View reviewed changes
Comment thread Modules/_csv.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@aisk
Fix the return type of field_size_limit
d73b827
kumaraditya303
kumaraditya303 approved these changes Oct 11, 2024
View reviewed changes
@kumaraditya303 kumaraditya303 self-assigned this Oct 11, 2024
@kumaraditya303 kumaraditya303 added the needs backport to 3.13 only security fixes label Oct 11, 2024
@kumaraditya303
kumaraditya303 merged commit a00221e into python:main Oct 11, 2024
@miss-islington-app

miss-islington-app Bot commented Oct 11, 2024

Copy link
Copy Markdown

Thanks @aisk for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Oct 11, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 11, 2024
@aisk @miss-islington
pythongh-116738: Make _csv module thread-safe (pythonGH-118344)
fa1d5f9
(cherry picked from commit a00221e)

Co-authored-by: AN Long <[email protected]>
@bedevere-app

bedevere-app Bot commented Oct 11, 2024

Copy link
Copy Markdown

GH-125328 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Oct 11, 2024
kumaraditya303 pushed a commit that referenced this pull request Oct 11, 2024
@miss-islington @aisk
[3.13] gh-116738: Make _csv module thread-safe (GH-118344) (#125328)
f589513
gh-116738: Make `_csv` module thread-safe (GH-118344)
(cherry picked from commit a00221e)

Co-authored-by: AN Long <[email protected]>
@aisk
aisk deleted the csv-thread-safe branch October 12, 2024 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury left review comments
@Fidget-Spinner Fidget-Spinner Fidget-Spinner left review comments
@corona10 corona10 corona10 approved these changes
@kumaraditya303 kumaraditya303 kumaraditya303 approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@kumaraditya303 kumaraditya303

Labels

skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@aisk @colesbury @corona10 @Fidget-Spinner @kumaraditya303 @Eclips4
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.