GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-111962: Make dtoa thread-safe in `--disable-gil` builds. by colesbury · Pull Request #112049 · python/cpython

Conversation

@colesbury

@colesbury colesbury commented Nov 13, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This avoids using the Bigint free-list in --disable-gil builds and pre-computes the needed powers of 5 during interpreter initialization.

  • Issue: dtoa: thread safety in --disable-gil builds #111962

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@colesbury colesbury added 3.13 only security fixes topic-free-threading labels Nov 13, 2023
@colesbury
colesbury requested a review from mdickinson November 13, 2023 20:39
@bedevere-app bedevere-app Bot added the awaiting review label Nov 13, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Nov 13, 2023
dtoa: thread safety in --disable-gil builds #111962
Closed
@colesbury colesbury added the skip news label Nov 13, 2023
@colesbury
colesbury marked this pull request as draft November 13, 2023 21:26
@bedevere-app bedevere-app Bot removed the awaiting review label Nov 13, 2023
@colesbury
colesbury marked this pull request as ready for review November 13, 2023 22:39
@bedevere-app bedevere-app Bot added the awaiting review label Nov 13, 2023
colesbury added 2 commits November 20, 2023 10:43
@colesbury
pythongh-111962: Make dtoa thread-safe in --disable-gil builds.
d53c7f4
This avoids using the Bigint free-list in `--disable-gil` builds
and pre-computes the needed powers of 5 during interpreter initialization.
@colesbury
Fix size of cached powers of 5 array.
4de2fb8
We need the powers of 5 up to 5**512 because we only jump straight to
underflow when the exponent is less than -512 (or larger than 308).
@colesbury
colesbury force-pushed the dtoa-thread-safety branch from 4fd7747 to 4de2fb8 Compare November 20, 2023 15:43
@colesbury
Rename Py_NOGIL to Py_GIL_DISABLED
893f264
mdickinson
mdickinson approved these changes Nov 26, 2023
View reviewed changes

@mdickinson mdickinson left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. A couple of naming and comment nitpicks, and a request for a comment justifying the assert(k < (1 << (Bigint_Pow5max))); assertion.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Include/internal/pycore_dtoa.h Outdated
#define Bigint_Kmax 7

/* The size of the cached powers of 5 array */
#define Bigint_Pow5max 8

@mdickinson mdickinson Nov 26, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I'm wondering whether there's a better name here; the "max" part seems potentially confusing (the max value stored here would be 5**2**9, right?). Bigint_Pow5count? Bigint_Pow5size?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@colesbury colesbury Nov 26, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed it to Bigint_Pow5size

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mdickinson reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Include/internal/pycore_dtoa.h Outdated

struct _dtoa_state {
/* p5s is a linked list of powers of 5 of the form 5**(2**i), i >= 2 */
/* p5s is an array of powers of 5 of the form 5**(2**i), i >= 2 */

@mdickinson mdickinson Nov 26, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Could we adjust this comment to be more explicit about the indexing - i.e., make it clear that p5s[i] is 5**(2**(i+2)) for 0 <= i < Bigint_Pow5max?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

colesbury reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Include/internal/pycore_dtoa.h
#endif // !Py_USING_MEMORY_DEBUGGER


/* These functions are used by modules compiled as C extension like math:

@mdickinson mdickinson Nov 26, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted because the comment is out of date, presumably? 👍

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@colesbury colesbury Nov 26, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, these comments were about why the below functions used PyAPI_FUNC (were "exported"), but the symbols are no longer exported.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/dtoa.c Outdated

if (!(k >>= 2))
return b;
assert(k < (1 << (Bigint_Pow5max)));

@mdickinson mdickinson Nov 26, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This single line is by far the hardest to review, and reminds me yet again why I'd be delighted to see dtoa.c go away entirely. :-)

This says that pow5mult will never be called with k >= 1024, right?

I do believe that there's an upper limit on possible k values here, based on:

  • For double-to-string conversion, our input space is naturally bounded, and since we're assuming IEEE 754 binary64 format, we have at most 767 significant decimal digits in any output (an example worst case is something like (2**53 - 1) * 2**-1074), and dtoa.c is clever enough to pad with zeros when requested (e.g., with something crazy like format(math.pi, '.2000f')) rather than try to compute those zeros.
  • For string-to-double conversion, the input string has a potentially unbounded number of decimal digits, all of which may need to be taken into account for correct-rounding corner cases, but in those corner cases dtoa.c (in bigcomp) takes the approach of computing decimal digits of binary64 tie values and comparing with the decimal input, rather than trying to convert the decimal input to bigint-land (which would potentially require huge powers of 5).

And based on the above, 1023 certainly seems plausible as an upper bound for possible k values. I'd be hard pushed to give a proof that k can never exceed 1023, though. (At least, not without spending a lot more time than I currently have available.)

@colesbury Does the above roughly match your reasoning? Would it be possible to add a comment to the code justifying the assertion?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@colesbury colesbury Nov 27, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a comment and moved the assertion up. The limits are related to the maximum base-10 exponent. For double-to-string, that's DBL_MAX_10_EXP (308). As you say, we set the limits ourselves for string-to-double, which are e=308 for overflow and e=-512 for underflow. But our exponent can be adjusted based on the number of digits, so we can see values as larges as k=535 float('1' +'0'*38 + '1E-535'), where the limits is from the combination of 512+STRTOD_DIGLIM-DBL_DIG-1.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mdickinson reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Nov 26, 2023
colesbury added 2 commits November 27, 2023 10:49
@colesbury
Changes from review
4907b59
@colesbury
Fix assertion placement
d9536e5
mdickinson
mdickinson approved these changes Nov 27, 2023
View reviewed changes

@mdickinson mdickinson left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates! LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@colesbury

colesbury commented Dec 5, 2023

Copy link
Copy Markdown
Contributor Author

@mdickinson - would you please merge this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mdickinson

mdickinson commented Dec 7, 2023

Copy link
Copy Markdown
Member

@colesbury Apologies; I'd been assuming you'd merge it (and was beginning to wonder why you hadn't). :-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mdickinson
mdickinson merged commit 2d76be2 into python:main Dec 7, 2023
@bedevere-app bedevere-app Bot removed the awaiting merge label Dec 7, 2023
@Eclips4

Eclips4 commented Dec 7, 2023

Copy link
Copy Markdown
Member

@colesbury Apologies; I'd been assuming you'd merge it (and was beginning to wonder why you hadn't). :-)

Probably Sam doesn't have rights to merge

mdickinson reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@colesbury
colesbury deleted the dtoa-thread-safety branch December 12, 2023 19:36
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@colesbury @aisk
pythongh-111962: Make dtoa thread-safe in --disable-gil builds. (py…
d202c31
…thon#112049)

This updates `dtoa.c` to avoid using the Bigint free-list in --disable-gil builds and
to pre-computes the needed powers of 5 during interpreter initialization.

* pythongh-111962: Make dtoa thread-safe in `--disable-gil` builds.

This avoids using the Bigint free-list in `--disable-gil` builds
and pre-computes the needed powers of 5 during interpreter initialization.

* Fix size of cached powers of 5 array.

We need the powers of 5 up to 5**512 because we only jump straight to
underflow when the exponent is less than -512 (or larger than 308).

* Rename Py_NOGIL to Py_GIL_DISABLED

* Changes from review

* Fix assertion placement
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@colesbury @Glyphack
pythongh-111962: Make dtoa thread-safe in --disable-gil builds. (py…
d4301c6
…thon#112049)

This updates `dtoa.c` to avoid using the Bigint free-list in --disable-gil builds and
to pre-computes the needed powers of 5 during interpreter initialization.

* pythongh-111962: Make dtoa thread-safe in `--disable-gil` builds.

This avoids using the Bigint free-list in `--disable-gil` builds
and pre-computes the needed powers of 5 during interpreter initialization.

* Fix size of cached powers of 5 array.

We need the powers of 5 up to 5**512 because we only jump straight to
underflow when the exponent is less than -512 (or larger than 308).

* Rename Py_NOGIL to Py_GIL_DISABLED

* Changes from review

* Fix assertion placement
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@mdickinson mdickinson mdickinson approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

3.13 only security fixes skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@colesbury @mdickinson @Eclips4
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.