GeezerVison is showing github.com — Decimal to int conversion with engineering notation costly despite failing · Issue #139957 · python/cpython
Decimal to int conversion with engineering notation costly despite failing #139957DescriptionBug report
Bug description:
Adding many digits to engineering notation causes cpython to work very hard trying to do the conversion, only for it to fail anyway.
>>> x = decimal.Decimal('1.0E+10000')
>>> int(x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limitBut the time it takes goes up when you add extra digits. It appears to be unbounded.
>>> timeit.timeit(setup="import decimal", stmt="x = decimal.Decimal('1.0E+10000'); i = int(x)", number=10)
0.027457250020233914
>>> timeit.timeit(setup="import decimal", stmt="x = decimal.Decimal('1.0E+100000'); i = int(x)", number=10)
1.8902547920006327The decimal to int conversion should shortcut to a failure when there is no possibility of generating a real number, e.g. by examining the length of the substring after + or - (in conjunction with sys.set_int_max_str_digits or maybe some other setting).
CPython versions tested on:
3.11, 3.12
Operating systems tested on:
macOS Activitypicnixz commented on Oct 11, 2025 mdickinson commented on Oct 11, 2025 Note that the conversion from Decimal succeeds. It's the attempt to print the resulting int at the prompt that fails.
Python 3.13.7 (main, Aug 15 2025, 23:01:14) [Clang 17.0.0 (clang-1700.0.13.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from decimal import Decimal
>>> x = Decimal('1e10000')
>>> i = int(x) # succeeds
>>> i # fails
Traceback (most recent call last):
File "<python-input-3>", line 1, in <module>
i
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limitSo this doesn't really have anything to do with Decimal. It's no different from trying to print 10**10000 directly. Or Fraction("1e10000"), for that matter.
>>> from fractions import Fraction
>>> f = Fraction('1e10000')
>>> i = int(f) # succeeds
>>> i # fails
Traceback (most recent call last):
File "<python-input-7>", line 1, in <module>
i # fails
^
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limitReacted by Sergey B Kirpichev Reacted by Tim Peters picnixz commented on Oct 11, 2025 Right. I always forget that we have an implicit repr(x) in the REPL. So, probably nothing to do here? tim-one commented on Oct 11, 2025 Although note that it's still not all unicorns and rainbows 😉.
>>> x = Decimal('1e1000000') # add more zeroes
>>> i = int(x) # succeeds, but takes "a long time"This is a consequence of that Decimal<->int conversions are still quadratic time. A report on that was open for years, but not long ago closed as "not planned". It would be best if (the external) libmpdec did the heavy lifting to use a sub-quadratic approach, but short of that nobody has been both willing and able to put "the smarts" into CPython's wrapping of such conversions. picnixz commented on Oct 11, 2025 I'm going to close this one as a duplicate of #90716. As Gregory said, we first need a good plan if we want to work on this. serhiy-storchaka commented on Oct 11, 2025 This returns almost instantly on my computer.
$ ./python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e1_000_000')" "int(x)"
1 loop, best of 1: 262 msec per loop
Adding more zeroes can making it taking "a long time".
$ ./python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e10_000_000')" "int(x)"
1 loop, best of 1: 10.4 sec per loop
String to integer conversion takes comparable time
$ ./python -X int_max_str_digits=0 -m timeit -n1 -r1 -s "from decimal import Decimal; x = '1'+'0'*10_000_000" "int(x)"
1 loop, best of 1: 13.6 sec per loop
so I do not think that we can optimize Decimal to integer conversion by utilizing Decimal to string to integer conversion. tim-one commented on Oct 11, 2025 @serhiy-storchaka, believe it or not, 10 million digits is still on the small side(!) for O() behavior of str->int to become apparent. Increase it to 100 million digits, and you should find that it takes a lot closer to 10 times (sub-quadratic) longer than 100 times .(quadratic). On my box, about 12 times longer.
$ py -X int_max_str_digits=0 -m timeit -n1 -r1 -s "x = '1'+'0'*10_000_000" "int(x)"
1 loop, best of 1: 10.1 sec per loop
$ py -X int_max_str_digits=0 -m timeit -n1 -r1 -s "x = '1'+'0'*100_000_000" "int(x)"
1 loop, best of 1: 119 sec per loop
Ironically enough, CPython's str->int uses decimal internally to exploit its advanced (quasi-linear) multiplication. mdickinson commented on Oct 12, 2025 @serhiy-storchaka Those Decimal timings are impressive. I get very different performance results on my M4 Pro laptop, off by a couple of orders of magnitude (60 times to 150 times slower than on your machine).
mdickinson@MacBook-Pro heron % python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e1_000_000')" "int(x)"
1 loop, best of 1: 15.4 sec per loop
mdickinson@MacBook-Pro heron % python -m timeit -n1 -r1 -s "from decimal import Decimal; x = Decimal('1e10_000_000')" "int(x)"
1 loop, best of 1: 1.53e+03 sec per loop
But str-to-int conversion is faster:
mdickinson@MacBook-Pro heron % python -X int_max_str_digits=0 -m timeit -n1 -r1 -s "from decimal import Decimal; x = '1'+'0'*1_000_000" "int(x)"
1 loop, best of 1: 155 msec per loop
mdickinson@MacBook-Pro heron % python -X int_max_str_digits=0 -m timeit -n1 -r1 -s "from decimal import Decimal; x = '1'+'0'*10_000_000" "int(x)"
1 loop, best of 1: 6.4 sec per loop
I'm not sure how to account for the extreme difference in timings for the Decimal conversions.
mdickinson@MacBook-Pro heron % python -VV
Python 3.13.8 (main, Oct 11 2025, 01:55:48) [Clang 17.0.0 (clang-1700.0.13.5)]
serhiy-storchaka commented on Oct 12, 2025 It seems that libmpdec is not provided on Ubuntu, so I tested pure Python implementation. Reacted by Mark Dickinson and Tim Peters skirpichev commented on Oct 12, 2025 From CPython CI:
# Workaround missing libmpdec-dev on ubuntu 24.04:
# https://launchpad.net/~ondrej/+archive/ubuntu/php
# https://deb.sury.org/
sudo add-apt-repository ppa:ondrej/php
apt-get update
apt-get -yq install libmpdec-dev
serhiy-storchaka commented on Oct 12, 2025 I would prefer not to use untrusted PPAs. tim-one commented on Oct 12, 2025
It seems that libmpdec is not provided on Ubuntu, so I tested pure Python implementation.
Ah - missed that. Ironically, the pure P:ython implementation can do Decimal->int much faster than when using libmpdec. In this case, offhand it looks like the pure Python run is timing mostly CPython's 10 ** someint speed, which benefits at least from sub-quadratic Karatsuba multiplication. There's no reason for why libmpdec couldn't do even better than that. But it doesn't now.
As mentioned, when libmpdec is available, Python's own str->int does exploit decimal's faster-than-Karatsuba multiplication.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|