GeezerVison is showing github.com — Python3.13 performance Issue with python.org macOS installers on ARM Macs · Issue #122580 · python/cpython
Python3.13 performance Issue with python.org macOS installers on ARM Macs #122580DescriptionBug description:
Python3.13rc1 seems to be about 20-25% slower than Python3.12.4.
Platform: M1 Mac running MacOS Sonoma 14.5
I first noticed the issue when running this repository where Python3.13rc1 ran on average 20-25% slower than Python3.12.4
With help from the Python discord server, we were able to create this minimal reproducible example:
from time import process_time
class Lexer:
def __init__(self, data):
self.data = data
self.index = 0
def detect(self, text):
i = 0
for char in text:
if self.peek(i) != char:
return False
i += 1
return True
def peek(self, amt=1):
if self.index + amt < len(self.data):
return self.data[self.index + amt]
else:
return None
text = "return text + indent * ' ' + '}'\n" * 200
needle = "return text + indent * ' ' + '}'\n" * 199 + "Nope\n"
lexer = Lexer(text)
start = process_time()
for x in range(10_000):
lexer.detect(needle)
print(process_time() - start)These were the results across many tests:
3.12.4: 7.742122s
3.13rc1: 8.309326s
The performance issue was not reproducible on other Operating Systems.
CPython versions tested on:
3.11.1, 3.12.4, 3.13
CPython Options:
For my testing, I used the official Python3.13rc1 installer from Python.org, which has the following configuration:
OPT:
3.13rc1: -DNDEBUG -g -O3 -Wall
3.12.4: -DNDEBUG -g -O3 -Wall
CONFIG_ARGS:
3.13rc1: '--enable-framework' '--with-framework-name=Python' '--enable-universalsdk=/' '--with-universal-archs=universal2' '--enable-optimizations' '--with-lto' '--without-ensurepip' '--with-system-libmpdec' '--with-openssl=/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local' 'LIBLZMA_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBLZMA_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -llzma' 'LIBMPDEC_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBMPDEC_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -lmpdec -lm' 'LIBSQLITE3_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBSQLITE3_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -lsqlite3' 'TCLTK_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'TCLTK_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -ltcl -ltk' 'CC=clang'
3.12.4: '-C' '--enable-framework' '--enable-universalsdk=/' '--with-universal-archs=universal2' '--with-computed-gotos' '--without-ensurepip' '--with-openssl=/tmp/_py/libraries/usr/local' '--enable-optimizations' '--with-lto' 'TCLTK_CFLAGS=-I/tmp/_py/libraries/usr/local/include' 'TCLTK_LIBS=-ltcl8.6 -ltk8.6' 'LDFLAGS=-g' 'CFLAGS=-g' 'CC=clang'
Operating systems tested on:
Linux, macOS, Windows Reacted by Hudson Gouge and devdanzin ActivityZeroIntensity commented on Aug 1, 2024 Me and @devdanzin have figured out that this is only on ARM, Windows and Linux x86 seemed to be faster based on the benchmarks. mdboom commented on Aug 1, 2024 Thanks for this report. Across a broad range of benchmarks measuring on the Faster CPython benchmarking machine, we see between a 3% and 8% speedup for 3.13.0b4 vs. 3.12.0 on a MacOS ARM machine.
That doesn't mean that this particular example hasn't found a narrow regression case. In our own benchmarking, some specific benchmarks are in fact slower, but only telco is that much slower (which is a known regression in the decimal module). Maybe some line profiling of 3.12 vs. 3.13 would be helpful here, or some pystats results.
I also noticed that you have --with-computed-gotos for 3.12, but not 3.13 here. That may not be relevant since it's on by default anyway (and I'm not sure it even applies for clang), but we should make sure all other variables are equal. mdboom commented on Aug 1, 2024 I just tried to reproduce the above result on our benchmarking M1 Mac, also running macOS 14.5, and got different results:
3.12.0: 7.471261
3.13.0rc1: 5.986742
I suspect this comes down to different configurations between the builds. It looks like 3.12.0 on your machine was built on a different machine? Try building each in exactly the same environment and see if you can still reproduce. hg0428 commented on Aug 1, 2024
I just tried to reproduce the above result on our benchmarking M1 Mac, also running macOS 14.5, and got different results:
3.12.0: 7.471261
3.13.0rc1: 5.986742
I suspect this comes down to different configurations between the builds. It looks like 3.12.0 on your machine was built on a different machine? Try building each in exactly the same environment and see if you can still reproduce.
I am using 3.12.4, not 3.12.0.
I have tested 3.11, 3.12.1, 3.12.3, all from Python.org at different points in time.
I have also tested many different betas of 3.13, and the results have been consistent.
All the versions installed on my machine are the default ones from Python.org. brandtbucher commented on Aug 1, 2024 I also can't reproduce with local optimized builds on an M2 Pro running Sonoma 14.5 (3 runs each):
v3.12.4: 5.718666, 6.035776, 5.577658
v3.13.0rc1: 4.933225, 5.214851, 4.837962
v3.13.0rc1 (JIT): 4.746776, 4.747096, 4.764923
However, I can reproduce with the python.org downloads:
3.12.4: 5.684782, 5.808816, 5.734691
3.13.0rc1: 6.673098, 6.930707, 7.369320
So the issue seems to be with the releases themselves. @ned-deily, anything different with macOS releases this time around? Reacted by Ned Deily, Peter Bierma, Hudson Gouge and Matt Wozniski brandtbucher commented on Aug 1, 2024 So, this is kinda strange. Consider this reproducer:
import time
s = "X" * 100_000_000
start = time.perf_counter()
for _ in s:
pass
duration = time.perf_counter() - start
print(f"global: {duration:.3f}s")
def foo():
for _ in s:
pass
start = time.perf_counter()
foo()
duration = time.perf_counter() - start
print(f"function: {duration:.3f}s")With the 3.12.4 python.org download, this prints:
global: 1.237s
function: 0.364s
On 3.13.0rc1:
global: 0.930s
function: 0.680s
So the loop over the string is faster at module level on 3.13, but faster at function level in 3.12. And 3.13 is faster on both if I replace "X" * 100_000_000 with range(100_000_000). brandtbucher commented on Aug 1, 2024 The issue could be some sort of messed-up (or skipped) PGO with the release build. The global version has lots of FOR_ITER -> STORE_NAME pairs while the function version has lots of FOR_ITER -> STORE_FAST pairs. That's really the only difference that I can see. I'd expect PGO to make the latter very fast, at the expense of the former.
But at this point I'm just guessing. mdboom commented on Aug 1, 2024
However, I can reproduce with the python.org downloads:
Just dittoing this. I agree, there may be something that changed in the build environment between releases. ned-deily commented on Aug 1, 2024 I can reproduce a similar slower performance between the python.org 3.12.4 and 3.13.0rc1 builds on an M3 MacBook Pro with macOS 14.6. However, using the same python.org installers on an Intel i9 iMac also with macOS 14.6 shows a performance improvement between 3.12.4 and 3.13.0rc1. These are both universal2 builds meaning they should be running native arm or x86_64 code on each. And, interestingly, if I force running the universal2 builds in x86_64 mode using Rosetta2 emulation on the same M3 (by using python3.1x-intel64), I see performance differences similar to running natively on the Intel Mac, that is 3.13.0rc1 is somewhat faster.
There are a number of differences in the build environments used for 3.12.4 and for 3.13.0rc1, including the compiler versions and deployment targets. I agree that something with optimizations seems a likely culprit. I will dig into it and report back.
Thanks for the report and the simple reproducer, @hg0428! Reacted by Brandt Bucher, devdanzin, Michael Droettboom and Hudson Gouge Reacted by Michael Droettboom ned-deily commented on Aug 2, 2024 A quick update: I've run the test case on the full range of python.org macOS installers from 3.13.0a5 through 3.13.0rc1 on both Apple Silicon (M3, arm64) and Intel (i9) Macs and it appears that rc1 on Apple Silicon is the outlier. I also ran the test case on all of the macOS installers where we also provided a free-threading build, that is, 3.13.0b2 through 3.13.0rc1, again, on both CPU types; there, the test-case performance differences roughly mirror those of the the normal (GIL) builds except that the rc1 free-threading version does not exhibit the performance degradation that the normal version does on Apple Silicon. So, the good news at this point is that this particular performance issue appears to be isolated but there is more work to be done to pin it down further. More tomorrow. ned-deily commented on Sep 3, 2024 Extensive testing with the microbenchmarks provided in this issue and in #122832 confirm that (1) there was a significant performance degradation for at least some test cases with the 3.13.0rc1 provided by the python.org installer particularly on Apple Silicon Macs and that (2) that performance degradation appears to be due to the use of the ThinLTO (Link Time Optimization) Python build option (--with-lto defaulting to --with-lto=thin) with recent Apple-supplied LLVM/Clang build tools. As noted earlier in this issue, the degradation was seen with very simple builds of the test case(s) when --with-lto was included. It was subsequently determined through extensive testing that using the original monolithic LTO build option (--with-lto=full) provides a substantial performance boost for these two benchmarks and, in all test configurations shown here, the three build configurations using --with-lto=full always provided the best performance with these microbenchmarks, in some cases, dramatically better.
Since performance can vary greatly due to many factors, it is unreasonable to expect the same level of improvement in more extensive real-world benchmarks but it seems likely that an overall improvement of some magnitude may be seen with most benchmarks. From reading a bit about ThinLTO, it seems one of its biggest advantages was expected to be providing faster incremental builds rather than doing the monolithic optimization for each build. For the python.org installer case, that isn't an advantage since we always build the binary components of the installer from scratch for each release. And the documentation implies that, in most cases, the monolithic LTO would be expected to provide better run-time performance anyway. So all of that seems to indicate that full LTO is the right choice for the installer build in any case.
Moving forward, with the imminent release of 3.13.0rc2, we will use --with-lto=full to build the macOS installer for it and will run these microbenchmarks again before release. It would be great if more extensive benchmarks could be run with the released installer and results reported here to this issue. I will leave it open for that purpose for the immediate future.
While there are still unanswered questions about exactly what it is about ThinLTO that causes the performance degradation in these environments, I am not planning to delve deeper into it since it's not something I have particular expertise in and it seems that using full LTO is a perfectly satisfactory solution for the macOS installer. Feel free to dive in yourself!
Thanks again to everyone who has helped out with reporting and investigation this!
Appended here is a summary of the more pertinent test results while investigating this issue:
Test Cases
Apple Silicon (MacBook Pro 14-inch, Nov 2023, M3, 16GB)
Summary
test_122580 (version = 3.13.0rc1+-xcode) ran
1.00 ± 0.02 times faster than test_122580 (version = 3.13.0rc1-lto-full)
1.04 ± 0.02 times faster than test_122580 (version = 3.13.0rc1+-xcode-beta)
1.11 ± 0.04 times faster than test_122580 (version = 3.13.0rc1-no-lto)
1.16 ± 0.02 times faster than test_122580 (version = 3.12.5)
1.17 ± 0.04 times faster than test_122580 (version = 3.13.0b4)
1.18 ± 0.02 times faster than test_122580 (version = 3.12.4)
1.20 ± 0.03 times faster than test_122580 (version = 3.13.0b1)
1.61 ± 0.04 times faster than test_122580 (version = 3.13.0rc1)
test_122832 (version = 3.13.0rc1+-xcode-beta) ran
1.02 ± 0.02 times faster than test_122832 (version = 3.13.0rc1-lto-full)
1.03 ± 0.01 times faster than test_122832 (version = 3.13.0rc1+-xcode)
1.07 ± 0.01 times faster than test_122832 (version = 3.12.4)
1.08 ± 0.01 times faster than test_122832 (version = 3.12.5)
1.12 ± 0.03 times faster than test_122832 (version = 3.13.0b4)
1.17 ± 0.01 times faster than test_122832 (version = 3.13.0rc1-no-lto)
1.28 ± 0.02 times faster than test_122832 (version = 3.13.0b1)
1.79 ± 0.02 times faster than test_122832 (version = 3.13.0rc1)
| Command |
Mean [s] |
Min [s] |
Max [s] |
Relative |
test_122580 (version = 3.12.4) |
4.851 ± 0.039 |
4.797 |
4.894 |
1.18 ± 0.02 |
test_122580 (version = 3.12.5) |
4.778 ± 0.030 |
4.739 |
4.813 |
1.16 ± 0.02 |
test_122580 (version = 3.13.0b1) |
4.954 ± 0.089 |
4.822 |
5.053 |
1.20 ± 0.03 |
test_122580 (version = 3.13.0b4) |
4.840 ± 0.141 |
4.673 |
4.954 |
1.17 ± 0.04 |
test_122580 (version = 3.13.0rc1) |
6.627 ± 0.138 |
6.403 |
6.751 |
1.61 ± 0.04 |
test_122580 (version = 3.13.0rc1-no-lto) |
4.577 ± 0.142 |
4.385 |
4.740 |
1.11 ± 0.04 |
test_122580 (version = 3.13.0rc1-lto-full) |
4.128 ± 0.076 |
4.032 |
4.225 |
1.00 ± 0.02 |
test_122580 (version = 3.13.0rc1+-xcode) |
4.121 ± 0.066 |
4.006 |
4.169 |
1.00 |
test_122580 (version = 3.13.0rc1+-xcode-beta) |
4.304 ± 0.073 |
4.199 |
4.372 |
1.04 ± 0.02 |
| Command |
Mean [s] |
Min [s] |
Max [s] |
Relative |
test_122832 (version = 3.12.4) |
16.645 ± 0.059 |
16.588 |
16.744 |
1.07 ± 0.01 |
test_122832 (version = 3.12.5) |
16.843 ± 0.064 |
16.772 |
16.919 |
1.08 ± 0.01 |
test_122832 (version = 3.13.0b1) |
19.956 ± 0.248 |
19.749 |
20.364 |
1.28 ± 0.02 |
test_122832 (version = 3.13.0b4) |
17.402 ± 0.370 |
17.173 |
18.058 |
1.12 ± 0.03 |
test_122832 (version = 3.13.0rc1) |
27.907 ± 0.206 |
27.669 |
28.202 |
1.79 ± 0.02 |
test_122832 (version = 3.13.0rc1-no-lto) |
18.313 ± 0.161 |
18.128 |
18.558 |
1.17 ± 0.01 |
test_122832 (version = 3.13.0rc1-lto-full) |
15.988 ± 0.195 |
15.834 |
16.294 |
1.02 ± 0.02 |
test_122832 (version = 3.13.0rc1+-xcode) |
16.057 ± 0.075 |
15.960 |
16.160 |
1.03 ± 0.01 |
test_122832 (version = 3.13.0rc1+-xcode-beta) |
15.599 ± 0.139 |
15.355 |
15.692 |
1.00 |
Intel (iMac Retina 5K 27-inch, 2019, 3.6 GHz 8-Core Intel Core i9, 40GB)
Summary
test_122580 (version = 3.13.0rc1+-xcode-beta) ran
1.08 ± 0.03 times faster than test_122580 (version = 3.13.0rc1+-xcode)
1.08 ± 0.02 times faster than test_122580 (version = 3.13.0rc1-lto-full)
1.19 ± 0.02 times faster than test_122580 (version = 3.13.0rc1-no-lto)
1.20 ± 0.02 times faster than test_122580 (version = 3.13.0b4)
1.24 ± 0.02 times faster than test_122580 (version = 3.13.0rc1)
1.43 ± 0.08 times faster than test_122580 (version = 3.13.0b1)
1.46 ± 0.04 times faster than test_122580 (version = 3.12.4)
1.47 ± 0.03 times faster than test_122580 (version = 3.12.5)
test_122832 (version = 3.13.0rc1+-xcode-beta) ran
1.08 ± 0.04 times faster than test_122832 (version = 3.13.0rc1+-xcode)
1.11 ± 0.02 times faster than test_122832 (version = 3.13.0rc1-lto-full)
1.13 ± 0.01 times faster than test_122832 (version = 3.12.5)
1.13 ± 0.01 times faster than test_122832 (version = 3.12.4)
1.16 ± 0.02 times faster than test_122832 (version = 3.13.0rc1-no-lto)
1.20 ± 0.02 times faster than test_122832 (version = 3.13.0b4)
1.26 ± 0.05 times faster than test_122832 (version = 3.13.0rc1)
1.35 ± 0.02 times faster than test_122832 (version = 3.13.0b1)
| Command |
Mean [s] |
Min [s] |
Max [s] |
Relative |
test_122580 (version = 3.12.4) |
10.535 ± 0.193 |
10.293 |
10.765 |
1.46 ± 0.04 |
test_122580 (version = 3.12.5) |
10.630 ± 0.088 |
10.540 |
10.753 |
1.47 ± 0.03 |
test_122580 (version = 3.13.0b1) |
10.328 ± 0.563 |
9.811 |
10.947 |
1.43 ± 0.08 |
test_122580 (version = 3.13.0b4) |
8.681 ± 0.093 |
8.606 |
8.832 |
1.20 ± 0.02 |
test_122580 (version = 3.13.0rc1) |
8.943 ± 0.060 |
8.867 |
9.013 |
1.24 ± 0.02 |
test_122580 (version = 3.13.0rc1-no-lto) |
8.576 ± 0.055 |
8.497 |
8.631 |
1.19 ± 0.02 |
test_122580 (version = 3.13.0rc1-lto-full) |
7.777 ± 0.130 |
7.613 |
7.905 |
1.08 ± 0.02 |
test_122580 (version = 3.13.0rc1+-xcode) |
7.772 ± 0.136 |
7.655 |
8.003 |
1.08 ± 0.03 |
test_122580 (version = 3.13.0rc1+-xcode-beta) |
7.210 ± 0.113 |
7.088 |
7.359 |
1.00 |
| Command |
Mean [s] |
Min [s] |
Max [s] |
Relative |
test_122832 (version = 3.12.4) |
33.019 ± 0.182 |
32.798 |
33.217 |
1.13 ± 0.01 |
test_122832 (version = 3.12.5) |
32.950 ± 0.186 |
32.651 |
33.158 |
1.13 ± 0.01 |
test_122832 (version = 3.13.0b1) |
39.386 ± 0.470 |
38.945 |
39.939 |
1.35 ± 0.02 |
test_122832 (version = 3.13.0b4) |
34.851 ± 0.509 |
34.294 |
35.525 |
1.20 ± 0.02 |
test_122832 (version = 3.13.0rc1) |
36.653 ± 1.359 |
35.594 |
39.035 |
1.26 ± 0.05 |
test_122832 (version = 3.13.0rc1-no-lto) |
33.851 ± 0.478 |
33.435 |
34.554 |
1.16 ± 0.02 |
test_122832 (version = 3.13.0rc1-lto-full) |
32.255 ± 0.505 |
31.864 |
33.100 |
1.11 ± 0.02 |
test_122832 (version = 3.13.0rc1+-xcode) |
31.530 ± 1.057 |
30.654 |
32.988 |
1.08 ± 0.04 |
test_122832 (version = 3.13.0rc1+-xcode-beta) |
29.104 ± 0.263 |
28.850 |
29.499 |
1.00 |
Build configurations
| Python Version |
Apple Clang |
Build System |
Deployment Target |
--with-lto= |
| 3.12.4 |
clang-1300.0.29.30 |
x86_64-apple-darwin20.6.0 |
10.9 |
thin |
| 3.12.5 |
clang-1300.0.29.30 |
x86_64-apple-darwin20.6.0 |
10.9 |
thin |
| 3.13.0b1 |
clang-1300.0.29.30 |
x86_64-apple-darwin20.6.0 |
10.9 |
thin |
| 3.13.0b4 |
clang-1500.3.9.4 |
aarch64-apple-darwin23.5.0 |
10.13 |
thin |
| 3.13.0rc1 |
clang-1500.3.9.4 |
aarch64-apple-darwin23.5.0 |
10.13 |
thin |
| 3.13.0rc1-no-lto |
clang-1500.3.9.4 |
aarch64-apple-darwin23.6.0 |
10.13 |
no |
| 3.13.0rc1-lto-full |
clang-1500.3.9.4 |
aarch64-apple-darwin23.6.0 |
10.13 |
full |
| 3.13.0rc1+-xcode |
clang-1500.3.9.4 |
aarch64-apple-darwin23.6.0 |
10.13 |
full |
| 3.13.0rc1+-xcode-beta |
clang beta |
aarch64-apple-darwin23.6.0 |
10.13 |
full |
Reacted by Hugo van Kemenade and Erlend E. Aasland Reacted by Itamar Oren and Erlend E. Aasland Reacted by Erlend E. Aasland mdboom commented on Sep 3, 2024 Thanks for all your hard work on this, @ned-deily. This finding seems reasonable. I've put the rc2 release date on my calendar and will run it against the full pyperformance suite then to confirm. brandtbucher commented on Sep 3, 2024 Thanks @ned-deily. I wonder, should the default for --with-lto be full instead of thin? I just assumed that was already the case (and I'm not sure what our benchmarking infrastructure assumes).
In general, if I'm passing --with-lto, I'm expecting the build to take a while (and be as fast as possible in return). Reacted by Matt Wozniski, Hugo van Kemenade and Erlend E. Aasland itamaro commented on Sep 3, 2024 What's the default LTO (full or thin) for the Linux builds? mdboom commented on Sep 3, 2024 IIUC, thin LTO is only supported on llvm/clang, so our gcc-based Linux builds don't use it. Reacted by Erlend E. Aasland and Itamar Oren ned-deily commented on Sep 3, 2024 The default for --with-lto is platform-dependent and compiler-dependent. configure.ac has a bunch of code to deal with it (see, for example, starting here). There has also been a fair amount of discussion about lto options (for example, here and here). If somebody wants to revisit that, it would be better to do it in another issue; I'd like to keep this focused on the macOS installer. Reacted by Hugo van Kemenade and Itamar Oren changed the title [-]Python3.13 Performance Issue on MacOS ARM[/-] [+]Python3.13 performance Issue with python.org macOS installers on ARM Macs[/+] on Sep 3, 2024rhettinger commented on Sep 7, 2024 Just tested 3.13rc2 and the performance issues are solved. Thank you. Reacted by Ned Deily and Hugo van Kemenade mdboom commented on Sep 9, 2024
Just tested 3.13rc2 and the performance issues are solved. Thank you.
Same (on our usual macOS lab machine). Reacted by Ned Deily and Hugo van Kemenade itamaro commented on Sep 9, 2024 closing as fixed, thanks for testing @rhettinger & @mdboom !
please re-open if there's anything left to do here. Reacted by Hugo van Kemenade and Peter Bierma ned-deily commented on Oct 1, 2024
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|