Description

diegorusso

opened on Oct 16, 2026

Issue body actions

Update: The results below used Release builds without PGO or LTCG. See the [PGO+LTCG follow-up](#159521 (comment) for updated results and the maintained Gist for reproduction instructions.

While testing and measuring the tail-call interpreter on Windows ARM64, with the goal of enabling it by default on WoA, I noticed a performance change across commit ad1513a, "Reduce the work done to spill and reload the stack around calls".
The change avoids redundant stores and reloads of the interpreter's stack pointer around calls, keeping that pointer live across calls and increasing register pressure. Changing the compiler from MSVC 14.51 to 14.52 also affects performance.

I wanted to separate these effects before quoting a tail-call speedup.
The detailed results below are native ARM64 on a Windows Dev Kit 2023, pinned to performance-class CPU.

In terms of commits, both revisions are 3.16.0a0:

• "before" means the commit's parent, d47c27e
• "after" means ad1513a.

1. Effect of the commit and compiler version on the switch interpreter

Put all four switch based builds on one common scale: before-commit MSVC 14.51 = 100. Higher means faster.

Source revision	MSVC 14.51 speed index	MSVC 14.52 speed index
Before the commit	100.00	100.43
After the commit	92.49	96.80

Before the commit, the compilers produce almost equally fast builds. After the commit, both switch builds slow down: by 7.51% with MSVC 14.51 and 3.62% with MSVC 14.52, each relative to its own before-commit build.
MSVC 14.52 partly mitigates the switch interpreter regression introduced by the commit. Its larger lead after the commit reflects losing less performance across that change. We have measured this interaction; we have not yet established which difference in generated machine code explains it.

Emulated x64. Testing x64 binaries under Windows emulation on the same ARM64 device also shows a switch slowdown across the commit with both compilers. In those runs, 14.52 does not provide the mitigation observed on native ARM64.

The overall result varies across individual benchmarks: asyncio_websockets shows a measured speed loss of about 56% with 14.52 on both revisions.

Most affected benchmarks when changing the compiler: 14.52 versus 14.51

Each row below compares the switch build made with MSVC 14.52 against the switch build made with MSVC 14.51 at the same source revision. Positive means faster; negative means slower. † marks high variability (coefficient of variation above 10% in either compared build).

Source revision	Largest measured gains	Largest measured losses
Before the commit	regex_effbot (+6.87%); base85_large (+5.88%); nqueens (+5.21%)	asyncio_websockets (-55.85%); tornado_http† (-33.75%); ascii85_large (-4.71%)
After the commit	generators (+30.37%); unpack_sequence (+28.93%); nbody (+16.72%)	asyncio_websockets (-55.68%); tornado_http† (-22.28%); unpickle_list (-4.45%)

Most affected benchmarks across the commit: after versus before

Each row below compares the after-commit switch build against its before-commit switch build, keeping the compiler unchanged. Positive means faster; negative means slower. † marks high variability (coefficient of variation above 10% in either compared build).

Compiler	Largest measured gains	Largest measured losses
MSVC 14.51	python_startup_no_site† (+1.65%); gc_traversal (+0.93%); regex_dna (+0.45%)	generators (-29.42%); tornado_http† (-25.86%); nbody (-25.79%)
MSVC 14.52	unpack_sequence (+2.76%); create_gc_cycles (+0.88%); gc_traversal (+0.59%)	spectral_norm (-17.16%); nbody (-15.87%); tornado_http† (-13.03%)

2. Tail-call versus switch: MSVC 14.52

Change only the interpreter. Both builds use MSVC 14.52. For each revision, the baseline is the switch build at that same revision.

Source revision	Tail-call build, compared with switch build using the same source and compiler
Before the commit	4.43% faster
After the commit	9.36% faster

Tail-call is faster than switch on both revisions: by 4.43% before the commit and 9.36% after it. These are two comparisons against the switch build at the corresponding revision. The previous table shows that the commit slows switch, which contributes to the larger after-commit tail-call advantage.
A claim of "9.36% faster" should therefore specify the MSVC 14.52 after-commit switch baseline. This is context for reporting relative speedups whilst keeping the goal to enabling tail-call by default on WoA.

Emulated x64. With MSVC 14.52 under x64 emulation, tail-call's advantage over switch at the same revision grows from 11.47% before the commit to 17.43% afterwards, mainly because switch slows down while tail-call speed changes little. These are relative comparisons within emulation; the percentages need confirmation on native x64 hardware.

Most affected benchmarks: three largest gains and losses per comparison

These use the same baseline as the table above. Positive means faster; negative means slower. † marks high variability (coefficient of variation above 10% in either compared build).

Source revision	Largest measured gains	Largest measured losses
Before the commit	tornado_http† (+55.63%); hexiom (+19.41%); deltablue (+16.50%)	dulwich_log (-0.17%); base85_large (-0.08%); ascii85_large (-0.07%)
After the commit	tornado_http† (+112.05%); spectral_norm (+30.13%); nbody (+28.81%)	pathlib† (-7.23%); asyncio_tcp (-2.25%); gc_traversal (-0.75%)

The tail-call interpreter should be enabled by default on Windows on Arm. Speedup claims should be precise and explicitly state the baseline, including the CPython revision, compiler version, and interpreter configuration.

I'd also welcome someone repeating these experiments on native x86-64 (x64) hardware running Windows, as I don't currently have access to such a machine. :)

The scripts and current reproduction instructions are maintained in the CPython Windows ARM64 PGO benchmark Gist. Download its four files together. To reproduce the PGO+LTCG follow-up, use the command there with --pgo; omitting --pgo selects the original non-PGO/no-LTCG configuration.

Activity

1. diegorusso

   added

   performancePerformance or resource usagePerformance or resource usage

   OS-windows

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Oct 16, 2026
2. diegorusso commented on Oct 16, 2026

   

   diegorusso

   on Oct 16, 2026

   ContributorAuthor

   More actions

   Some feedback I received: we should build and test PGO builds. The above results are without it. I've put the scripts in a gist for now that should address the PGO and the plots (everyone likes plots).

   Here's the link: https://gist.github.com/diegorusso/3a0cfca565ad919ff094c88778cc17bd

   I'm re-running the benchmarks again and update the above test with the new results.
3. kevinprimeraquecreo-stack

   added this to Project_VKand Project_VKon Oct 18, 2026
4. github-project-automation

   moved this to Todo in Project_VKon Oct 18, 2026
5. diegorusso commented on Oct 20, 2026

   

   diegorusso

   on Oct 20, 2026

   ContributorAuthor

   More actions

   PGO+LTCG follow-up: native Windows ARM64

   Following the build-configuration feedback, I repeated the six ARM64 configurations with PGO+LTCG, keeping the source revisions, compiler toolsets and Dev Kit CPU affinity unchanged from the original experiment. This repeat covers ARM64 only.

   PGO+LTCG overview: switch performance across the commit and same-revision tail-call speedups
   

   1. Effect of the commit and compiler version on the switch interpreter

   The two speed-index columns share one baseline: before-commit MSVC 14.51 switch = 100. Higher means faster. The final column compares compilers at the same revision, using that row's 14.51 switch build as its baseline.

   Source revision	MSVC 14.51 speed index	MSVC 14.52 speed index	14.52 versus 14.51 at this revision
   Before the commit	100.00	101.60	+1.60%
   After the commit	89.76	97.83	+9.00%

   Across the commit, switch performance falls by 10.24% with 14.51 and 3.71% with 14.52, each relative to its own compiler's before-commit switch build. The compiler's lead therefore grows from 1.60% to 9.00%. The switch slowdown and its partial mitigation by 14.52 persist with PGO+LTCG.

   Most affected benchmarks: compiler and commit comparisons

   Each comparison names its baseline below. † marks a coefficient of variation above 10% in either compared build. These are ranked observed changes, not claims of statistical significance.

   For compiler comparisons, the baseline is the same-revision 14.51 switch build, and the changed build uses 14.52.

   Revision	Largest measured gains	Largest measured losses
   Before	urlsafe_base64_small (+25.43%); base64_large (+23.71%); base16_small (+23.25%)	json_loads (−5.88%); scimark_fft (−3.61%); asyncio_tcp (−3.34%)
   After	tornado_http† (+64.84%); generators (+29.81%); richards (+24.28%)	unpickle_list (−8.29%); pathlib† (−7.36%); json_loads (−3.10%)

   For commit comparisons, the baseline is the same compiler's before-commit switch build, and the changed build is after the commit.

   Compiler	Largest measured gains	Largest measured losses
   14.51	base64_large (+24.28%); base32_large (+22.59%); base16_large (+18.90%)	tornado_http† (−36.20%); generators (−32.66%); logging_silent (−26.66%)
   14.52	asyncio_tcp (+10.28%); tornado_http† (+3.25%); connected_components (+1.60%)	logging_silent (−14.19%); generators (−14.00%); richards (−11.39%)

   2. Tail-call versus switch with MSVC 14.52

   For each row, the baseline is the PGO+LTCG switch build at that same revision, using the same 14.52 toolset. Only interpreter selection changes.

   Source revision	Tail-call speedup over the matching switch build
   Before the commit	+5.25%
   After the commit	+11.56%

   The larger after-commit advantage combines the switch build's 3.71% loss with the tail-call build's own 2.06% gain across the commit. These effects combine multiplicatively; 11.56% compares interpreter modes at the after revision.

   Most affected benchmarks when enabling tail-call

   The baseline is the same-revision 14.52 PGO+LTCG switch build; the changed build enables tail-call. † has the same variability definition as above.

   Revision	Largest measured gains	Largest measured losses
   Before	coroutines (+21.04%); tomli_loads (+21.01%); tornado_http† (+20.21%)	html5lib† (−18.87%); json_dumps (−12.21%); bench_mp_pool† (−11.39%)
   After	nbody (+35.69%); coroutines (+35.17%); generators (+34.12%)	asyncio_tcp (−4.06%); regex_dna (−1.31%); base16_large (−1.10%)

   The following bench_runner plots compare tail-call with the same-revision 14.52 switch baseline. Ratios above 1 mean faster. Red violins identify interpreter-heavy benchmarks. The violins use bench_runner's significance tests and outlier handling; the ALL row pools displayed ratios. The full-suite geomean in each title is the 121-measurement metric used in the tables above.

   Before the commit:

   Before the commit: MSVC 14.52 PGO+LTCG tail-call versus switch, with the switch build as baseline
   

   After the commit:

   After the commit: MSVC 14.52 PGO+LTCG tail-call versus switch, with the switch build as baseline.
   

   The maintained Gist contains the reproduction command; use --architectures ARM64 --pgo for this repeat.

   Overall summary

   Moving to PGO+LTCG changes the pattern of benchmark gains and losses, while preserving the main findings: the commit still slows the switch interpreter, and MSVC 14.52 partly mitigates that slowdown. In these runs, tail-call's advantage over the matching 14.52 switch build increases from 4.39% to 5.25% before the commit, and from 9.32% to 11.56% after it.

   Reacted by Chris Eibl

   Reacted by Chris Eibl
6. chris-eibl commented on Oct 20, 2026

   

   chris-eibl

   on Oct 20, 2026

   Member

   More actions

   For people interested in even more data, see my gist for

   • Arm64 Cobalt 100
   • x64 Intel i5-4570
   • x64 Ryzen 5625U

   Cobalt 100 results are similar to @diegorusso's findings here: tail-calling is a clear win for Arm64:

   Before the commit: MSVC 14.52 PGO+LTCG tail-call versus switch, with the switch build as baseline

   

   After the commit: MSVC 14.52 PGO+LTCG tail-call versus switch, with the switch build as baseline.

   

   x64 as well still shows big wins of tail-calling over switch-case like back then in #139922 (gist).

   The "peculiarities" around the before / after commit wrt MSVC version are far less pronounced on x64, though.

   x64 Intel i5-4570

   

   x64 Ryzen 5625U

   
7. diegorusso commented on Oct 20, 2026

   

   diegorusso

   on Oct 20, 2026

   ContributorAuthor

   More actions

   I guess the following can be claimed:

   On the Windows Dev Kit 2023, enabling the tail-call interpreter with MSVC 14.52 and PGO+LTCG delivered an 11.6% geometric-mean speedup across 121 successful pyperformance measurements, compared with the same-revision switch interpreter.

   Reacted by Zachary Ware, Chris Eibl and Sergey Miryanov
8. chris-eibl

   pinned and unpinned a comment on Oct 20, 2026
9. zware commented on Oct 21, 2026

   

   zware

   on Oct 21, 2026

   Member

   More actions

   My results such as they are can be found at https://gist.github.com/zware/bce63986f3765c731c5628b140013ec2

Issue actions