GeezerVison is showing github.com — 3.14: Windows on Arm build fails since windows-11-arm migrated to windows-11-vs2026-arm · Issue #158244 · python/cpython
3.14: Windows on Arm build fails since windows-11-arm migrated to windows-11-vs2026-arm #158244DescriptionThe Windows Arm64 jobs in the CI recently started failing on 3.14 (for example). Noted by @StanFromIreland in #158188 (comment): the _freeze_module
|
<FreezeProjects> |
|
<Platform>$(PreferredToolArchitecture)</Platform> |
can no longer be compiled for PreferredToolArchitecture=x86 and PlatformToolset=v143:
The build tools for Visual Studio 2022 (Platform Toolset = 'v143') cannot be found.
PreferredToolArchitecture=x86 is chosen, because we use the x86 version of msbuild in CI:
"C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin\msbuild.exe"
which is chosen by build.bat if no MSBUILD environment variable is set and msbuild is not in the path. 1
Compiling the _freeze_module for arm64 on an arm64 host by just using the arm64 version of msbuild would solve this, by e.g. just adding
env:
MSBUILD: ${{ inputs.arch == 'arm64' && 'C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin\arm64\msbuild.exe' || '' }}
to
|
- name: Build CPython |
|
run: >- |
|
.\\PCbuild\\build.bat |
because then PreferredToolArchitecture=arm64 and also the arm64 version of the compiler is chosen.
I think running the _freeze_module natively on an arm64 host is anyway the better fit?
Activitychris-eibl commented on Sep 26, 2026 Note, we cannot override PreferredToolArchitecture if we use the x86 version of msbuild via
build.bat "/p:PreferredToolArchitecture=arm64"
due PROCESSOR_ARCHITECTURE=x86 in this case, and via C:\Program Files\Microsoft Visual Studio\18\Community\MSBuild\Microsoft\VC\v180\Microsoft.Cpp.ToolsetLocation.props:
<PropertyGroup>
<!-- UseNativeEnvironment and _IsNativeEnvironment was used in prev versions to define if we want to use 64-bit tools when building for x64 platform. -->
<PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and ('$(UseNativeEnvironment)' == 'true' or '$(_IsNativeEnvironment)' == 'true')">x64</PreferredToolArchitecture>
<!-- By default we use the same bitness as the hosting platform -->
<PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and ('$(PROCESSOR_ARCHITECTURE)' == 'AMD64' and '$(Platform)' == 'x64')">x64</PreferredToolArchitecture>
<!-- prefer arm64 tools when on arm64 -->
<PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and '$(PROCESSOR_ARCHITECTURE)' == 'ARM64' and '$(HostARM64Suported)' != 'false'">arm64</PreferredToolArchitecture>
<PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == ''">x86</PreferredToolArchitecture>
<!-- If OS is not x64, we cannot use x64 tools even if preferred -->
<PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == 'x64' and ('$(PROCESSOR_ARCHITECTURE)' != 'AMD64' and '$(PROCESSOR_ARCHITEW6432)' != 'AMD64' and '$(PROCESSOR_ARCHITECTURE)' != 'ARM64')">x86</PreferredToolArchitecture>
<!-- If OS is not arm64, we cannot use arm64 tools even if preferred -->
<PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == 'arm64' and '$(PROCESSOR_ARCHITECTURE)' != 'ARM64'">x86</PreferredToolArchitecture>
</PropertyGroup>we always get PreferredToolArchitecture=x86. This is also the default, hence in all builds so far:
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostX86\x86\CL.exe /c /IC:\a\cpython\cpython\PCbuild\obj\314win32_Debug\_freeze_module\
Why this does not work anymore for the latest windows-11-arm image is not clear to me, especially because these compilers are part of it
Toolset HostX86_X86 HostX86_ARM64 HostARM64_X86 HostARM64_ARM64
------- ----------- ------------- ------------- ---------------
14.29.30133 True True False False
14.44.35207 True True False True
14.51.36231 True True True True
so 14.44.35207\bin\HostX86\x86 should be found.
Details
=== Compiler paths ===
--- 14.29.30133 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\cl.exe
--- 14.44.35207 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostarm64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostarm64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx86\x86\cl.exe
--- 14.51.36231 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\x86\cl.exe
See https://github.com/chris-eibl/cpython/actions/runs/36241148789/job/108401760423 in my fork.
chris-eibl commented on Sep 26, 2026 Short summary:
Running the _freeze_module natively on an arm64 host could be done in main as well and would fix 3.14 without the need of migrating to v145, which was done to get 3.14 CI green via backporting:
which could be reverted again.
Otherwise:
The current situation in 3.14 CI is that
- v145 is used for WoA builds
- but still v143 for AMD64 and x86, because they use windows-2022 that does not have VS2026.
Regarding official releases, AFAIU we could achieve the same (or even stick with v143 for Arm) since we cross-compile and only run the PGO step on @zooba's Azure VM (at least that's my understanding): we'd simply have to use the respective image, that is windows-2022 instead of windows-2025. For the first we get v143, for the latter v145, since this is chosen when VS 2026 is available now on 3.14. Previously, we've used v143 in this case, too.
Compiling and running the _freeze_module natively works on the new windows-11-arm image, tested here:
https://github.com/chris-eibl/cpython/actions/runs/36245087168/job/108412696751
note the
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostArm64\arm64\CL.exe
chris-eibl commented on Sep 26, 2026 zware commented on Sep 27, 2026 I do in fact have MSBUILD set on that worker; see line 19 for example. I found that was the only way to get a build to succeed at all with available RAM. Reacted by Chris Eibl Reacted by Chris Eibl chris-eibl commented on Sep 27, 2026 Yupp, I forgot to emphasize that using the native msbuild would of course use the native compiler for the whole build, which is an even stronger motivation.
Another way to achive that: react on PROCESSOR_ARCHITECTURE in build.bat to prefer the native msbuild which then uses the native compiler tool chain.
Btw, for the same reason, in the whole non-arm CI currently HostX86 is used - and most probably all build bots, except they do set MSBUILD or PreferredToolArchitecture (which I think is close to zero).
Using the native compiler tool chain would also give us faster compilation and lesser likelihood that the compiler runs out of memory. zooba commented on Sep 28, 2026 I think it's totally fine for buildbots to override either of those variables to get a native MSBuild/native build tools.
I also don't have a problem with the use of a newer toolset, provided it's going to be used for release builds as well and the vcruntime tag in cpython-bin-deps gets updated.
Keeping the older toolsets in some buildbots is a nice way to make sure we don't adopt brand new features too eagerly, but compatibility has been good enough that I'm not concerned about whether a contributor will probably be able to use their existing v143 while buildbots are running v145.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|