Description

itamaro

opened on Jul 4, 2025

Issue body actions

Bug report

Bug description:

When doing cross-builds, the PEP-739 build-details.json gets generated using inconsistent data, some of which based on the build python, and some from the target platform python.

See for example the following gist:
https://gist.github.com/itamaro/8f23ebec2e142fd7480241b76c16c4e0

The gist contains the build-details.json generated for a 3.15 GIL-disabled build, using a 3.15 GIL-enabled build as the build-python.

Concretely:

cd ~/pybuild/python/main-default
./configure --with-pydebug
make -j

cd ~/pybuild/python/main-nogil
./configure --with-pydebug --disable-gil
make -j

# now use the GIL-enabled python to regenerate build-details.json for the nogil build
_PYTHON_PROJECT_BASE=. _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_td_darwin_darwin _PYTHON_SYSCONFIGDATA_PATH=build/lib.macosx-14.5-arm64-3.15 ~/pybuild/python/main-default/python.exe -E Tools/build/generate-build-details.py /tmp/build-details.json

Some examples from the gist:

Incorrect - abi.flags is ["d"] instead of ["t", "d"]:

  "abi": {
    "flags": [
      "d"
    ]
  },

Correct - libpython.static has the correct SOABI, including t:

  "libpython": {
    "static": "/usr/local/lib/libpython3.15td.a"
  },

Incorrect - suffixes.extensions includes extensions for the GIL-enabled ABI:

    "extensions": [
      ".cpython-315d-darwin.so",
      ".cpython-315-darwin.so",
      ".abi3.so",
      ".so"
    ]

Notes

I used a "cross-build" on the same MacOS host here, with GIL enabled vs GIL disabled, because it was the easiest for a full end-to-end repro on a single machine.
The issue is more general though, as I originally observed it when generating build-info.json files for a MacOS-target runtime (both GIL enabled and disabled) using a Linux-GIL-enabled build-python. That setup is more involved, but the mixing is also more apparent, with some fields using "darwin" and other fields using "linux-gnu-x86_64" as the multiarch string.

CPython versions tested on:

CPython main branch, 3.15, 3.14

Operating systems tested on:

macOS, Linux

Linked PRs

• gh-136267: Use the target configuration in build-details.json #161529
• gh-136267: Read the target version from patchlevel.h in build-details.json #161532

Activity

1. itamaro

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   buildThe build process and cross-buildThe build process and cross-build

   needs backport to 3.14bugs and security fixesbugs and security fixes

   on Jul 4, 2025
2. StanFromIreland

   removed

   needs backport to 3.14bugs and security fixesbugs and security fixes

   on Jul 4, 2025
3. Zheaoli commented on Jul 4, 2025

   

   Zheaoli

   on Jul 4, 2025

   Contributor

   More actions

   I think this issue is caused by that we are mixing up the target and build info. I made a PoC patch for abi.flags and suffixes.extensions. I'm not sure this is acceptable

   Zheaoli@f7bfb6c

   Reacted by Filipe Laíns
4. itamaro commented on Jul 13, 2025

   

   itamaro

   on Jul 13, 2025

   ContributorAuthor

   More actions

   cc @FFY00 more generally, should build-details be generated strictly from sysconfigdata, so if sysconfigdata is accurate then build-details would also be accurate?
5. AA-Turner

   marked Crossbuild: sys.implementation and build-details.json #135785 as a duplicate of this issue on Sep 16, 2025
6. AA-Turner commented on Sep 16, 2025

   

   AA-Turner

   on Sep 16, 2025

   Member

   More actions

   Cross posting @sobolevn's comment from #135785 (closed as duplicate):

   After #135667 sys.implementation can now contain supports_isolated_interpreters field, which can be implementation dependent. For example, when building on Linux for WASI target, Linux build will support supports_isolated_interpreters and WASI won't (at least for now). But, since build-details.json is generated on the build machine, it will contain supports_isolated_interpreters=true. Which is not correct for the end environment.
7. FFY00 commented on Oct 4, 2025

   

   FFY00

   on Oct 4, 2025

   Member

   More actions

   cc @FFY00 more generally, should build-details be generated strictly from sysconfigdata, so if sysconfigdata is accurate then build-details would also be accurate?

   It should be generated from whatever is most accurate.
8. geofft commented on Oct 17, 2025

   

   geofft

   on Oct 17, 2025

   Contributor

   More actions

   Oh, I just tracked down why this is happening specifically for .suffixes.extensions. Tools/build/generate-build-details.py does

   data['suffixes']['extensions'] = importlib.machinery.EXTENSION_SUFFIXES

   which is in turn implemented in importlib._bootstrap_external as EXTENSION_SUFFIXES = _imp.extension_suffixes(), which is in turn implemented in C by turning const char *_PyImport_DynLoadFiletab[] into a Python list of strings.

   For many of the other things that generate-build-details.py looks at, they are set correctly set on cross builds because $(PYTHON_FOR_BUILD) has a number of environment variables set that override the behavior of various functions, mostly in sysconfig. Specifically, configure.ac sets the variable to:

       PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) _PYTHON_SYSCONFIGDATA_PATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`) '$with_build_python

   But this particular call path doesn't go through anything with an override.

   One option would be to try to read _PyImport_DynLoadFiletab out of the just-built Python binary or libpython, e.g. by opening it with ctypes (but that would break in a number of circumstances) or using a binary parser (which would require a binary parser, but honestly the new remote debugging stuff might have enough of a parser to pull this off, at least for the platforms it supports). Probably a cleaner option is to get the info that's in that variable out of the build in some other way.

   Similarly, ['abi']['flags'] is set from sys.abiflags, which is set in C as SET_SYS_FROM_STRING("abiflags", ABIFLAGS); and also doesn't get impacted by these environment variables. (This one, at least, is available in the Makefile, so we could just pass it as an argument to generate-build-details.py.)

   If it's helpful, python-build-standalone's interpreters exhibit both of these bugs: our debug and free-threaded variants use a --with-build-python that is neither of those, so we have the ['abi']['flags'] bug on those, and any of our cross-built interpreters (basically anything not x86_64 or arm64) has the['suffixes']['extensions'] bug, as reported in astral-sh/python-build-standalone#816.

   (FWIW I just opened #140266 which is also about incorrect data in build-details.json and kind of involves all those environment variables set in $(PYTHON_FOR_BUILD), but probably isn't super related, unless this is a motivation to more completely rework how the file is generated. As with that issue I'm happy to contribute a PR given guidance on how you want to approach this.)
9. manueljacob commented on Apr 7, 2026

   

   manueljacob

   on Apr 7, 2026

   Contributor

   More actions

   This very likely has the same root cause as #90067.
10. mgorny commented on Aug 13, 2026

    

    mgorny

    on Aug 13, 2026

    Contributor

    More actions

    Just hit this one as well. To be honest, I can only think of two ways of reliably solving this: we either mirror the whole logic from CPython in generate-build-details.py (or somewhere else in the build system), or we play with some hard C parsing to get it out of the preprocessed C code used to build CPython.

Issue actions