GeezerVison is showing github.com — Building and linking C extensions in a post-distutils world · Issue #99942 · python/cpython
Building and linking C extensions in a post-distutils world #99942DescriptionWith the removal of distutils from 3.12 alphas, I've recently taken a new look at the use of distutils within https://mesonbuild.com in the hopes that we were finally unblocked and could migrate to sysconfig.
- install schemes turn out to finally be viable since 3.10.3, when the
deb_system scheme patch on Debian operating systems for distutils was patched into sysconfig. This hard blocker is gone, thank G-d.
- linking to libpython itself may or may not be an issue, originally I thought it was, but now I think it may not be.
How can a C-extension supporting PEP 517 build backend know the best way to compile and link?
python <3.12
Meson has some code: https://github.com/mesonbuild/meson/blob/master/mesonbuild/modules/python.py#L338-L407
Here's the interesting bit:
def links_against_libpython():
from distutils.core import Distribution, Extension
cmd = Distribution().get_command_obj('build_ext')
cmd.ensure_finalized()
return bool(cmd.get_libraries(Extension('dummy', [])))python >=3.8
In bpo-36721, bpo-21536 etc the above code changes from returning True to False, on many Unix platforms.
New additional methods suitable for getting C extension arguments are available. Well, mostly suitable.
sysconfig.get_config_var('LIBPYTHON')
this is configured into:
pkg-config --cflags --libs python3python-config --embed
There's a couple problems with this:
- on Cygwin and Android, LIBPYTHON and thus pkg-config is hardcoded to link to libpython, but distutils only does so when
Py_ENABLE_SHARED
- none of it works on Windows, which since it is built with PCBuild, doesn't have Makefile config vars (and also / consequently? distributes neither of the latter two)
- pkg-config may not always be installed, so we want a fallback for that
- python-config has tons of CPython build-time junk so we cannot use it
...
It feels uncomfortably like there's still way too much undocumented magic here.
@vstinner, I assume the Cygwin/Android inconsistency is probably a configure.ac bug that should behave the same way distutils does/did? I can provide a patch to fix it but would like confirmation of which side to fix.
@FFY00, I think in the long term, sysconfig should expose a function that returns cflags / libs required to build an extension (and works on Windows without _generate_posix_vars, and doesn't include lots of -O3 -pipe -fstack-protector-strong -fdiagnostics-color=always and more -- i.e. works like pkg-config, not like python-config). Even without the question of "whether to link to libpython", there's a 140-line class for figuring out what the name of the library is, which directory to find it in, and what the include directory is too. Of course, this is specific to the case where pkg-config is not available, and most of it is for the Windows case.
Linked PRs
Activityzooba commented on Jan 12, 2023 The Windows case really isn't complicated because the linking is all set up to isolate the two sides of the ABI. The only thing that's going to bleed through is C runtime state when you mix a debug build with a release build (which use two different CRT instances). Virtually nobody actually has a debug build of CPython, and those who do will know because the filenames are different, so there's nothing really to pick up from the interpreter.
Now, the complicated bit is finding the compiler, in effect, this bit:
|
root = os.environ.get("ProgramFiles(x86)") or os.environ.get("ProgramFiles") |
|
if not root: |
|
return None, None |
|
|
|
try: |
|
path = subprocess.check_output([ |
|
os.path.join(root, "Microsoft Visual Studio", "Installer", "vswhere.exe"), |
|
"-latest", |
|
"-prerelease", |
|
"-requires", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64", |
|
"-property", "installationPath", |
|
"-products", "*", |
|
], encoding="mbcs", errors="strict").strip() |
|
except (subprocess.CalledProcessError, OSError, UnicodeDecodeError): |
|
return None, None |
This is assuming you want to compile with MSVC. If you want to compile an extension with a different compiler (many of which will work, since the ABI is well defined), you'll need different code. Again, this is best left to the tool figuring out how to run the build - one of the main reasons for dropping distutils is so that compiler options don't get trapped by the CPython release cycle.
Otherwise, the location of Python's header files (typically {prefix}\Include) and import libraries (typically {prefix}\libs) are all you need. They should both be in sysconfig somewhere already, but since you can't use that when cross-compiling, we either need a new data file containing the paths (which I'm in favour of, but it wasn't popular last time we discussed it) or to just stick with the assumptions tools use today. eli-schwartz commented on Jan 12, 2023 ContributorAuthor More actions
They should both be in sysconfig somewhere already, but since you can't use that when cross-compiling, we either need a new data file containing the paths (which I'm in favour of, but it wasn't popular last time we discussed it) or to just stick with the assumptions tools use today.
There's also the undocumented $_PYTHON_SYSCONFIGDATA_NAME environment variable, which is unfortunately only available on posix. But POSIX systems do often make the assumption that they can convince e.g. distutils to cross compile by setting that environment variable before running any build tool (pip, python -m build, python setup.py ...). added a commit that references this issue on Feb 16, 2023gh-99942: correct the pkg-config/python-config flags for cygwin/android added a commit that references this issue on Feb 22, 2023gh-99942: python.pc on android/cygwin should link to libpython per co… BryteLite commented on May 17, 2025 eli-schwartz commented on May 18, 2025 ContributorAuthor More actions
I welcome any technical critique or discussion this may prompt. The goal is clarity and resilience.
My technical critique is that this report appears to be AI generated. It contains numerous hallucinations which can be trivially disproven. It is also 100% unrelated to the current ticket, which has nothing to do with gcc (15 or otherwise), nor with C standard versions.
Please avoid posting distractingly incorrect info.
@sethmlarson we have another case of https://sethmlarson.dev/slop-security-reports I think Reacted by Steve Dower, Seth Larson and Sam James sethmlarson commented on May 19, 2025 @eli-schwartz Indeed, we received the same report to PSRT and have so far disregarded it entirely as being AI-generated nonsense. eli-schwartz commented on May 19, 2025 ContributorAuthor More actions Thanks. For the record, pandas / numpy / cython / matplotlib were also targeted (I linked them to your article and the reports were closed).
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|