GeezerVison is showing github.com — Use pkg-config autoconf macros to detect flags for Modules/Setup · Issue #89736 · python/cpython
Use pkg-config autoconf macros to detect flags for Modules/Setup #89736Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2021-10-22.15:22:43.638>
labels = ['type-feature', 'build', '3.11']
title = 'Use pkg-config autoconf macros to detect flags for Modules/Setup'
updated_at = <Date 2021-11-19.15:41:03.253>
user = 'https://github.com/tiran' bugs.python.org fields:
activity = <Date 2021-11-19.15:41:03.253>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2021-10-22.15:22:43.638>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 45573
keywords = ['patch']
message_count = 14.0
messages = ['404781', '404783', '404784', '406084', '406135', '406138', '406139', '406331', '406375', '406520', '406527', '406535', '406586', '406591']
nosy_count = 5.0
nosy_names = ['twouters', 'brett.cannon', 'christian.heimes', 'ned.deily', 'erlendaasland']
pr_nums = ['29164', '29534', '29559', '29594', '29614', '29615', '29616']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = '45774'
type = 'enhancement'
url = 'https://bugs.python.org/issue45573'
versions = ['Python 3.11'] Activitytiran commented on Oct 22, 2021 pkg-config [1] is a standard tool on Linux and other platforms to detect presence of dependencies as well as to figure out which compiler and linker flags they require. Development packages provide a .pc file, e.g. ncurses provides a ncursesw.pc file.
$ pkg-config --libs --cflags ncursesw
-D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -lncursesw -ltinfo I propose to use a modified version of pkg-config's PKG_HAVE_DEFINE_WITH_MODULES macro in our configure script. On succss the modified macro defines:
- HAVE_FOO=1
- FOO_CFLAGS="some compile flags"
- FOO_LIBS="some library flags for linker"
On error, it sets nothing and does not cause configure to stop with an error.
The macro also allows users to override flags by setting FOO_CFLAGS and FOO_LIBS env vars. HAVE_FOO is added to pyconfig.h. The FOO_CFLAGS/LIBS are added to Makefile, from where it can be consumed by Modules/Setup.
Eventually Python could use the flags in setup.py, too. For now I would like to start with Modules/Setup. It is only used by some power users and has less risk of breaking the setup of beginners.
[1] https://www.freedesktop.org/wiki/Software/pkg-config/ tiran commented on Oct 22, 2021 I added some whitespace to Modules/Setup. All lines starting with #([a-z_]) compile cleanly on Fedora 34 with all dependencies available. dbm, tkinter, and sqlite are missing or untested.
$ sed -E -i 's/^#([a-z_])/\1/g' Modules/Setup
$ make
...
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc _asyncio _bisect
_blake2 _bz2 _codecs_cn
_codecs_hk _codecs_iso2022 _codecs_jp
_codecs_kr _codecs_tw _contextvars
_crypt _csv _curses
_curses_panel _datetime _elementtree
_hashlib _heapq _json
_lsprof _lzma _md5
_multibytecodec _opcode _pickle
_posixsubprocess _queue _random
_sha1 _sha256 _sha3
_sha512 _socket _ssl
_statistics _struct _testbuffer
_testimportmultiple _testinternalcapi _testmultiphase
_typing _xxsubinterpreters _xxtestfuzz
_zoneinfo array audioop
binascii cmath fcntl
grp math mmap
nis ossaudiodev pwd
pyexpat readline resource
select spwd syslog
termios time unicodedata
xxlimited xxlimited_35 zlib
$ ./python -c "import sys; print(sys.builtin_module_names)"
('_abc', '_ast', '_asyncio', '_bisect', '_blake2', '_bz2', '_codecs', '_codecs_cn', '_codecs_hk', '_codecs_iso2022', '_codecs_jp', '_codecs_kr', '_codecs_tw', '_collections', '_contextvars', '_crypt', '_csv', '_curses', '_curses_panel', '_datetime', '_elementtree', '_functools', '_hashlib', '_heapq', '_imp', '_io', '_json', '_locale', '_lsprof', '_lzma', '_md5', '_multibytecodec', '_opcode', '_operator', '_pickle', '_posixsubprocess', '_queue', '_random', '_sha1', '_sha256', '_sha3', '_sha512', '_signal', '_socket', '_sre', '_ssl', '_stat', '_statistics', '_string', '_struct', '_symtable', '_testbuffer', '_testimportmultiple', '_testinternalcapi', '_testmultiphase', '_thread', '_tokenize', '_tracemalloc', '_typing', '_warnings', '_weakref', '_xxsubinterpreters', '_xxtestfuzz', '_zoneinfo', 'array', 'atexit', 'audioop', 'binascii', 'builtins', 'cmath', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'mmap', 'nis', 'ossaudiodev', 'posix', 'pwd', 'pyexpat', 'readline', 'resource', 'select', 'spwd', 'sys', 'syslog', 'termios', 'time', 'unicodedata', 'xx', 'xxlimited', 'xxlimited_35', 'xxsubtype', 'zlib')
$ ldd python
linux-vdso.so.1 (0x00007fffa1f40000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6032fe2000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6032fdb000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f6032fd6000)
libm.so.6 => /lib64/libm.so.6 (0x00007f6032e92000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f6032e58000)
libnsl.so.2 => /lib64/libnsl.so.2 (0x00007f6032e3c000)
libtirpc.so.3 => /lib64/libtirpc.so.3 (0x00007f6032e0a000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f6032df7000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f6032dcb000)
libz.so.1 => /lib64/libz.so.1 (0x00007f6032db1000)
libreadline.so.8 => /lib64/libreadline.so.8 (0x00007f6032d59000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f6032cbc000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f60329cc000)
libncursesw.so.6 => /lib64/libncursesw.so.6 (0x00007f603298d000)
libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f603295e000)
libpanel.so.6 => /lib64/libpanel.so.6 (0x00007f6032958000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6032789000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6033029000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f6032732000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6032652000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f603263a000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6032633000)
libncurses.so.6 => /lib64/libncurses.so.6 (0x00007f6032606000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f60325f5000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f60325ec000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f60325d2000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f60325a6000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f603250f000)tiran commented on Oct 22, 2021 $ find build/lib.linux-x86_64-3.11/ -name '*.so' | sort
build/lib.linux-x86_64-3.11/_ctypes.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_ctypes_test.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_dbm.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_decimal.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_gdbm.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_multiprocessing.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_posixshmem.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_sqlite3.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_testcapi.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_tkinter.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_uuid.cpython-311-x86_64-linux-gnu.so tiran commented on Nov 10, 2021 We can detect majority of our dependencies with pkg-config. The use of pkg-config has some benefits:
-
Distro's provide the .pc files in their -dev / -devel packages. The presence of a .pc file indicates that all development dependencies are available.
-
pkg-config does the right thing for non-standard include and libraries directories as well as multiarch builds. In case a library or header is not on the default search path, pkg-config returns necessary -I and -L flags.
-
At least the pkgconf implementation of pkg-config standard search /usr/local and ~/.local/ directories for .pc files. Cases like bpo-45774: Autoconfiscate SQLite detection (GH-29507) #29507 are handled correctly. On FreeBSD "pkgconf sqlite3 --cflags --libs" returns "-I/usr/local/include -L/usr/local/lib -lsqlite3".
-
pkg-config understands dependencies. For example "pkg-config --libs tk" returns linker flags for TK *and* TCL.
-
pkg-config can check for module version, e.g. "pkg-config sqlite3 --atleast-version=3.7.15"
pkg-config modules:
readline, libedit
ncursesw, ncurses, panel, tinfo
sqlite3
zlib
bzip2
liblzma
expat
uuid (Linux's util-linux uuid)
libffi
libnsl, libtirpc
libcrypt
tcl, tk
openssl, libssl, libcrypto
modules / libraries without pkg-config modules:
decimal: libmpdec
gdbm: gdbm
dbm: gdbm_compat, ndbm, libdb (bdb)
To simplify use of flags in Modules/Setup, I propose to add two make variables for each module that needs cflags and ldflags:
f"MODULE_{ext.name.upper()}CFLAGS"
f"MODULE{ext.name.upper()}_LDFLAGS"
e.g. for the _ssl module:
MODULE__SSL_CFLAGS=
MODULE__SSL_LDFLAGS=-lssl -lcrypto Then use the flags from Makefile in setup.py:
def update_extension_flags(self, ext):
name = ext.name.upper()
cflags = sysconfig.get_config_var(f"MODULE_{name}_CFLAGS")
if cflags:
ext.extra_compile_args.extend(shlex.split(cflags))
ldflags = sysconfig.get_config_var(f"MODULE_{name}_LDFLAGS")
if ldflags:
ext.extra_link_args.extend(shlex.split(ldflags))
return extFinally update Modules/makesetup to use the new variables, too. brettcannon commented on Nov 10, 2021 ned-deily commented on Nov 10, 2021 SGTM,2
This all sounds great. I think a goal here should be to remove all header and lib file searching from setup.py as that has always been a bug magnet. Perhaps one workaround for those libs that don't (yet) provide .pc files would be for us to supply reasonable default .pc for them, if possible, so that most builders wouldn't have to set the MODULE_xxx_*FLAGS variables while still removing the header/lib file searching from setup.py? Also it would be great to document how using pkg-config works in generic cross-compiling cases; we should strive to make at least the most common cases just work with minimal tweaking of configure arguments. tiran commented on Nov 10, 2021 gdbmmodule and dbmmodule need special treatment anyway. macOS has dbm-API build into libc. Linux has either libgdbm_compat, libndbm, or libdb. The --with-dbmliborder makes it even more interesting. Users can override in which order they want to probe for gdbm, ndbm, and libdb. We have to keep the "manual" library and header checks.
Cross-compilation with pkg-config uses a trivial wrapper script, https://autotools.io/pkgconfig/cross-compiling.html . The build system has to create and provide a script with correct sysroot setting.
Is traditional cross-compiling work still useful these days anyway? Emulation has made big leaps in the last decade. Emulated cross compiling with qemu has become widespread. It also has the big advantage that you can run the test suite on the emulated hardware and verify that your binaries work. AFAIK Fedora's build system uses qemu for a bunch of hardware targets. tiran commented on Nov 14, 2021 brettcannon commented on Nov 15, 2021
Is traditional cross-compiling work still useful these days anyway?
Yes for things like WebAssembly where there is no equivalent of an emulated CPU under QEMU. tiran commented on Nov 18, 2021 tiran commented on Nov 18, 2021 tiran commented on Nov 18, 2021 tiran commented on Nov 19, 2021 Erlend implemented pkg-config for sqlite3 in bpo-45774. Thanks to his tireless effort we now have a blue print how to port other checks to optional pkg-config lookup. tiran commented on Nov 19, 2021 erlend-aasland commented on Jan 22, 2023 AFAICS, this is implemented. If you disagree, please re-open or create a new issue describing the problem.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|