GeezerVison is showing github.com — Statically linked `sqlite3` fails checks due to missing `-lm` · Issue #128321 · python/cpython
Statically linked sqlite3 fails checks due to missing -lm #128321DescriptionBug report
Bug description:
On Linux, if I build a static version of sqlite3 (e.g., ./configure --disable-shared && make from a sqlite3 tarball)
You can see that ./configure for CPython fails the SQLite checks (added back in #30016):
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... no
checking for sqlite3_column_decltype in -lsqlite3... no
checking for sqlite3_column_double in -lsqlite3... no
checking for sqlite3_complete in -lsqlite3... no
checking for sqlite3_progress_handler in -lsqlite3... no
checking for sqlite3_result_double in -lsqlite3... no
checking for sqlite3_set_authorizer in -lsqlite3... no
checking for sqlite3_trace_v2 in -lsqlite3... no
checking for sqlite3_trace in -lsqlite3... no
checking for sqlite3_value_double in -lsqlite3... no
checking for sqlite3_load_extension in -lsqlite3... no
checking for sqlite3_serialize in -lsqlite3... no
checking for --enable-loadable-sqlite-extensions... no
And in the config.log there are a bunch of failures due to undefined references to functions in libm, e.g.:
configure:15469: checking for sqlite3_bind_double in -lsqlite3
configure:15492: gcc -o conftest -I/usr/local/include -I$(srcdir)/Modules/_sqlite -L/usr/local/lib -lsqlite3 -L/usr/local/lib conftest.c -lsqlite3 -L/usr/local/lib -lsqlite3 -ldl >&5
/usr/bin/ld: /usr/local/lib/libsqlite3.a(sqlite3.o): in function `logFunc':
/repo/sqlite-autoconf-3470100/sqlite3.c:131765:(.text+0x26858): undefined reference to `log2'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131768:(.text+0x268d4): undefined reference to `log'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131762:(.text+0x268ec): undefined reference to `log10'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131750:(.text+0x2691c): undefined reference to `log'
CPython is determing SQLite's requirements via pkg-config if it can at:
|
PKG_CHECK_MODULES( |
|
[LIBSQLITE3], [sqlite3 >= 3.15.2], [], [ |
|
LIBSQLITE3_CFLAGS=${LIBSQLITE3_CFLAGS-""} |
|
LIBSQLITE3_LIBS=${LIBSQLITE3_LIBS-"-lsqlite3"} |
|
] |
|
) |
And -lm is included in the pkg-config for the static sqlite3 build, but it's in the private libs and requires the --static flag to be included in the --libs output:
$ pkg-config --static --libs sqlite3
-L/usr/local/lib -lsqlite3 -lm
$ pkg-config --libs sqlite3
-L/usr/local/lib -lsqlite3
However, PKG_CHECK_MODULES doesn't pass this flag. There's a PKG_CHECK_MODULES_STATIC macro which does, but CPython doesn't seem to use this yet. The alternative is set the flag manually at configure time:
PKG_CONFIG="$PKG_CONFIG --static"
which changes all the PKG_CHECK_MODULES calls to include the --static flag. However, there's a missing export for LIBS which prevents this from resolving the issue:
$ git rev-parse HEAD
f9a5a3a3ef34e63dc197156e9a5f57842859ca04
$ export PKG_CONFIG="pkg-config --static"
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... no
checking for sqlite3_column_decltype in -lsqlite3... no
checking for sqlite3_column_double in -lsqlite3... no
checking for sqlite3_complete in -lsqlite3... no
checking for sqlite3_progress_handler in -lsqlite3... no
checking for sqlite3_result_double in -lsqlite3... no
checking for sqlite3_set_authorizer in -lsqlite3... no
checking for sqlite3_trace_v2 in -lsqlite3... no
checking for sqlite3_trace in -lsqlite3... no
checking for sqlite3_value_double in -lsqlite3... no
checking for sqlite3_load_extension in -lsqlite3... no
checking for sqlite3_serialize in -lsqlite3... no
checking for --enable-loadable-sqlite-extensions... no
checking for stdlib extension module _sqlite3... missing
^C
This is "solved" with the patch:
diff --git a/configure.ac b/configure.ac
index badb19d5589..5ed147f5482 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4221,6 +4221,7 @@ dnl bpo-45774/GH-29507: The CPP check in AC_CHECK_HEADER can fail on FreeBSD,
dnl hence CPPFLAGS instead of CFLAGS.
CPPFLAGS="$CPPFLAGS $LIBSQLITE3_CFLAGS"
LDFLAGS="$LIBSQLITE3_LIBS $LDFLAGS"
+ LIBS="$LIBSQLITE3_LIBS $LIBS"
AC_CHECK_HEADER([sqlite3.h], [
have_sqlite3=yesAs demonstrated by:
$ export PKG_CONFIG="pkg-config --static"
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... yes
checking for sqlite3_column_decltype in -lsqlite3... yes
checking for sqlite3_column_double in -lsqlite3... yes
checking for sqlite3_complete in -lsqlite3... yes
checking for sqlite3_progress_handler in -lsqlite3... yes
checking for sqlite3_result_double in -lsqlite3... yes
checking for sqlite3_set_authorizer in -lsqlite3... yes
checking for sqlite3_trace_v2 in -lsqlite3... yes
checking for sqlite3_value_double in -lsqlite3... yes
checking for sqlite3_load_extension in -lsqlite3... yes
checking for sqlite3_serialize in -lsqlite3... yes
checking for --enable-loadable-sqlite-extensions... no
For context, this was reported in astral-sh/python-build-standalone#449 and some of the findings here are repeated there.
I've posted the patch in #128322
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Activityzanieb commented on Dec 29, 2024 ContributorAuthor More actions Reacted by Erlend E. Aasland zanieb commented on Dec 29, 2024 ContributorAuthor More actions In addition to #128322, we may also want to include -lm at
|
LIBSQLITE3_LIBS=${LIBSQLITE3_LIBS-"-lsqlite3"} |
If the pkg-config check fails, we fallback to including these flags. However, without -lm all of our subsequent checks will fail! Happy to post a pull request for that too. eli-schwartz commented on Dec 29, 2024
In addition to #128322, we may also want to include -lm at
|
LIBSQLITE3_LIBS=${LIBSQLITE3_LIBS-"-lsqlite3"} |
If the pkg-config check fails, we fallback to including these flags. However, without -lm all of our subsequent checks will fail! Happy to post a pull request for that too.
This will result in over-linking when sqlite3 is used as a shared library (no pkg-config --static). Reacted by Erlend E. Aasland and Zanie Blue zanieb commented on Dec 30, 2024 ContributorAuthor More actions
This will result in over-linking when sqlite3 is used as a shared library
Ah true — we could just include it during the checks but it seems like a bit of a pain. Reacted by Erlend E. Aasland added a commit that references this issue on Dec 30, 2024gh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 requirem… solvingj commented on Feb 12, 2025 @zanieb Thank you for finding and fixing this. It's a huge help.
This will result in over-linking when sqlite3 is used as a shared library
Ah true — we could just include it during the checks but it seems like a bit of a pain.
I'm probably missing something, and if so please forgive me.
Since cpython won't be using pkgconfig --static any time soon, is it therefor impossible for anyone to compile sqlite3 in statically without patching configure themselves just to append -lm to the LIBS, and just to successfully satisfy the check? If so, does the overlinking represent a risk for the sqlite symbol check that is worth leaving it in this state, which is basically just broken for this use case? eli-schwartz commented on Feb 12, 2025 Typically what happens is people who are interested in a static build export PKG_CONFIG="pkg-config --static" to ensure that all dependencies are detected statically. That will include -lm and also -lz (the latter wasn't mentioned here and it is needed if you build a dynamic python that is statically linked to the sqlite library).
So everything works fine using pkg-config, the question is what to do about the fallback.
For a fallback, what you'd typically do is perform a series of compile checks, to see if you can link successfully to sqlite on its own and upon failure check again to see if you can link successfully to sqlite when adding -lm -lz.
I have no objections to the "series of compile checks". I don't think it's fully appropriate in terms of general theory to simply add it unconditionally though. I'm also fine with only supporting pkg-config at all, for my own personal use, but that's another story. :) solvingj commented on Feb 12, 2025 I see. It seems to me the project is designed to support static/dynamic choices (and all configuration choices for that matter), on a per-dependency basis. I definitely see it being "very common" that people who want "some deps" compiled in statically want "all deps" statically. On the other hand, it's also "not uncommon" for people to want a subset of deps static. Regardless, it doesn't seem appropriate even in principle to have an "all static or nothing static" support story, just because pkgconfig makes the "all static" case easy. So many other configurations and mechanisms in cpython are in place to make it flexible in this way.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|