GeezerVison is showing github.com — extension built with a shared python cannot be loaded with a static python · Issue #65735 · python/cpython
extension built with a shared python cannot be loaded with a static python #65735Description
| BPO |
21536 |
| Nosy |
@malemburg, @loewis, @warsaw, @ncoghlan, @pitrou, @vstinner, @merwok, @xdegaye, @embray, @dstufft |
| PRs |
bpo-21536: C extensions are no longer linked to libpython #12946bpo-21536: Revert Makefile change on python-config #12971 bpo-21536: On Android, C extensions are linked to libpython #12989bpo-21536: Fix What's New in Python 3.8 entry #13242bpo-21536: On Cygwin, C extensions must be linked with libpython #13549bpo-21536: Follow-up to #13549 #13552 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-04-29.14:44:31.749>
created_at = <Date 2014-05-19.18:50:02.385>
labels = ['3.8', 'type-feature', 'library']
title = 'extension built with a shared python cannot be loaded with a static python'
updated_at = <Date 2020-12-14.09:17:25.183>
user = 'https://github.com/pitrou' bugs.python.org fields:
activity = <Date 2020-12-14.09:17:25.183>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-29.14:44:31.749>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2014-05-19.18:50:02.385>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 21536
keywords = ['patch']
message_count = 34.0
messages = ['218806', '218809', '218813', '219181', '219186', '220293', '340814', '340817', '340818', '340819', '340821', '340837', '340852', '340854', '340870', '340914', '340924', '340926', '340927', '340940', '340973', '340974', '341014', '341046', '341085', '342149', '343372', '343386', '343396', '343397', '343410', '343418', '343420', '343664']
nosy_count = 13.0
nosy_names = ['lemburg', 'loewis', 'barry', 'ncoghlan', 'pitrou', 'vstinner', 'eric.araujo', 'xdegaye', 'erik.bray', 'piotr.dobrogost', 'dstufft', 'reimar', 'zhtw1234']
pr_nums = ['12946', '12971', '12989', '13242', '13549', '13552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21536'
versions = ['Python 3.8'] Activitypitrou commented on May 19, 2014 When a C extension is built (using distutils) with a shared library Python, it cannot be loaded with an otherwise identical statically linked Python. The other way round works fine. Trivial example using the _ssl module:
>>> import sys
>>> sys.path.insert(0, '/home/antoine/cpython/shared/build/lib.linux-x86_64-3.5/')
>>> import _ssl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory This is probably because of an additional -L flag that is passed when linking a C extension with a shared library Python. I don't think the flag is useful under Linux (or perhaps under any other OS at all), since the relevant symbols are already loaded when the interpreter tries to load the C extension.
(AFAIK, systems notorious for providing shared library Pythons are RedHat-alike systems, while Debian/Ubuntu provide statically linked Pythons) pitrou commented on May 19, 2014 Actually, it's not a -L flag but a -l flag. Removing the "-lpython3.5m" flag from the linker line works fine under Linux, and allows the resulting extension to be loaded with both a shared libary Python and a statically-linked Python. loewis commented on May 19, 2014 I think you are right. It would IMO be useful to research a few comparable systems. E.g. Apache modules don't link a shared library, but still refer to apr_ functions as undefined symbols - but then, there isn't an APR shared library in the first place (at least not on Debian - how about Redhat?)
PHP might be close to our case: Debian includes a libphp5.so (in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so link against it, and all the PHP modules (in /usr/lib/php5/20100525+lfs/) don't link with the shared library - on Debian. I wonder how it is on systems that actually use the PHP shared library. pitrou commented on May 26, 2014
PHP might be close to our case: Debian includes a libphp5.so
(in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so
link against it, and all the PHP modules (in
/usr/lib/php5/20100525+lfs/) don't link with the shared library -
on Debian. I wonder how it is on systems that actually use the PHP
shared library.
Regardless of libphp5.so, the modules in /usr/lib/php5/20100525 don't link against /usr/lib/apache2/modules/libphp5.so, though the latter is obviously able to load those modules. pitrou commented on May 26, 2014 Hmm, apparently the -l flag was added in bpo-832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system). The OP there also mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the desired effect).
(also, the OP didn't mention why he used a shared library build, instead of linking Python statically with the dlopened library) pitrou commented on Jun 11, 2014 Martin, what do you think about the aforementioned use case? vstinner commented on Apr 25, 2019 vstinner commented on Apr 25, 2019 Antoine:
Hmm, apparently the -l flag was added in bpo-832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system). The OP there also mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the desired effect).
(also, the OP didn't mention why he used a shared library build, instead of linking Python statically with the dlopened library)
bpo-34814 is linked to this use case: https://bugzilla.redhat.com/show_bug.cgi?id=1585201 is an example of Python embedded in C using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW). Problem: some C extensions of the standard library cannot be loaded in this case, like _struct.
On Fedora and RHEL, some C extensions like _struct are built by the "*shared*" section of Modules/Setup. In this case, these C extensions are not explicitly linked to libpython.
IHMO it's a bad usage of dlopen(): libpython must always be loaded with RTLD_GLOBAL.
bpo-832799 has been fixed by the following commit which modify distutils to link C extensions to libpython:
commit 10acfd0
Author: Martin v. Löwis <[email protected]>
Date: Mon Apr 10 12:39:36 2006 +0000
Patch bpo-1429775: Link Python modules to libpython on linux if
--enable-shared. Fixes bpo-832799.
vstinner commented on Apr 25, 2019 vstinner commented on Apr 25, 2019 I wrote PR 12946: "On Unix, C extensions are no longer linked to libpython".
Using PR 12946, the use case described in the initial message (msg218806) now works as expected. I can load a C extension built by a shared library Python with a statically linked Python:
$ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38notshared/bin/python3.8
Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:10:57)
>>> import sys; sys.path.insert(0, '/opt/py38shared/lib/python3.8/lib-dynload')
>>> import _ssl
>>> _ssl
<module '_ssl' from '/opt/py38shared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'> /opt/py38notshared/bin/python3.8 is statically linked, whereas /opt/py38shared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so comes from a shared library Python.
Install shared libray Python into /opt/py38shared:
git clean -fdx; ./configure CFLAGS="-O0" --enable-shared --prefix /opt/py38shared && make && make install
Install statically linked Python into /opt/py38notshared:
git clean -fdx; ./configure CFLAGS="-O0" --prefix /opt/py38notshared && make && make install
--
As Antoine said, the opposite already works. Just in case, I also tested and I confirm that it still works:
$ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38shared/bin/python3.8
Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:09:02)
>>> import sys; sys.path.insert(0, '/opt/py38notshared/lib/python3.8/lib-dynload')
>>> import _ssl
>>> _ssl
<module '_ssl' from '/opt/py38notshared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'> _ssl comes from statically linked Python (/opt/py38notshared) and is loaded in shared library Python (/opt/py38shared). vstinner commented on May 24, 2019
This completely breaks building extension modules on Windows-based platforms like Cygwin and MinGW, where it is necessary when building even shared libraries for all global symbols to resolvable at link time.
C extensions are always linked to libpython on Android. I'm open to do something similar for Cygwin and/or MinGW if you can come up with a PR :-) See LIBPYTHON variable in configure.ac.
You can use Xavier's change as an example: commit 254b309, PR 12989. vstinner commented on May 24, 2019 New changeset c994c8f by Victor Stinner (E. M. Bray) in branch 'master':
bpo-21536: On Cygwin, C extensions must be linked with libpython (GH-13549)
c994c8f vstinner commented on May 24, 2019 I merged E. M. Bray's PR to get in Python 3.8 beta 1. If anything goes wrong, we can fix it later ;-) vstinner commented on May 24, 2019 xdegaye commented on May 24, 2019 FWIW I checked the Android build after those last changes. vstinner commented on May 24, 2019
FWIW I checked the Android build after those last changes.
Thank you! I was overconfident when I merged the Cygwin which looked good to me, whereas it broke Android :-( Hopefully it was quickly fixed. I hope that everything will be alright for the the beta1 to start from a good milestone! embray commented on May 27, 2019 Thanks everyone. And FWIW I agree the original change is positive overall, if a bit presumptuous about different linkers' behaviors :) kpajko79 commented on Sep 2, 2022 Just ran into this issue while cross-compiling python 3.9.2 for Android API 32 and managed to get the shared extensions working. Made only a few small modifications to the Makefile.
First added -Wl,-Bsymbolic to all shared libraries in the Android build as the Debian package has a patch to add this to the Linux build (actually -Wl,-Bsymbolic-functions) and i seemed like a good idea to copy that. Mainly because some time ago it was a de-facto standard to add this flag to every native library build [0]. Probably this is unrelated and not required [1].
Then found an interesting section in the man page of ld:
global
This option is only meaningful when building a shared
object. It makes the symbols defined by this shared
object available for symbol resolution of subsequently
loaded libraries.
So rebuilt libpython3.9.so with -Wl,-z,global
And voila, it rocks! Virtually this marks the shared library to be treated as if it had been loaded with RTLD_GLOBAL, moving all its symbols to the global namespace.
The parameter just turns on a flag in the ELF header, so it could be possible to try its effect with an ELF editor much faster than recompiling.
0x000000000000001e (FLAGS) SYMBOLIC BIND_NOW
0x000000006ffffffb (FLAGS_1) Flags: NOW GLOBAL
[0] https://gcc.gcc.gnu.narkive.com/ceGm0WCb/android-the-reason-why-bsymbolic-is-turned-on-by-default
[1] https://www.technovelty.org/c/what-exactly-does-bsymblic-do.html
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|