GeezerVison is showing github.com — Default dlopenflags inconsistent between Linux and macOS · Issue #137880 · python/cpython
Default dlopenflags inconsistent between Linux and macOS #137880DescriptionFeature or enhancement
Proposal:
The default dlopen flags for loading extension modules specify neither RTLD_GLOBAL nor RTLD_LOCAL.
Unfortunately, the behavior when neither flag is specified differs between Linux and macOS: on Linux, RTLD_LOCAL is the default, while on macOS RTLD_GLOBAL seems to be the default.
POSIX itself is not helpful on the matter: "If neither RTLD_GLOBAL nor RTLD_LOCAL is specified, the default behavior is unspecified".
Having two different behaviors on two extremely popular platforms may make support more difficult for maintainers of complex extension modules that bundle popular third-party C/C++ libraries (here is a possible instance).
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response Activitypitrou commented on Aug 17, 2025 gpshead commented on Aug 17, 2025 cc @pablogsal @Yhg1s - by "default", pun intended, I'd assume the default dlopen extension module loading flag behaviors are depended on in platform specific manners by users by now? pitrou commented on Aug 17, 2025 (as a potentially related issue, sys.setdlopenflags is a process-wide setting and therefore not suitable for use in a library) pablogsal commented on Aug 18, 2025 I have been burned countless times for this. The classic case is two extensions statically compiling libstdc++ or some other dependency and then being loaded in macOS and everything burns.
On the other hand I don't think we can change anything here at this point as any change is backwards incompatible no? gpshead commented on Aug 18, 2025 If it is a compatibility issue, changing this might line up with "if we get an abi4" (importing those could be explicit about the flags defined). rgommers commented on Aug 18, 2025 I can't remember having seen issues on macOS with this, but there's other ways to run into this on Linux as well (from a quick browse through the SciPy issue tracker):
- Musl is more strict than glibc about symbol visibility in some ways, so one may have to use
RTLD_GLOBAL and then see an issue in musllinux that doesn't show up on manylinux
- Embedded interpreters can behave differently, I found one issue where it required messing with the default
dlopen mode which then gave a symbol conflict (Problems with using libsndfile & scipy scipy/scipy#8130).
In SciPy we found that the most reliable way to avoid such issues is to use -fvisibility=hidden (or inlineshidden for C++) by default as well as use a linker script to hide all symbols except Py_Init_*.
Re backwards compat: It feels similar to the multiprocessing default start method, both are annoying footguns. Unless there is a reasonable justification for the platform difference for dlopen, a pre-announced migration may be justifiable.
sys.setdlopenflags is a process-wide setting and therefore not suitable for use in a library)
It may be nice to allow using setdlopenflags as a context manager perhaps? Reacted by Gregory P. Smith pitrou commented on Aug 18, 2025
sys.setdlopenflags is a process-wide setting and therefore not suitable for use in a library)
It may be nice to allow using setdlopenflags as a context manager perhaps?
If it becomes a thread-local setting, yes, otherwise a context manager will merely save a few lines of code without fixing the underlying problem. Reacted by Ralf Gommers and Gregory P. Smith pablogsal commented on Aug 18, 2025
sys.setdlopenflags is a process-wide setting and therefore not suitable for use in a library)
It may be nice to allow using setdlopenflags as a context manager perhaps?
If it becomes a thread-local setting, yes, otherwise a context manager will merely save a few lines of code without fixing the underlying problem.
We do control the flags when we call dlopen on import so it can totally become a thread local Reacted by Gregory P. Smith and Antoine Pitrou pablogsal commented on Aug 18, 2025
Re backwards compat: It feels similar to the multiprocessing default start method, both are annoying footguns.
Well, as much as I agree with the sentiment is technically not a footgun: it's literally the platform default and everything else behaves like that on macOS. Changing the platform default is an order of magnitude more tricky than changing an option (fork) that we know is fundamentally broken in the platform (confirmed by the platform owners) pitrou commented on Aug 18, 2025
Well, as much as I agree with the sentiment is technically not a footgun: it's literally the platform default and everything else behaves like that on macOS.
It is, but dlopen is a very low-level API and people importing an extension module don't think "this is going to call dlopen with the default platform flags and it is what I want". pablogsal commented on Aug 18, 2025
It is, but dlopen is a very low-level API and people importing an extension module don't think "this is going to call dlopen with the default platform flags and it is what I want".
Yeah, but as an extension author I can tell you I surely have in mind that this is the case and all my extensions are compiled in macOS with that in mind. For example we compile with -Wl,-exported_symbols_list to only allow the init symbol in Darwin systems.
My point here is that end users surely don't need to know but extension authors do.
Also just to be super clear: I am in the camp of not wanting RTLD_GLOBAL on Mac, I am just saying that changing this may be harder than we think Reacted by Gregory P. Smith
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|