GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-111225: Link extension modules against libpython on Android by mhsmith · Pull Request #115780 · python/cpython

Conversation

@mhsmith

@mhsmith mhsmith commented Feb 21, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

For the reasons explained in PEP 738, extension modules must be linked against libpython on Android.

Cygwin has the same requirement, and it looks like there were two existing attempts to implement this:

  • One of them, in makesetup, only applied to Cygwin.
  • The other, in configure.ac, applied to both Cygwin and Android, and worked by setting a LIBPYTHON variable. But this had no effect, because the variable wasn't actually being used anywhere.

I've removed the first attempt in favor of the second one, and fixed it as follows:

  • Renamed the variable to MODULE_LDFLAGS to more accurately reflect its purpose.
  • Edited makesetup to use the variable when linking extension modules.
  • Edited the Makefile so that extension modules depend on libpython on these platforms.
  • Restored -fPIC on Android. It was removed several years ago with a note that the toolchain used it automatically, but this is no longer the case. Omitting it causes all linker commands to fail with an error like relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol '_Py_FalseStruct'; recompile with -fPIC.

Before:

$ readelf -d _socket.cpython-313.so | grep -E 'NEEDED|SONAME' 
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]

After:

 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libpython3.13.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
  • Fixes Cross Compile for Android and issues with PyExc_OSError #111225

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@mhsmith
Link extension modules against libpython on Android
09a58b7
@mhsmith
mhsmith requested review from corona10 and erlend-aasland as code owners February 21, 2024 18:42
@bedevere-app bedevere-app Bot added the awaiting review label Feb 21, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Feb 21, 2024
Cross Compile for Android and issues with PyExc_OSError #111225
Closed
@mhsmith mhsmith mentioned this pull request Feb 21, 2024
gh-71052: Add Android build script #115576
Closed
erlend-aasland
erlend-aasland reviewed Feb 21, 2024
View reviewed changes
Comment thread Modules/makesetup Resolved

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland
erlend-aasland previously approved these changes Feb 21, 2024
View reviewed changes

@erlend-aasland erlend-aasland left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good; thanks a lot for making this bug-fix in a separate PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Feb 21, 2024
@erlend-aasland

This comment was marked as resolved.

Sign in to view
@erlend-aasland
erlend-aasland dismissed their stale review February 21, 2024 21:18

macOS shared build broke

@bedevere-app bedevere-app Bot added awaiting review and removed awaiting merge labels Feb 21, 2024
@erlend-aasland

erlend-aasland commented Feb 21, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Hm, seems the macOS build is broke for shared builds on main; so regard my previous comment (I marked it as resolved).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland
erlend-aasland approved these changes Feb 21, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Feb 21, 2024
erlend-aasland
erlend-aasland requested changes Feb 21, 2024
View reviewed changes

@erlend-aasland erlend-aasland left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the back and forth; I had forgotten to clean the build folder. I redid my tests and I now can confirm that this breaks --enable-shared builds on macOS (main works fine).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting changes and removed awaiting merge labels Feb 21, 2024
@bedevere-app

This comment was marked as outdated.

Sign in to view
@erlend-aasland

erlend-aasland commented Feb 21, 2024

Copy link
Copy Markdown
Contributor

Sorry for the back and forth; I had forgotten to clean the build folder. I redid my tests and I now can confirm that this breaks --enable-shared builds on macOS (main works fine).

Aha, this PR does not include 074bbec; rebasing onto main fixed the problem. Looks like this is good to go again 😄

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@erlend-aasland
Merge branch 'main' into android-link-modules
44f3c82
erlend-aasland
erlend-aasland approved these changes Feb 21, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting changes labels Feb 21, 2024
@erlend-aasland
erlend-aasland enabled auto-merge (squash) February 21, 2024 22:41
@erlend-aasland
erlend-aasland merged commit 7f5e3f0 into python:main Feb 21, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Feb 21, 2024
@mhsmith mhsmith mentioned this pull request Feb 25, 2024
gh-114099: Add configure and Makefile targets to support iOS compilation. #115390
Merged
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this pull request Mar 4, 2024
@mhsmith @woodruffw
pythongh-111225: Link extension modules against libpython on Android (p…
90ac592
…ython#115780)

Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
This was referenced Mar 13, 2024
BLD: not all @LIBPYTHON@ changed to @MODULE_LDFLAGS@ #116745
Closed
gh-116745: remove all internal usage of @LIBPYTHON@ #116746
Merged
tacaswell added a commit to tacaswell/cpython that referenced this pull request Mar 13, 2024
@tacaswell
pythongh-116745: remove all internal usage of @LIBPYTHON@
3a44865
Follow on to python#115780 .

Without this change numpy's build is broken.
befeleme pushed a commit to fedora-python/cpython that referenced this pull request Mar 20, 2024
@tacaswell @befeleme
00424: pythongh-116745: remove all internal usage of @LIBPYTHON@
dddd390
Follow on to python#115780 .

Without this change numpy's build is broken.
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@mhsmith @diegorusso
pythongh-111225: Link extension modules against libpython on Android (p…
1bd2625
…ython#115780)

Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
@eli-schwartz

eli-schwartz commented Jul 28, 2024

Copy link
Copy Markdown
Contributor

Renamed the variable to MODULE_LDFLAGS to more accurately reflect its purpose.

Renaming it breaks the de-facto API that the sysconfig module has provided for 5 years, and which downstream projects rely on.

Cygwin has the same requirement, and it looks like there were two existing attempts to implement this:

  • One of them, in makesetup, only applied to Cygwin.

  • The other, in configure.ac, applied to both Cygwin and Android, and worked by setting a LIBPYTHON variable. But this had no effect, because the variable wasn't actually being used anywhere.

That check was originally written to configure the python-config script -- I later updated it to also configure the python.pc file. distutils had its own, second copy of the configure.ac logic which was probably dropped during the migration away from internally using distutils. Not really the fault of the variable itself but what can you do. :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhsmith

mhsmith commented Jul 30, 2024

Copy link
Copy Markdown
Member Author

It's unfortunate that sysconfig exposes all the Makefile variables in this way, because most of them were created with no intention of becoming a stable API. Can you give some examples of projects that rely on this variable?

Setuptools, at least, does not rely on it. It has its own logic for deciding which platforms need to link against libpython, and it already knows about Windows, Cygwin and Android.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eli-schwartz

eli-schwartz commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

https://github.com/mesonbuild/meson/blob/fa4f2339465ce3d755e2df802ebd5aa962e2ad27/mesonbuild/scripts/python_info.py#L66-L79

Lacking any formalization of the information which might come via #107956, it is necessary to poke at raw Makefile variables -- in this case, the raw Makefile variable that corresponds to whether the "C-API extension modules" pkg-config file is configured to link to libpython. The setuptools logic is actually a copy-paste of the stdlib distutils logic, which was manually kept in sync with Makefile.pre.in up until it got deleted from the stdlib.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@rgommers

rgommers commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

This looks like it's going to break the build for NumPy on Android - the flag to link to libpython is going to silently disappear. And there is a fair amount of usage of NumPy on Android, judging by bug reports over the past few years. And we won't be able to work around this for NumPy 2.1.0 which is about to be released with Python 3.13 support, even if Meson adapts to this rename.

LIBPYTHON is the most obvious way to determine whether the linker flag is needed yes or no. It's one of the more important flags one can access through get_config_vars. A libpython key is also present in the current draft of PEP 739: https://peps.python.org/pep-0739/#libpython for that reason.

It's unfortunate that sysconfig exposes all the Makefile variables in this way, because most of them were created with no intention of becoming a stable API

I couldn't agree more, but unfortunately that's the situation we're in. It's best not to rename variables unless the actual content/semantics of the variable changes. In this case, reverting the rename looks like the right thing to do.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhsmith mhsmith mentioned this pull request Aug 6, 2024
gh-116622: Rename build variable MODULE_LDFLAGS back to LIBPYTHON #122764
Merged
@mhsmith

mhsmith commented Aug 6, 2024

Copy link
Copy Markdown
Member Author

Thanks for the explanation.

This looks like it's going to break the build for NumPy on Android

Actually as @eli-schwartz says in #116745 (comment), it looks like it will break the build on every platform except Android, because the Meson code defaults to linking against libpython when the variable is missing from sysconfig.

So I agree the rename must be reverted. I've done that in #122764.

rgommers and erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eli-schwartz

eli-schwartz commented Aug 7, 2024

Copy link
Copy Markdown
Contributor

Thanks for looking into this! :)

Fingers crossed that PEP 739 can be rolled out in the near future (hopefully as a beautiful gift included in 3.14).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Sanny388cry

Sanny388cry commented Aug 9, 2024

Copy link
Copy Markdown

mhsmith:android-link-modules

Sanny388cry reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhsmith mhsmith mentioned this pull request Sep 30, 2024
Problem building Python 3.12 chaquo/chaquopy#1143
Closed
LukasWoodtli pushed a commit to LukasWoodtli/cpython that referenced this pull request Jan 22, 2025
@mhsmith @LukasWoodtli
pythongh-111225: Link extension modules against libpython on Android (p…
b86840b
…ython#115780)

Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
@lazka lazka mentioned this pull request Sep 11, 2025
.pc for Android contains invalid syntax #138800
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@erlend-aasland erlend-aasland erlend-aasland approved these changes
@corona10 corona10 Awaiting requested review from corona10 corona10 is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Cross Compile for Android and issues with PyExc_OSError

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@mhsmith @erlend-aasland @eli-schwartz @rgommers @Sanny388cry
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.