GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-44133: Link Python executable with object files by vstinner · Pull Request #30556 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Jan 12, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

When Python is built without --enable-shared, the "python" program is
now linked to object files, rather than be linked the Python static
library (libpython.a), to make sure that all symbols are exported.
Previously, the linker omitted some symbols like the Py_FrozenMain()
function.

When Python is configured with --without-static-libpython, the Python
static library (libpython.a) is no longer built.

  • Reorganize configure.ac to check --without-static-libpython
    earlier.
  • Add LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variables to Makefile.

https://bugs.python.org/issue44133

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jan 12, 2022
@bedevere-bot bedevere-bot added the awaiting core review label Jan 12, 2022
@vstinner

vstinner commented Jan 12, 2022

Copy link
Copy Markdown
Member Author

With this change, import ctypes; assert hasattr(ctypes.pythonapi, 'Py_FrozenMain') no longer fails: the symbol is now available in all tests. I tested 4 cases.

The "python" file only exports the symbol when it's not linked to libpython.so.

When libpython.so exists, il always exports the symbol.

Case 1 (no option):

$ ls *.a *.so
# no file
# grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(LIBRARY_OBJS)
++ objdump -T ./python | grep Py_FrozenMain
00000000006722b2 g    DF .text	0000000000000285  Base        Py_FrozenMain
# ./python test.py
python binary has Py_FrozenMain symbol? True

Case 2, --without-static-libpython --enable-shared:

$ ls *.a *.so
libpython3.11d.so
$ grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(BLDLIBRARY)
$ objdump -T ./python | grep Py_FrozenMain
# not found
$ objdump -T libpython3.11d.so | grep Py_FrozenMain
00000000003b368a g    DF .text	00000000000002a9  Base        Py_FrozenMain
# ./python test.py
python binary has Py_FrozenMain symbol? True

Case 3, --without-static-libpython:

$ ls *.a *.so
# no file
$ grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(LIBRARY_OBJS)
$ objdump -T ./python | grep Py_FrozenMain
00000000006722b2 g    DF .text	0000000000000285  Base        Py_FrozenMain
# ./python test.py
python binary has Py_FrozenMain symbol? True

Case 4, --enable-shared:

$ ls *.a *.so
libpython3.11d.a  libpython3.11d.so
$ grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(BLDLIBRARY)
$ objdump -T ./python | grep Py_FrozenMain
# no result
$ objdump -T libpython3.11d.so | grep Py_FrozenMain
00000000003b368a g    DF .text	00000000000002a9  Base        Py_FrozenMain
$ ./python test.py
python binary has Py_FrozenMain symbol? True
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Jan 12, 2022

Copy link
Copy Markdown
Member Author

I don't know how to test this change on macOS to build Python with the "framework".
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Jan 12, 2022

Copy link
Copy Markdown
Member Author

@tiran @erlend-aasland: Would you mind to review this change?

cc @encukou
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Jan 12, 2022

Copy link
Copy Markdown
Member Author

On my Fedora 35, if I compare objdump -T ./python output on Python configure with --without-static-libpython, the difference with this PR is a single additional symbol: Py_FrozenMain() is now exported. Hopefully, no other symbol was missing previously.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
edmorley
edmorley reviewed Jan 12, 2022
View reviewed changes
Comment thread Misc/NEWS.d/next/Build/2022-01-12-13-34-52.bpo-44133.HYCNXb.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@vstinner vstinner mentioned this pull request Jan 12, 2022
bpo-44133: Export Py_FrozenMain() symbol. #29876
Closed
@vstinner
bpo-44133: Link Python executable with object files
fb2f8b1 
When Python is built without --enable-shared, the "python" program is
now linked to object files, rather than being linked to the Python
library (libpython.a), to make sure that all symbols are exported.
Previously, the linker omitted some symbols like the Py_FrozenMain()
function.

When Python is configured with --without-static-libpython, the Python
static library (libpython.a) is no longer built.

* Check --without-static-libpython earlier in configure.ac
* Add LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variables to Makefile.
* test_capi now ensures that the "Py_FrozenMain" symbol is exported.
@vstinner

vstinner commented Jan 12, 2022

Copy link
Copy Markdown
Member Author

PR rebased to fix merge conflicts.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Jan 12, 2022

Copy link
Copy Markdown
Member Author

I may be a good idea to test this build change on buildbots before merging the PR :-) Fedora and RHEL buildbots use --without-static-libpython.
shihai1991 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner
Test also PyThread_get_thread_native_id
c5c7102
@shihai1991

shihai1991 commented Jan 13, 2022

Copy link
Copy Markdown
Member

The gate failed. the test of this PR should be skipped in wins?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shihai1991 shihai1991 added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 13, 2022
@bedevere-bot

bedevere-bot commented Jan 13, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @shihai1991 for commit c5c7102 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 13, 2022
@vstinner
vstinner requested a review from a team as a code owner January 13, 2022 11:06
@vstinner

vstinner commented Jan 13, 2022

Copy link
Copy Markdown
Member Author

@shihai1991: "The gate failed. the test of this PR should be skipped in wins?"

No, the symbol must be present on all platforms. I pushed a fix.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Jan 13, 2022

Copy link
Copy Markdown
Member Author

Ah, adding frozenmain.c to the pythoncore project makes the Windows build to fail:

frozenmain.obj : error LNK2001: unresolved external symbol _PyWinFreeze_ExeTerm [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
frozenmain.obj : error LNK2001: unresolved external symbol _PyInitFrozenExtensions [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
frozenmain.obj : error LNK2001: unresolved external symbol _PyWinFreeze_ExeInit [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner
Skip the test on Windows
f0d052b
@vstinner

vstinner commented Jan 13, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Ok, I give up for this PR (I skip the test on Windows): I have no idea how to add Py_FrozenMain() on Windows if it requires 3 symbols when Python is linked.

Maybe Py_FrozenMain() should be modified to load these symbols at runtime on Windows?
erlend-aasland reacted with eyes emoji
All reactions
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner
vstinner merged commit 6be8489 into python:main Jan 13, 2022
@bedevere-bot bedevere-bot removed the awaiting core review label Jan 13, 2022
@vstinner
vstinner deleted the link_python_objs branch January 13, 2022 18:24
@vstinner vstinner mentioned this pull request Jan 26, 2022
make regen-all now suggests running: make autoconf #30893
Merged
@vstinner vstinner mentioned this pull request Apr 29, 2022
Some C-API symbols (e.g. Py_FrozenMain) are not always exported on Unix #88299
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@edmorley edmorley edmorley left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@vstinner @shihai1991 @bedevere-bot @edmorley @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.