GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-45950: Introduce Bootstrap Python again by tiran · Pull Request #29859 · python/cpython

Conversation

@tiran

@tiran tiran commented Nov 30, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

The build system now uses a :program:_bootstrap_python interpreter for
freezing and deepfreezing again. To speed up build process the build tools
:program:_bootstrap_python and :program:_freeze_module are no longer
build with LTO.

Cross building depends on a build Python interpreter, which must have same
version and bytecode as target host Python.

https://bugs.python.org/issue45950

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Nov 30, 2021
@bedevere-bot bedevere-bot added the awaiting core review label Nov 30, 2021
@tiran
tiran force-pushed the bootstrap_again branch from fe57850 to b757d5b Compare November 30, 2021 10:05
vstinner
vstinner reviewed Nov 30, 2021
View reviewed changes
Comment thread Programs/_bootstrap_python.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Programs/_bootstrap_python.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021
@bedevere-bot

bedevere-bot commented Nov 30, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit b757d5bd526cfffeb134eb6744ac86c561e61e9d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021
@tiran
tiran force-pushed the bootstrap_again branch from b757d5b to fa4cd7f Compare November 30, 2021 11:07
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021
@bedevere-bot

bedevere-bot commented Nov 30, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit fa4cd7fc9f3763ff1309d8bfbc97853223f17239 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021
@gvanrossum
gvanrossum self-requested a review November 30, 2021 15:05
@gvanrossum

gvanrossum commented Nov 30, 2021

Copy link
Copy Markdown
Member

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 30, 2021

Copy link
Copy Markdown
Member

it's just an extra link step

When LTO is used, a link step can be quite slow. I noticed that when there was a second _bootstrap_python binary. I didn't check if _testembed has a similar issue or not. I'm mostly developing Python without --enable-shared: Python is linked "statically" in all built executables.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 30, 2021

Copy link
Copy Markdown
Member

Maybe LTO flags can be skipped on these less important binaries.

gvanrossum and erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 30, 2021

Copy link
Copy Markdown
Member

FYI the revert #29717 fixed an issue with out of tree build. Please try to make sure that this bug is not reintroduced: https://bugs.python.org/issue45866#msg406764

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran

tiran commented Dec 1, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

We need some extra logic in configure.ac and Makefile anyway. The _bootstrap_python approach is incompatible with cross compiling. When you cross compile for a different architecture or platform, then any compiled code is not executable on the build host.

The approach also speeds up local development as long as you have a recent Python installation on your computer. With _bootstrap_python almost every code file gets rebuild whenever a core file changes. Scratch that, the Program/_freeze_module dependency triggers a partial rebuild.

I'll come up with a better way to test _buildbot_python.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran
tiran force-pushed the bootstrap_again branch 3 times, most recently from 757a652 to 551fb30 Compare December 1, 2021 12:03
gvanrossum
gvanrossum approved these changes Dec 1, 2021
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am in favor of this. Thanks for doing this!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Dec 1, 2021
@tiran
tiran force-pushed the bootstrap_again branch from 551fb30 to cfe4551 Compare December 1, 2021 16:28
@tiran tiran changed the title Bootstrap Python again bpo-45950: Introduce Bootstrap Python again Dec 1, 2021
@tiran
tiran marked this pull request as ready for review December 1, 2021 16:29
@hroncok

hroncok commented Dec 2, 2021

Copy link
Copy Markdown
Contributor

I'll try to build this as RPM in Fedora to see if there are some problems with our bootstrapped OOT build.

tiran reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hroncok

hroncok commented Dec 2, 2021

Copy link
Copy Markdown
Contributor

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

...
gcc -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -g   -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -g  -fno-lto -o Programs/_freeze_module Programs/_freeze_module.o Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o   Python/pydtrace.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/xxsubtype.o -ldl                         -lm 
/usr/bin/ld: Programs/_freeze_module.o: in function `runtime_init':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:44: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o: in function `_Py_DECREF':
/builddir/build/BUILD/cpython-bootstrap_again/Include/object.h:493: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:(.data.rel.ro+0x0): undefined reference to `__gcov_indirect_call'
/usr/bin/ld: Programs/_freeze_module.o: in function `main':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:191: undefined reference to `__gcov_indirect_call'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_indirect_call_profiler_v4'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:211: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: more undefined references to `__gcov_time_profiler_counter' follow
...
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran
tiran force-pushed the bootstrap_again branch from cfe4551 to 02da340 Compare December 2, 2021 15:28
@tiran

tiran commented Dec 2, 2021

Copy link
Copy Markdown
Member Author

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran

tiran commented Dec 2, 2021

Copy link
Copy Markdown
Member Author

OOT build with optimizations is passing for me on F35:

../../configure -C --enable-optimizations``
make
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2021
@bedevere-bot

bedevere-bot commented Dec 2, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit 02da340e62db33a12aad1b03adf83a488e0255fb 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2021
@hroncok

hroncok commented Dec 2, 2021

Copy link
Copy Markdown
Contributor

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

That helped.

tiran reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran tiran mentioned this pull request Dec 2, 2021
bpo-45949: Pure Python freeze module for cross builds (GH-29899) #29899
Merged
@tiran
tiran force-pushed the bootstrap_again branch from 02da340 to 252b53a Compare December 3, 2021 10:48
@tiran
tiran requested a review from a team as a code owner December 3, 2021 10:48
@tiran

tiran commented Dec 3, 2021

Copy link
Copy Markdown
Member Author

Rebased on top of #29902

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran
tiran force-pushed the bootstrap_again branch from 252b53a to c630ffd Compare December 3, 2021 11:59
@tiran
bpo-45950: Bootstrap Python again
ba35cc4
The build system now uses a `_bootstrap_python` interpreter for freezing
and deepfreezing again. Cross building depends on a build Python interpreter.

`_bootstrap_python` and `_freeze_module` are no longer compiled with
LTO. Link time optimization is very slow.
@tiran
tiran force-pushed the bootstrap_again branch from c630ffd to ba35cc4 Compare December 3, 2021 14:23
@tiran
tiran merged commit 84ca123 into python:main Dec 3, 2021
@tiran
tiran deleted the bootstrap_again branch December 3, 2021 15:01
@bedevere-bot

bedevere-bot commented Dec 3, 2021

Copy link
Copy Markdown

@tiran: Please replace # with GH- in the commit message next time. Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting merge label Dec 3, 2021
@tiran tiran mentioned this pull request May 9, 2022
Reintroduce bootstrap_python for freezing #90108
Closed
@kumaraditya303 kumaraditya303 mentioned this pull request Jun 23, 2022
Deepfreeze using 3.10, not bootstrap interpreter. #93771
Closed
@corona10 corona10 mentioned this pull request Sep 12, 2022
Link-time-optimization with clang is broken #96761
Closed
@corona10 corona10 mentioned this pull request Oct 11, 2023
test_embed fails if Python is built with LTO and LLVM clang on macOS #110313
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
@gvanrossum gvanrossum gvanrossum approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@tiran @bedevere-bot @gvanrossum @vstinner @hroncok @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.