GeezerVison is showing github.com — bpo-45950: Introduce Bootstrap Python again by tiran · Pull Request #29859 · python/cpython
Conversation
tiran
commented
Nov 30, 2021
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
The build system now uses a :program:_bootstrap_python interpreter for
freezing and deepfreezing again. To speed up build process the build tools
:program:_bootstrap_python and :program:_freeze_module are no longer
build with LTO.
Cross building depends on a build Python interpreter, which must have same
version and bytecode as target host Python.
https://bugs.python.org/issue45950
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
|
🤖 New build scheduled with the buildbot fleet by @tiran for commit b757d5bd526cfffeb134eb6744ac86c561e61e9d 🤖
If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
🤖 New build scheduled with the buildbot fleet by @tiran for commit fa4cd7fc9f3763ff1309d8bfbc97853223f17239 🤖
If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
it's just an extra link step
When LTO is used, a link step can be quite slow. I noticed that when there was a second _bootstrap_python binary. I didn't check if _testembed has a similar issue or not. I'm mostly developing Python without --enable-shared: Python is linked "statically" in all built executables.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Maybe LTO flags can be skipped on these less important binaries.
|
gvanrossum and erlend-aasland reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
tiran
commented
Dec 1, 2021
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.
We need some extra logic in configure.ac and Makefile anyway. The _bootstrap_python approach is incompatible with cross compiling. When you cross compile for a different architecture or platform, then any compiled code is not executable on the build host.
The approach also speeds up local development as long as you have a recent Python installation on your computer. With _bootstrap_python almost every code file gets rebuild whenever a core file changes. Scratch that, the Program/_freeze_module dependency triggers a partial rebuild.
I'll come up with a better way to test _buildbot_python.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am in favor of this. Thanks for doing this!
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
tiran
changed the title
Bootstrap Python again
bpo-45950: Introduce Bootstrap Python again
Dec 1, 2021
|
I'll try to build this as RPM in Fedora to see if there are some problems with our bootstrapped OOT build.
|
tiran reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:
...
gcc -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -fno-lto -o Programs/_freeze_module Programs/_freeze_module.o Modules/getbuildinfo.o Parser/token.o Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o Python/pydtrace.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o Modules/faulthandler.o Modules/posixmodule.o Modules/signalmodule.o Modules/_tracemalloc.o Modules/_codecsmodule.o Modules/_collectionsmodule.o Modules/errnomodule.o Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o Modules/itertoolsmodule.o Modules/_sre.o Modules/_threadmodule.o Modules/timemodule.o Modules/_weakref.o Modules/_abc.o Modules/_functoolsmodule.o Modules/_localemodule.o Modules/_operator.o Modules/_stat.o Modules/symtablemodule.o Modules/pwdmodule.o Modules/xxsubtype.o -ldl -lm
/usr/bin/ld: Programs/_freeze_module.o: in function `runtime_init':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:44: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o: in function `_Py_DECREF':
/builddir/build/BUILD/cpython-bootstrap_again/Include/object.h:493: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:(.data.rel.ro+0x0): undefined reference to `__gcov_indirect_call'
/usr/bin/ld: Programs/_freeze_module.o: in function `main':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:191: undefined reference to `__gcov_indirect_call'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_indirect_call_profiler_v4'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:211: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: more undefined references to `__gcov_time_profiler_counter' follow
...
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:
Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
OOT build with optimizations is passing for me on F35:
../../configure -C --enable-optimizations``
make
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
🤖 New build scheduled with the buildbot fleet by @tiran for commit 02da340e62db33a12aad1b03adf83a488e0255fb 🤖
If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:
Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.
That helped.
|
tiran reacted with hooray emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
The build system now uses a `_bootstrap_python` interpreter for freezing
and deepfreezing again. Cross building depends on a build Python interpreter.
`_bootstrap_python` and `_freeze_module` are no longer compiled with
LTO. Link time optimization is very slow.
|
@tiran: Please replace # with GH- in the commit message next time. Thanks!
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|