GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Cross compiling on Linux is untested, undocumented, and broken · Issue #90039 · python/cpython

Cross compiling on Linux is untested, undocumented, and broken #90039

New issue

Copy link

New issue

Copy link

Closed

Cross compiling on Linux is untested, undocumented, and broken#90039

Copy link

Labels

3.10 (EOL)end of lifeend of life3.11only security fixesonly security fixes3.9 (EOL)end of lifeend of lifebuildThe build process and cross-buildThe build process and cross-builddocsDocumentation in the Doc dirDocumentation in the Doc dirtestsTests in the Lib/test dirTests in the Lib/test dir

Description

tiran

opened

on Nov 23, 2021

Issue body actions

BPO	45881
Nosy	@brettcannon, @tiran, @moreati
PRs	bpo-45881: Use CC from env first for cross building (GH-29752) #29752 [3.10] bpo-45881: Use CC from env first for cross building (GH-29752). (GH-29753) #29753 [3.9] bpo-45881: Use CC from env first for cross building (GH-29752) (GH-29754) #29754 bpo-45881: configure --with-freeze-module --with-build-python (GH-29835) #29835
Dependencies	bpo-45886: Fix Program/_freeze_module for cross compiling Python bpo-45909: sysconfig --generate-posix-vars creates wrong file when cross compiling

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-23.16:37:45.271>
labels = ['3.9', '3.10', '3.11', 'build', 'tests', 'docs']
title = 'Cross compiling on Linux is untested, undocumented, and broken'
updated_at = <Date 2021-11-29.16:23:38.440>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-11-29.16:23:38.440>
actor = 'christian.heimes'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Build', 'Documentation', 'Tests', 'Cross-Build']
creation = <Date 2021-11-23.16:37:45.271>
creator = 'christian.heimes'
dependencies = ['45886', '45909']
files = []
hgrepos = []
issue_num = 45881
keywords = ['patch']
message_count = 7.0
messages = ['406853', '406860', '406935', '406942', '407016', '407165', '407292']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'christian.heimes', 'docs@python', 'Alex.Willmer']
pr_nums = ['29752', '29753', '29754', '29835']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue45881'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Activity

tiran commented on Nov 23, 2021
tiran
on Nov 23, 2021
MemberAuthor
More actions
Cross compiling is an approach to compile a program for a different CPU architecture and platform, e.g. compile for an ARM64 (aarch64) or WASM on a x86_64 build system.

Python configure script, Makefile, and setup.py have multiple references to cross compiling. However I could not find any documentation in the devguide or Python docs how to cross compile. We also lack CI (buildbot) to test cross compiling. This lack of awareness and testing leads to breakage of the feature. For example the design of Programs/_freeze_module in main (3.11-dev) is incompatible with cross compiling.

I kinda got cross compiling working with 3.10, but only with some additional hacks and patches. I also ran into other problems like _PYTHON_HOST_PLATFORM env var is not automatically forwarded to setup.py. The helper functions add_multiarch_paths() and add_cross_compiling_paths() break builds for me, too. Cross compiling only works when the methods are commented out.
Reacted by Thierry Laurion
tiran
added
3.9 (EOL)end of lifeend of life

3.10 (EOL)end of lifeend of life

3.11only security fixesonly security fixes
on Nov 23, 2021
tiran
assigned
docspython
on Nov 23, 2021
tiran
added
buildThe build process and cross-buildThe build process and cross-build

docsDocumentation in the Doc dirDocumentation in the Doc dir

3.9 (EOL)end of lifeend of life

testsTests in the Lib/test dirTests in the Lib/test dir

3.10 (EOL)end of lifeend of life

3.11only security fixesonly security fixes
on Nov 23, 2021
tiran
assigned
docspython
on Nov 23, 2021
tiran
added
buildThe build process and cross-buildThe build process and cross-build

docsDocumentation in the Doc dirDocumentation in the Doc dir

testsTests in the Lib/test dirTests in the Lib/test dir
on Nov 23, 2021
tiran commented on Nov 23, 2021
tiran
on Nov 23, 2021
MemberAuthor
More actions
I have uploaded a reproducer at https://github.com/tiran/cpython_builddep/compare/cross-aarch64

$ podman run -ti --rm -v $(pwd):/cpython:Z quay.io/tiran/cpythonbuild:ubuntu-impish-aarch64

...
aarch64-linux-gnu-gcc -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I/cpython/Include/internal -I/usr/include/x86_64-linux-gnu -I. -IObjects -IPython -I/usr/include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/cpython/Include -I/cpython/builddep/ubuntu-impish-x86_64 -c /cpython/Modules/_ctypes/_ctypes.c -o build/temp.linux-aarch64-3.10/cpython/Modules/_ctypes/_ctypes.o -DPy_BUILD_CORE_MODULE -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1
In file included from /usr/aarch64-linux-gnu/include/features.h:508,
from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
from /usr/aarch64-linux-gnu/include/limits.h:26,
from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/limits.h:203,
from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/syslimits.h:7,
from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/limits.h:34,
from /cpython/Include/Python.h:11,
from /cpython/Modules/zlibmodule.c:8:
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory
7 | # include <gnu/stubs-32.h>
tiran commented on Nov 24, 2021
tiran
on Nov 24, 2021
MemberAuthor
More actions
bpo-45886 addresses the cross build issue with freeze_module command.

The wrong header files come from the fact that setup.py uses CC variable from sysconfig instead of environment. The sysconfig variable contains the C compiler of the build interpreter instead of the host interpreter. The correct value is in os.environ.
tiran commented on Nov 24, 2021
tiran
on Nov 24, 2021
MemberAuthor
More actions
New changeset b30bf45 by Christian Heimes in branch 'main':
bpo-45881: Use CC from env first for cross building (GH-29752)
b30bf45
tiran commented on Nov 25, 2021
tiran
on Nov 25, 2021
MemberAuthor
More actions
New changeset cd6d257 by Christian Heimes in branch '3.9':
[3.9] bpo-45881: Use CC from env first for cross building (GH-29752) (GH-29754)
cd6d257
tiran commented on Nov 27, 2021
tiran
on Nov 27, 2021
MemberAuthor
More actions
New changeset 545aebd by Christian Heimes in branch '3.10':
[3.10] bpo-45881: Use CC from env first for cross building (GH-29752). (GH-29753)
545aebd
tiran commented on Nov 29, 2021
tiran
on Nov 29, 2021
MemberAuthor
More actions
New changeset 992565f by Christian Heimes in branch 'main':
bpo-45881: configure --with-freeze-module --with-build-python (GH-29835)
992565f
ezio-melotti
transferred this issue from on Apr 10, 2022
erlend-aasland commented on Jan 4, 2024
erlend-aasland
on Jan 4, 2024
Contributor
More actions
Some improvements were applied:

bpo-45881: Use CC from env first for cross building (GH-29752) #29752

I've created follow-up issues on the devguide and buildbot repos. Suggesting to close this, as there is no more actionable items at the moment.
erlend-aasland
closed this as completed on Jan 4, 2024

Metadata

Assignees

No one assigned

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Cross compiling on Linux is untested, undocumented, and broken #90039

Description

Activity

tiran commented on Nov 23, 2021

tiran commented on Nov 23, 2021

tiran commented on Nov 24, 2021

tiran commented on Nov 24, 2021

tiran commented on Nov 25, 2021

tiran commented on Nov 27, 2021

tiran commented on Nov 29, 2021

erlend-aasland commented on Jan 4, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions