GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Consider supporting emscripten/webassembly as a build target · Issue #84461 · python/cpython

Consider supporting emscripten/webassembly as a build target #84461

New issue

Copy link

New issue

Copy link

Closed

Consider supporting emscripten/webassembly as a build target#84461

Copy link

Labels

3.11only security fixesonly security fixesOS-wasibuildThe build process and cross-buildThe build process and cross-buildtype-featureA feature request or enhancementA feature request or enhancement

Description

SimonBiggs

mannequin

opened

on Apr 14, 2020

Issue body actions

BPO	40280
Nosy	@warsaw, @brettcannon, @terryjreedy, @tiran, @mcepl, @pmp-p, @rdb, @ethanhs, @corona10, @miss-islington, @tirkarthi, @erlend-aasland
PRs	bpo-40280: Add wasm cross build targets (GH-29771) #29771 bpo-40280: Use Setup.stdlib static for wasm builds (GH-29784) #29784 bpo-40280: Move hard-coded feature checks to configure (GH-29789) #29789 bpo-40280: clean and ignore .wasm files (GH-29794) #29794 bpo-40280: Add configure check for socket shutdown (GH-29795) #29795 bpo-40280: Disable unusable core extension modules on emscripten (GH-29834) #29834 bpo-40280: Emscripten systems use .wasm suffix by default (GH-29842) #29842 bpo-40280: Emscripten has no support for subprocesses #29872 bpo-40280: Emscripten defaults to --with-ensurepip=no (GH-29873) #29873 bpo-40280: Emscripten with_ensurepip=no, second attempt (GH-29884) #29884 bpo-40280: Optimize ints and and startup on wasm (GH-29887) #29887 bpo-40280: Enable assertions in Emscripten debug builds #29892 bpo-40280: Update what's new (GH-29893) #29893 bpo-40280: Add Tools/wasm with helpers for cross building (GH-29984) #29984 bpo-40280: Disable epoll_create in Emscripten config.site (GH-30494) #30494 bpo-40280: Add build target for Emscripten/node.js #30495 bpo-40280: Skip IPPROTO_SCTP tests on Emscripten #30538 bpo-40280: Add --with-emscripten-target to build for browser or node (GH-30552) #30552 bpo-40280: Allow to compile _testcapi as builtin module (GH-30559) #30559 bpo-40280: Build WASM stdlib bundle and more modules for node (GH-30597) #30597 bpo-40280: Block more syscalls that are causing crashes in tests (GH-30601) #30601 bpo-40280: Skip subprocess-based tests on wasm32-emscripten (GH-30615) #30615 bpo-40280: Change subprocess imports for cleaner error on wasm32 (GH-30620) #30620 bpo-40280: Add requires_fork test helper (GH-30622) #30622 bpo-40280: Misc fixes for wasm32-emscripten (GH-30722) #30722 bpo-40280: Get help() working and more (GH-30858) #30858 bpo-40280: Use presence of msvcrt module to detect Windows #30930 bpo-40280: Address more test failures on Emscripten (GH-31050) #31050 bpo-40280: Block more non-working syscalls in Emscripten (GH-31757) #31757 bpo-40280: Skip dysfunctional pipe tests on Emscripten (GH-31770) #31770 bpo-40280: Skip more tests/features that don't apply to Emscripten (GH-31791) #31791 bpo-40280: Disable AF_UNIX, AF_PACKET, SO_REUSE* on Emscripten #31829 bpo-40280: select: Use NULL for empty fdset (GH-31865) #31865 bpo-40280: Detect presence of time.tzset and thread_time clock (GH-31898) #31898 bpo-40280: Skip wakeup_fd pipe tests on Emscripten (GH-31909) #31909 bpo-40280: Skip more tests on Emscripten (GH-31947) #31947 bpo-40280: Skip socket, fork, subprocess tests on Emscripten #31986 bpo-40280: Add wasm32-emscripten and wasm32-wasi SOABI #32095 bpo-40280: Emscripten fork_exec now fails early (GH-32224) #32224 bpo-40280: Add debug Emscripten flavors (GH-32233) #32233 bpo-40280: Enable most file-at() and nanosleep APIs again (GH-32238) #32238 bpo-40280: Detect if WASM platform supports threading #32243 bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) #32253 bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266) #32266 bpo-40280: Add limited Emscripten REPL (GH-32284) #32284 bpo-40280: Detect missing threading on WASM platforms (GH-32352) #32352 bpo-40280: WASM defaults to no dynamic linking (GH-32360) #32360 bpo-40280: WASM docs and smaller browser builds (GH-32412) #32412
Dependencies	bpo-23325: Turn SIG_DFL and SIG_IGN into functions bpo-33393: update config.guess and config.sub bpo-41498: Undefinied _Py_Sigset_Converter function when HAVE_SIGSET_T not set bpo-45881: Cross compiling on Linux is untested, undocumented, and broken bpo-45886: Fix Program/freeze_module for cross compiling Python bpo-45898: ctypes cfield.c defines duplicate ffi_type* symbols bpo-46383: _zoneinfo module_free has invalid function signature bpo-46390: Multiple test failures on Alpine 3.15 / musl-1.2.2-r7 bpo-46408: signal module wrongly relies on small int singletons bpo-47162: Add call trampoline to work around bad fpcasts on Emscripten bpo-47176: Interrupt handling for wasm32-emscripten builds without pthreads bpo-47196: Function pointer cast in test_imp bpo-47197: ctypes mishandles `void` return type

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-14.07:11:35.905>
labels = ['type-feature', 'build', '3.11']
title = 'Consider supporting emscripten/webassembly as a build target'
updated_at = <Date 2022-04-08.10:17:32.793>
user = 'https://bugs.python.org/SimonBiggs'

bugs.python.org fields:

activity = <Date 2022-04-08.10:17:32.793>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2020-04-14.07:11:35.905>
creator = 'Simon Biggs'
dependencies = ['23325', '33393', '41498', '45881', '45886', '45898', '46383', '46390', '46408', '47162', '47176', '47196', '47197']
files = []
hgrepos = []
issue_num = 40280
keywords = ['patch']
message_count = 65.0
messages = ['366369', '366374', '366375', '366379', '369598', '394081', '406985', '406986', '407011', '407012', '407013', '407014', '407015', '407018', '407020', '407021', '407030', '407046', '407048', '407049', '407052', '407080', '407264', '407284', '407484', '407498', '407519', '407524', '407541', '408855', '410138', '410400', '410423', '410554', '410577', '410647', '410727', '410743', '411039', '411514', '411560', '411847', '412595', '414748', '414769', '414780', '414845', '414940', '415136', '415259', '415297', '415398', '415757', '415976', '416494', '416509', '416541', '416542', '416579', '416681', '416763', '416865', '416916', '416933', '416934']
nosy_count = 14.0
nosy_names = ['barry', 'brett.cannon', 'terry.reedy', 'christian.heimes', 'mcepl', 'pmpp', 'rdb', 'ethan smith', 'corona10', 'miss-islington', 'Roman Yurchak', 'xtreak', 'erlendaasland', 'Simon Biggs']
pr_nums = ['29771', '29784', '29789', '29794', '29795', '29834', '29842', '29872', '29873', '29884', '29887', '29892', '29893', '29984', '30494', '30495', '30538', '30552', '30559', '30597', '30601', '30615', '30620', '30622', '30722', '30858', '30930', '31050', '31757', '31770', '31791', '31829', '31865', '31898', '31909', '31947', '31986', '32095', '32224', '32233', '32238', '32243', '32253', '32266', '32284', '32352', '32360', '32412']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40280'
versions = ['Python 3.11']

Reacted by Felipe S. S. Schneider and Yisi

Reacted by Felipe S. S. Schneider and Rui Lopes

Reacted by Felipe S. S. Schneider

Activity

SimonBiggs commented on Apr 14, 2020
SimonBiggsmannequin
on Apr 14, 2020
MannequinAuthor
More actions
Since asm.js came on the scene, and now Web Assembly people have created CPython patches to support building CPython with emscripten. See:

https://github.com/PeachPy/EmCPython -- Python 2.7

https://github.com/dgym/cpython-emscripten/tree/master/3.5.2/patches -- Python 3.5.2

https://github.com/iodide-project/pyodide/tree/master/cpython/patches -- Python 3.7.4

To ease the compiling of CPython with emscripten it would be helpful if patches that achieved these ends for the compiling to Web Assembly with emscripten were built into the upstream source repository itself.

If web assembly were to became a supported compilation target of the upstream CPython repository this would significantly reduce the friction of allowing CPython, and the latest CPython, to become a language readily usable within the browser.

Cheers,
Simon
Reacted by Marcel Guinhos and Felipe S. S. Schneider
SimonBiggs
added
3.9 (EOL)end of lifeend of life

buildThe build process and cross-buildThe build process and cross-build

type-featureA feature request or enhancementA feature request or enhancement
on Apr 14, 2020
serhiy-storchaka commented on Apr 14, 2020
serhiy-storchaka
on Apr 14, 2020
Member
More actions
Do you want to provide a pull request?
pmp-p commented on Apr 14, 2020
pmp-pmannequin
on Apr 14, 2020
Mannequin
More actions
you can add

https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset -- Python 3.8.x

(wasm not asm.js, clang-10+ required)

demo https://pmp-p.github.io/python-next/test.html

CPython can already run in the browser with very little patching, but major issues are :

asyncify'ing the whole wasm VM to have pre-emption over cPython's one to prevent blocking I/O slows down things *a lot* (10x)
=> (very?) bad user experience.

the size of vm + stdlib ~ 30 MiB and wasm compilation time.
=> bad user experience on first load or slow connexion.

the lack of threading in wasm MinimumViableProduct specification (but this is the browser standard for now), that leads to rewrite bits of stdlib ( like eg asyncio module )
=> adding more maintenance burden on stdlib (!)

i tested them all and my personnal opinion is : I can see no use case that would favour "stock" cPython wasm versus a blazing fast MicroPytho (or pycopy) wasm flavour or supercharged full stack pyodide.
SimonBiggs commented on Apr 14, 2020
SimonBiggsmannequin
on Apr 14, 2020
MannequinAuthor
More actions
Hi pmp-p and Serhiy,

I'd be more than happy to attempt a pull request, but I imagine a change
such as this needs to be discussed first, trying not to "rush to make a
patch" (https://www.youtube.com/watch?v=voXVTjwnn-U&feature=youtu.be&t=2546).
Also, I doubt I will do a good job of it... but I am more than happy to try.

A note regarding "supercharged full stack pyodide", potentially without
efforts such as upstreaming into CPython and emscripten the relevant
patches, that supercharged full stack may just unfortunately stagnate. See
pyodide/pyodide#635 (comment)

With respect to blocking when running Python as WASM, I have found running
the WebAssembly CPython within a webworker and signalling data back and
forth causes there to be no UI issues. It ends up being quite a neat set
up. Main down side right now however is the set up is currently going
stale, hence me believing reaching out like this is in the best interests
of Python going forward.

Cheers,
Simon

On Tue, 14 Apr 2020 at 18:58, pmp-p <[email protected]> wrote:

pmp-p <[email protected]> added the comment:

you can add
*
https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset
-- Python 3.8.x

(wasm not asm.js, clang-10+ required)

demo https://pmp-p.github.io/python-next/test.html

CPython can already run in the browser with very little patching, but
major issues are :

asyncify'ing the whole wasm VM to have pre-emption over cPython's one
to prevent blocking I/O slows down things *a lot* (10x)
=> (very?) bad user experience.

the size of vm + stdlib ~ 30 MiB and wasm compilation time.
=> bad user experience on first load or slow connexion.

the lack of threading in wasm MinimumViableProduct specification (but
this is the browser standard for now), that leads to rewrite bits of stdlib
( like eg asyncio module )
=> adding more maintenance burden on stdlib (!)

i tested them all and my personnal opinion is : I can see no use case that
would favour "stock" cPython wasm versus a blazing fast MicroPytho (or
pycopy) wasm flavour or supercharged full stack pyodide.

----------
nosy: +pmpp

Python tracker <[email protected]>
<https://bugs.python.org/issue40280\>
Beuc commented on May 22, 2020
Beucmannequin
on May 22, 2020
Mannequin
More actions
I've been maintaining a Python Emscripten build for the Ren'Py (game engine) web port:
https://github.com/python-emscripten/python
https://renpy.beuc.net/

I recently tackled Python3 with a minimal/embeddable approach and checking the other ports already pointed in the discussion:
https://github.com/python-emscripten/python/tree/trunk/3.8/
(2 patches, and a short pyconfig.h fix-up)

There is also a Cython module to use the Emscripten C API.

Here's a demo at:
https://www.beuc.net/python-emscripten/demo/3/
(hello-world size: 3MB, with a few common modules: 4MB)

I can provide a pull request with a first few core changes.
Should this be done on github?

cross-compilation handling appears to follow an incorrect logic, in particular by querying 'dpkg' or parsing compiler output to detect include paths -- it is the (cross-)compiler's responsibility to provide the system paths, and detecting them manually causes conflicts. I had to patch setup.py. Let me know if I missed something.

Usually cross-compiling is triggered by non-matching build-type/host-type. Here cross-compilation logic is apparently triggered when exporting _PYTHON_HOST_PLATFORM=xxx manually (it's in the Makefile but not exported, and breaks normal build when exported). Is this the way it's meant to be used?
Reacted by kristafari
ethanhs commented on May 20, 2021
ethanhsmannequin
on May 20, 2021
Mannequin
More actions
I think the first thing we should do is figure out whether we want to support Emscripten or WASI (or both).

Emscripten uses Javascript polyfills for some syscalls, while WASI makes direct calls the VM it is running in. They both can use WebAssembly for executing the code. This means Emscripten has wider API support, but WASI is lighter weight in many ways.

I think starting with patches to support Emscripten would be best, as it is easier to target, then add support for WASI later.

I think supporting WASI has a lot of value, because it can be run deterministically, which would be great for data science (Imagine a jupyter notebook that runs the same everywhere!)

One issue with WASI, and may be an issue with Emscripten, is threads. In 3.8 (or 3.9?) threadless builds were removed. However, WebAssembly's threading API is not really meant to emulate pthread, and SharedArrayBuffer, the primitive it is built on, is disabled in several browsers due to Spectre concerns.

Would patches to re-add a threadless build mode be accepted?
tiran commented on Nov 25, 2021
tiran
on Nov 25, 2021
Member
More actions
I have added wasm32/wasm64 architectures with emscripten/wasi operating system as cross-build targets. The values are based on Rust targets:

$ rustc --print target-list | grep wasm wasm32-unknown-emscripten wasm32-unknown-unknown wasm32-wasi wasm64-unknown-unknown

wasm (WebAssembly) is "native instruction set" for the JavaScript VM while wasi or emscripten provide operating system facilities like memory management and I/O.
tiran
added
3.11only security fixesonly security fixes
and removed
3.9 (EOL)end of lifeend of life
on Nov 25, 2021
177 remaining items
Load more actions
tiran
added a commit that references this issue on Aug 26, 2022
pythongh-84461: Omit resource mod and getresuid funcs on Emscripten
00a31cc
tiran
added a commit that references this issue on Sep 11, 2022
gh-84461: Omit resource mod and getresuid funcs on Emscripten (GH-96303)
a36235d
miss-islington
added 2 commits that reference this issue on Sep 11, 2022
pythongh-84461: Omit resource mod and getresuid funcs on Emscripten (p…
9d804cf
gh-84461: Omit resource mod and getresuid funcs on Emscripten (GH-96303)
8238fa9
brettcannon
added
OS-wasi
on Oct 4, 2022
brettcannon
added this to WASI to tier 2on Jun 14, 2023
github-project-automation
moved this to Todo in WASI to tier 2on Jun 14, 2023
vstinner
added 3 commits that reference this issue on Sep 2, 2023
Revert "[3.11] pythongh-84461: Fix Emscripten umask and permission is…
9689029
pythongh-84461: Fix Emscripten umask and permission issues (pythonGH-…
cb9c17d
[3.11] gh-108822: Backport libregrtest changes from the main branch (#…
79f7a4c
brettcannon
closed this as completed on Oct 19, 2023
github-project-automation
moved this from Todo to Done in WASI to tier 2on Oct 19, 2023
arhadthedev
added 2 commits that reference this issue on Dec 11, 2023
index on testlogging-noasyncore: a36235d pythongh-84461: Omit resourc…
6ea36d6
WIP on testlogging-noasyncore: a36235d pythongh-84461: Omit resource …
8611067

Metadata

Assignees

No one assigned

Labels

3.11only security fixesonly security fixesOS-wasibuildThe build process and cross-buildThe build process and cross-buildtype-featureA feature request or enhancementA feature request or enhancement

Projects

WASI to tier 2
Closed
Status
Done
Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Consider supporting emscripten/webassembly as a build target #84461

Description

Activity

SimonBiggs commented on Apr 14, 2020

serhiy-storchaka commented on Apr 14, 2020

pmp-p commented on Apr 14, 2020

SimonBiggs commented on Apr 14, 2020

Beuc commented on May 22, 2020

ethanhs commented on May 20, 2021

tiran commented on Nov 25, 2021

177 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions