Description

tiran

opened on Jan 9, 2022

Issue body actions

BPO	46315
Nosy	@tiran, @pmp-p, @kumaraditya303
PRs	bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) #30507 bpo-46315: Use fopencookie() to avoid dup() in _PyTokenizer_FindEncodingFilename #32033 bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266) #32266

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2022-01-09.14:47:43.508>
labels = ['type-feature', 'build', '3.11']
title = 'Add support for WebAssembly System Interface (wasm32-wasi)'
updated_at = <Date 2022-04-02.21:11:46.918>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-04-02.21:11:46.918>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2022-01-09.14:47:43.508>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 46315
keywords = ['patch']
message_count = 4.0
messages = ['410153', '410198', '410470', '416588']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'pmpp', 'kumaraditya']
pr_nums = ['30507', '32033', '32266']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46315'
versions = ['Python 3.11']

Reacted by Richard Zak, Milan Raj and Felipe S. S. Schneider

Reacted by Felipe S. S. Schneider and Rui Lopes

Reacted by Felipe S. S. Schneider

Reacted by Felipe S. S. Schneider

Reacted by Felipe S. S. Schneider

Activity

1. tiran commented on Jan 9, 2022

   

   tiran

   on Jan 9, 2022

   MemberAuthor

   More actions

   WASI is another WebAssembly platform similar to Emscripten (bpo-40280). Simply speaking Emscripten binaries (wasm32-emscripten) run inside a browser while WASI binaries target standalone runtimes like wasmtime [2][3] on the host. The lines are a bit blurry, as it is possible to run WASI binaries in the browser with help of JS-polyfills. WASI provides compile once, run anyway with JIT/AOT and sandboxing.

   WASI is still under development and is lacking several core features:

   • threading support and pthread API
   • sockets
   • signals are emulated (_WASI_EMULATED_SIGNAL and -lwasi-emulated-signal)
   • DAC APIs like chmod, umask
   • user-related APIs like pwd, grp
   • dup(), dup2(), F_DUPFD

   For 3.11 I plan to fix our use of #ifdef HAVE_FEATURE to make it easier to experiment with WASI. The pthread APIs need stubs, which I won't commit to 3.11 upstream yet.

   [1] https://wasi.dev/
   [2] https://github.com/bytecodealliance/wasmtime
   [3] https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
2. tiran

   added

   3.11only security fixesonly security fixes

   on Jan 9, 2022
3. tiran

   self-assigned this

   on Jan 9, 2022
4. tiran

   added

   buildThe build process and cross-buildThe build process and cross-build

   type-featureA feature request or enhancementA feature request or enhancement

   3.11only security fixesonly security fixes

   on Jan 9, 2022
5. tiran

   self-assigned this

   on Jan 9, 2022
6. tiran

   added

   buildThe build process and cross-buildThe build process and cross-build

   type-featureA feature request or enhancementA feature request or enhancement

   on Jan 9, 2022
7. tiran commented on Jan 10, 2022

   

   tiran

   on Jan 10, 2022

   MemberAuthor

   More actions

   dup() is required by _PyTokenizer_FindEncodingFilename(). I came up with this hack:

   // from wasi-libc libc-top-half/musl/src/internal/stdio_impl.h
   struct _IO_FILE {
       unsigned flags;
       unsigned char *rpos, *rend;
       int (*close)(FILE *);
       unsigned char *wend, *wpos;
       // incomplete
   };
   
   static int
   dummy_close(FILE *fp) {
       return 0;
   };
   
   static FILE *
   _Py_fdopen_borrow(int fd, const char *mode) {
       FILE *fp = fdopen(fd, mode);
       ((struct _IO_FILE*)fp)->close = dummy_close;
       return fp;
   }

   keithw on #wasi pointed out that fopencookie() can archive the same outcome without resorting to ABI-specific hack. A trivial implementation is straight forward:

   typedef union {
       void *cookie;
       int fd;
   } borrowed;

   static ssize_t
   borrow_read(void *cookie, char *buf, size_t size)
   {
   borrowed b;
   b.cookie = cookie;
   return read(b.fd, (void *)buf, size);
   }

   static ssize_t
   borrow_write(void *cookie, const char *buf, size_t size)
   {
       errno = ENOTSUP;
       return -1;
   }
   
   static int
   borrow_seek(void *cookie, off_t *off, int whence)
   {
       borrowed b;
       b.cookie = cookie;
       off_t pos;
       pos = lseek(b.fd, *off, whence);
       if (pos == (off_t)-1) {
           return -1;
       } else {
           *off = pos;
           return 0;    
       }
   }
   
   static int
   borrow_close(void *cookie)
   {
       // does not close(fd)
       return 0;
   }
   
   FILE *
   _Py_fdopen_borrow(int fd, const char *mode) {
       // only support read for now
       if (strcmp(mode, "r") != 0) {
           return NULL;
       }
       cookie_io_functions_t cookie_io = {
           borrow_read, borrow_write, borrow_seek, borrow_close
       };
     	// cookie is just the fd
       borrowed b;
       b.fd = fd;
       return fopencookie(b.cookie, "r", cookie_io);
   }
8. tiran commented on Jan 13, 2022

   

   tiran

   on Jan 13, 2022

   MemberAuthor

   More actions

   New changeset a6ca8ee by Christian Heimes in branch 'main':
   bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507)
   a6ca8ee
9. tiran commented on Apr 2, 2022

   

   tiran

   on Apr 2, 2022

   MemberAuthor

   More actions

   New changeset 3df0e63 by Christian Heimes in branch 'main':
   bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266)
   3df0e63
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022
11. 81 remaining items
12. tiran

    added 3 commits that reference this issue on Jul 23, 2022

    pythongh-90473: Fix more tests on platforms without umask

    c2450e6

    pythongh-90473: Check for HAVE_KILL in main.c:exit_sigint

    796cf42

    gh-90473: Fix more tests on platforms without umask (GH-95164)

    6839324
13. miss-islington

    added a commit that references this issue on Jul 23, 2022

    pythongh-90473: Fix more tests on platforms without umask (pythonGH-9…

    0b5686b
14. tiran

    added a commit that references this issue on Jul 23, 2022

    gh-90473: Check for HAVE_KILL in main.c:exit_sigint (GH-95165)

    d291a82
15. miss-islington

    added 3 commits that reference this issue on Jul 23, 2022

    pythongh-90473: Check for HAVE_KILL in main.c:exit_sigint (pythonGH-9…

    5cf53b1

    gh-90473: Fix more tests on platforms without umask (GH-95164)

    0d688bd

    gh-90473: Check for HAVE_KILL in main.c:exit_sigint (GH-95165)

    f15be2f
16. brettcannon

    added

    OS-wasi

    on Oct 4, 2022
17. brettcannon commented on Mar 28, 2023

    

    brettcannon

    on Mar 28, 2023

    Member

    More actions

    Since we can build for WASI without patches now, I'm closing this as finished!

    Reacted by Rui Lopes
18. brettcannon

    closed this as completedon Mar 28, 2023
19. brettcannon

    added this to WASI to tier 2on Jun 3, 2023
20. github-project-automation

    moved this to Todo in WASI to tier 2on Jun 3, 2023
21. brettcannon

    moved this from Todo to Done in WASI to tier 2on Jun 3, 2023

Issue actions