GeezerVison is showing github.com — emscripten cross-compile wasm-ld: error: duplicate symbol `_Py_LibHacl_Hacl_Hash_*` · Issue #133042 · python/cpython
emscripten cross-compile wasm-ld: error: duplicate symbol _Py_LibHacl_Hacl_Hash_* #133042DescriptionBug report
Bug description:
For the emscripten build, during building of the python executable, all the LIBHACL_*_LIB_SHARED are used as linker flags, however the LIBHACL_HMAC_LIB_SHARED flag contains objects that are already contained in other LIBHACL_*_LIB_SHARED flags. This happens, because LIBHACL_MD5_OBJS, LIBHACL_SHA1_OBJS, LIBHACL_SHA2_OBJS, LIBHACL_SHA3_OBJS and LIBHACL_BLAKE2_OBJS are all included in LIBHACL_HMAC_LIB_SHARED through LIBHACL_HMAC_OBJS via the Makefile.pre.in file.
The configure script then sets LIBHACL_HMAC_LDFLAGS=LIBHACL_HMAC_LIB_${LIBHACL_LDEPS_LIBTYPE} which for a shared emscripten build leads to duplicate dependencies in the final build command which leads to duplicate symbols.
Since I don't know the purpose of the object file references in the LIBHACL_HMAC_OBJS, I do not feel comfortable just removing them, even though that change fixes the emscripten build.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Activitypicnixz commented on Apr 27, 2025 Mmh, EMScripten with shared dependencies is experimental so I should actually force static linking in this case. You will likely encounter the same issues with the other HACL* primitives as they would also have duplicated symbols. As you can see on your PR, the change is breaking for the regular builds.
I'll see what I can do (probably a similar issue as we had with WASI). Lukasdoe commented on Apr 27, 2025 ContributorAuthor More actions Thx for the quick reply! Yes, just removing the objects is just breaking the regular builds. However, this is the only part that fails my shared build.
One part of the configure script also caught my eye:
|
if test "$ac_sys_system" = "WASI"; then |
|
LIBHACL_LDEPS_LIBTYPE=STATIC |
|
AC_MSG_RESULT([static]) |
|
else |
|
LIBHACL_LDEPS_LIBTYPE=SHARED |
|
AC_MSG_RESULT([shared]) |
|
fi |
picnixz commented on Apr 27, 2025 WASI requires static linking but I'm unsure whether emscripten can have the corresponding built-in modules. If needed, I can just.. disable the HACL* primitives. I can't test locally because I have a "python.mjs" not found and I don't know how to fix it. picnixz commented on Apr 27, 2025 Question: would forcing LIBHACL_LDEPS_LIBTYPE=STATIC be sufficient on your side to fix the EMscripten build? picnixz commented on Apr 27, 2025 OK I think I know the issue. At the final makefile step, we have something like that:
-lm -lm -lm -lm -lm Modules/_decimal/libmpdec/libmpdec.a -sUSE_ZLIB -sUSE_BZIP2 -sUSE_ZLIB Modules/_hacl/Hacl_Hash_MD5.o Modules/_hacl/Hacl_Hash_SHA1.o Modules/_hacl/Hacl_Hash_SHA2.o Modules/_hacl/Hacl_Hash_SHA3.o Modules/_hacl/Hacl_Hash_Blake2s.o Modules/_hacl/Hacl_Hash_Blake2b.o Modules/_hacl/Lib_Memzero0.o Modules/_hacl/Hacl_HMAC.o Modules/_hacl/Hacl_Streaming_HMAC.o Modules/_hacl/Hacl_Hash_MD5.o Modules/_hacl/Hacl_Hash_SHA1.o Modules/_hacl/Hacl_Hash_SHA2.o Modules/_hacl/Hacl_Hash_SHA3.o Modules/_hacl/Hacl_Hash_Blake2s.o Modules/_hacl/Hacl_Hash_Blake2b.o Modules/_hacl/Lib_Memzero0.o
The first occurrences are of Hacl* files are coming from each individual modules, but the last ones are indeed coming from HMAC. It's not an issue for regular compilers but for emscripten it appears that it's an issue (though I don't know why??) I'll make it so that the configure script can figure out the duplicates. picnixz commented on Apr 27, 2025 Ok it's much more complex than I thought. Question, can you check if the following would solve your issue:
-if test "$ac_sys_system" = "WASI"; then
+if test "$ac_sys_system" = "WASI" -o "$ac_sys_system" = "Emscripten"; then I don't have time now to fix the build entirely (sorry) but I'll have a look again tomorrow. If this still fails, we should make it so that only unique .o are retained when linking the Python executable. picnixz commented on Apr 27, 2025 I'm marking it as a release blocker because I'm leaving tomorrow and won't be back before the next beta release so we need to find a solution in the next 24 hours (otherwise it's annoying for those working with Emscripten). It's not a true release blocker though but it's to keep in mind that this must be addressed before releasing the next sources. picnixz commented on May 9, 2025 Ok, so here's the story:
- Let's say I build md5 and HMAC. Everything works fine until building the interpreter actually. IOW, I can build both modules and their corresponding shared extensions without issues.
- During the linking phase for the interpreter, we're passing
MODLIBS to it. It contains all *_LDFLAGS variables. Now, in my case, I have built MD5 and HMAC. Since HMAC depends on MD5, the _LDFLAGS variable for HMAC contains the MD5 objects, that is Modules/_hacl/Hacl_Hash_MD5.o. So MODULE__MD5_LDFLAGS is Modules/_hacl/Hacl_Hash_MD5.o while MODULE__HMAC_LDFLAGS contains Modules/_hacl/Hacl_Hash_MD5.o.
- In
make python.mjs, we will have a duplicated Modules/_hacl/Hacl_Hash_MD5.o entry, and emcc complains at this point.
So how can I fix this? well, the dirty way to do it is to do everything normally, and only for python.mjs should we re-compute MODLIBS correctly, eliminating the duplicated .o. While I can do it using GNU Make, it's quite hard to do it for a portable Makefile.
An alternative I had in mind is to install, alongside libpython.so, libpythonhacl.so as a dynamic library. All other vendored libraries are actually statically linked, but we wanted to get rid of the static archives of libhacl as they were annoying for freeze (why does freeze work for the rest is still a mystery to me).
So I need some help from someone who would know the build systems even more. We can keep the status quo of not having a HACL* HMAC for Emscripten (we don't have OpenSSL and so Emscripten would rely on HACL* MD5 & co + Python implementation of HMAC, which is fine IMO). OTOH, I would really be interested in being able to make it available on Emscripten without changing the entire build configuration... hoodmane commented on May 9, 2025 Thanks for looking into this @picnixz ! Is this broken also natively if we statically link everything? Why is emscripten specifically affected? If making it dynamically load this fixes the problem then that'd be okay I think. picnixz commented on May 10, 2025
Is this broken also natively if we statically link everything
I wasn't able to make it work properly for that. The issue seems to be the fact that I'm specifying objects twice, whether it's static or dynamic.
Why is emscripten specifically affected?
In Emscripten, apparently, the compiler complains if the linker flags contain duplicated .o files, but gcc doesn't seem to complain. I don't know why WASM isn't affected, but maybe it's because it's statically linked. I'll try again.
Btw, this is orthogonal, but I also need to hack my own wasm.py script because python3 isn't picking up the correct interpreter, even if I do PYTHON=python3.12 python3.12 Tools/.... build. In make_emscripten_python, I needed to replace:
call(
["make", "--jobs", str(cpu_count()), "all"],
env=updated_env(),
quiet=context.quiet,
)by
call(
["make", "PYTHON=python3.12", "--jobs", str(cpu_count()), "all"],
env=updated_env(),
quiet=context.quiet,
)otherwise I can't compile in the end. I don't know why this is the case but I'll open a separate issue later. picnixz commented on May 10, 2025 Ok, so I confirm that static linking still fails (sorry, I forgot to regen configure). Now the issue is as follows (for instance):
wasm-ld: error: duplicate symbol: _Py_LibHacl_Hacl_Hash_SHA1_update_multi
>>> defined in Modules/_hacl/libHacl_Hash_SHA1.a(Hacl_Hash_SHA1.o)
>>> defined in Modules/_hacl/libHacl_HMAC.a(Hacl_Hash_SHA1.o)
To be precise: it's because libHacl_HMAC contains all its dependencies as well. So my alternative is the following: instead of passing the entire .a when creating python.mjs, what we can do is instead to build a static object that only contains the HMAC.a definitions, even if incomplete. However, this requires to change makesetup and other magical stuff I think and I really don't want to go down that road :c
Is there some magical flag for the compiler that I can pass in order to suppress those duplicate entry points if they are actually doing the same? hoodmane commented on May 20, 2025 hoodmane commented on May 20, 2025 hoodmane commented on May 20, 2025 I'm having trouble finding a minimal reproducer for this. I tried to make two libs with a duplicated object file:
x.c, y.c, z.c
int x(void) {
return 3;
}and similar
main.c
int x();
int y();
int z();
int main() {
x();
y();
z();
return 0;
}Compile and link
emcc -c x.c y.c z.c main.c
ar -mc liby.a x.o y.o
ar -mc libz.a x.o z.o
emcc main.o -ly -lz -L. It links fine... picnixz commented on May 20, 2025 I will be away for a few days until Friday, so I'll try to make it work on my side. I'll give more results when I'm back and hopefully I'll see what we can do. hoodmane commented on May 20, 2025 encukou commented on Jun 16, 2025 The third beta of 3.14 is tomorrow.
I'm not sure this is a deferred blocker -- it might be blocking Emscripten from becoming a Tier 3 platform, but, even T3 platforms don't necessarily block releases. Also, AFAICS, there's a workaround in place.
@hugovk, will you delay 3.14.0 for this issue? hugovk commented on Jun 16, 2025 No, I don't think this needs to delay 3.14.0 or be a deferred-blocker (but please let me know if you think it should). As a bug it can be fixed as normal.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|