GeezerVison is showing github.com — test_hashlib fails when run on Free Threading with TSAN · Issue #153201 · python/cpython
test_hashlib fails when run on Free Threading with TSAN #153201DescriptionCrash report
TSAN CI job: https://github.com/python/cpython/actions/runs/28747943876/job/85377269260?pr=152717
test_hashlib fails when run on Free Threading with TSAN on the main branch.
Commands:
export CC=clang
export TSAN_OPTIONS="$PWD/log_path=san_log suppressions=$PWD/Tools/tsan/suppressions_free_threading.txt handle_segv=0"
./configure --config-cache --with-thread-sanitizer --with-pydebug --disable-gil
make -j4
./python -m test --tsan-parallel --parallel-threads=4 -j4 -W --timeout=600 --slowest
Tests output (truncated):
Using random seed: 1463135961
0:00:00 load avg: 2.65 mem: 267.1 MiB Run 2 tests in parallel using 2 worker processes (timeout: 10 min, worker timeout: 15 min)
0:00:06 load avg: 2.76 mem: 611.2 MiB [1/2] test_abc passed
0:00:14 load avg: 2.78 mem: 278.3 MiB [2/2/1] test_hashlib worker non-zero exit code (Exit code 66)
test_algorithms_available (test.test_hashlib.HashLibTestCase.test_algorithms_available) [threads=4] ... ok
test_algorithms_guaranteed (test.test_hashlib.HashLibTestCase.test_algorithms_guaranteed) [threads=4] ... ok
(...)
test_scrypt_types (test.test_hashlib.TestScrypt.test_scrypt_types) [threads=4] ... ok
test_scrypt_validate (test.test_hashlib.TestScrypt.test_scrypt_validate) [threads=4] ... ok
----------------------------------------------------------------------
Ran 88 tests in 9.698s
OK (skipped=56)
== Tests result: FAILURE ==
TSAN logs (truncated):
WARNING: ThreadSanitizer: data race (pid=30628)
Read of size 8 at 0x721c0000e3a8 by thread T456:
#0 memcmp <null> (python+0x10d49e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
#1 <null> <null> (libcrypto.so.3+0x24f2d1) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
#2 pbkdf2_hmac /home/runner/work/cpython/cpython/./Modules/clinic/_hashopenssl.c.h:1490:20 (_hashlib.cpython-316td-x86_64-linux-gnu.so+0x7299) (BuildId: 1c2fdd533a623cb5e65ca60ec9662a58487ffb8b)
Previous write of size 8 at 0x721c0000e3a8 by thread T454 (mutexes: write M0, write M1):
#0 memcpy <null> (python+0xfc022) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
#1 <null> <null> (libcrypto.so.3+0x24f0e6) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
#2 pbkdf2_hmac /home/runner/work/cpython/cpython/./Modules/clinic/_hashopenssl.c.h:1490:20 (_hashlib.cpython-316td-x86_64-linux-gnu.so+0x7299) (BuildId: 1c2fdd533a623cb5e65ca60ec9662a58487ffb8b)
Mutex M0 (0x721000000500) created at:
#0 pthread_rwlock_init <null> (python+0x103e6e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
#1 CRYPTO_THREAD_lock_new <null> (libcrypto.so.3+0x275485) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
#2 CRYPTO_THREAD_run_once <null> (libcrypto.so.3+0x27556c) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
#3 run_exec_func /home/runner/work/cpython/cpython/Objects/moduleobject.c:707:15 (python+0x374a4a) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
Mutex M1 (0x7210000008c0) created at:
#0 pthread_rwlock_init <null> (python+0x103e6e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
#1 CRYPTO_THREAD_lock_new <null> (libcrypto.so.3+0x275485) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
#2 CRYPTO_THREAD_run_once <null> (libcrypto.so.3+0x27556c) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
#3 run_exec_func /home/runner/work/cpython/cpython/Objects/moduleobject.c:707:15 (python+0x374a4a) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
Full TSAN logs: tsan_full.log.
Linked PRs
Activityvstinner commented on Jul 6, 2026 I can reproduce the issue with "OpenSSL 3.5.7 9 Jun 2026" on Fedora 44 with commands:
export TSAN_OPTIONS="$PWD/log_path=san_log suppressions=$PWD/Tools/tsan/suppressions_free_threading.txt handle_segv=0"
./python -u -m test --parallel-threads=4 -v test_hashlib -m KDFTests 2>&1|tee log
It's the test_pbkdf2_hmac_c() test which triggers the data race. vstinner commented on Jul 6, 2026 vstinner commented on Jul 6, 2026 I tried but failed (so far) to reproduce the issue with OpenSSL built with TSAN.
I applied this patch (extracted from #143750):
diff --git a/Tools/ssl/multissltests.py b/Tools/ssl/multissltests.py
index 1a213187b89..d5e38993d97 100755
--- a/Tools/ssl/multissltests.py
+++ b/Tools/ssl/multissltests.py
@@ -163,6 +163,12 @@
dest='keep_sources',
help="Keep original sources for debugging."
)
+parser.add_argument(
+ '--tsan',
+ action='store_true',
+ dest='tsan',
+ help="Build with thread sanitizer. (Disables fips in OpenSSL 3.x)."
+)
class AbstractBuilder(object):
@@ -317,6 +323,8 @@ def _build_src(self, config_args=()):
"""Now build openssl"""
log.info("Running build in {}".format(self.build_dir))
cwd = self.build_dir
+ if self.args.tsan:
+ config_args += ("-fsanitize=thread",)
cmd = [
"./config", *config_args,
"shared", "--debug",I ran commands:
./Tools/ssl/multissltests.py --steps=library --base-directory ~/multissl --openssl 3.5.7 --system Linux --tsan
export LD_LIBRARY_PATH=~/multissl/openssl/3.5.7/lib/
./configure --with-thread-sanitizer --with-pydebug --disable-gil --with-openssl=/home/vstinner/multissl/openssl/3.5.7
make clean
make
./python -u -m test --parallel-threads=4 -v test_hashlib -m KDFTests 2>&1|tee log
Sadly, so far I failed to reproduce the test_hashlib failure this way. kumaraditya303 commented on Jul 7, 2026 I think it might be a false-positive race because openssl was not compiled with tsan enabled. sobolevn commented on Jul 7, 2026 picnixz commented on Jul 7, 2026 The race could be in get_openssl_evp_md_by_utf8name which is a known failure however that I recently fixed #153019. I want to emit a nice commit message which I unfortunately didn't have time to write for now but it was also very hard to reproduce in general. kumaraditya303 commented on Jul 10, 2026 sobolevn commented on Jul 10, 2026 So we can merge #153201 and backports :)
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|