GeezerVison is showing github.com — Data races in OpenSSL bindings · Issue #143756 · python/cpython
Data races in OpenSSL bindings #143756DescriptionBug report
We have some data races in our OpenSSL bindings that we were not previously detecting because we didn't compile OpenSSL with TSan (#143750):
test_load_cert_chain_thread_safety
Race between use_certificate_chain_file and SSL_CTX_set_default_passwd_cb:
WARNING: ThreadSanitizer: data race (pid=3543367)
Read of size 8 at 0x726c000007b8 by thread T6:
#0 use_certificate_chain_file /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_rsa.c:477:32 (libssl.so.3+0x80b0f) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#1 SSL_CTX_use_certificate_chain_file /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_rsa.c:587:12 (libssl.so.3+0x80a19) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#2 _ssl__SSLContext_load_cert_chain_impl /home/sgross/cpython/./Modules/_ssl.c:4595:9 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1f9b8) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
#3 _ssl__SSLContext_load_cert_chain /home/sgross/cpython/./Modules/clinic/_ssl.c.h:1833:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1f9b8)
...
Previous write of size 8 at 0x726c000007b8 by thread T4:
#0 SSL_CTX_set_default_passwd_cb /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:4497:34 (libssl.so.3+0x6c654) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#1 _ssl__SSLContext_load_cert_chain_impl /home/sgross/cpython/./Modules/_ssl.c:4640:5 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1fd64) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
#2 _ssl__SSLContext_load_cert_chain /home/sgross/cpython/./Modules/clinic/_ssl.c.h:1833:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1fd64)
test_thread_recv_while_main_thread_sends
WARNING: ThreadSanitizer: data race (pid=3543528)
Write of size 4 at 0x72880000f068 by main thread:
#0 ssl3_write_bytes /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/record/rec_layer_s3.c:285:16 (libssl.so.3+0x14050b) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#1 ssl3_write /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/s3_lib.c:4648:12 (libssl.so.3+0x3eee9) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#2 ssl_write_internal /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2604:16 (libssl.so.3+0x65007) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#3 SSL_write_ex2 /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2707:15 (libssl.so.3+0x653b5) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#4 SSL_write_ex /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2701:12 (libssl.so.3+0x65339) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#5 _ssl__SSLSocket_write_impl /home/sgross/cpython/./Modules/_ssl.c:2792:18 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2a809) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
#6 _ssl__SSLSocket_write /home/sgross/cpython/./Modules/clinic/_ssl.c.h:651:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2a809)
#7 _PyEval_EvalFrameDefault /home/sgross/cpython/Python/generated_cases.c.h:4009:35 (python+0x5129d9) (BuildId: 09bdcea4d998e789f62d66f1eea397f4c0970fe2)
...
Previous write of size 4 at 0x72880000f068 by thread T492:
#0 ssl3_read_bytes /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/record/rec_layer_s3.c:681:16 (libssl.so.3+0x141f69) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#1 ssl3_read_internal /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/s3_lib.c:4666:9 (libssl.so.3+0x3f2ca) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#2 ssl3_read /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/s3_lib.c:4689:12 (libssl.so.3+0x3f127) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#3 ssl_read_internal /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2379:16 (libssl.so.3+0x63af1) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#4 SSL_read_ex /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2407:15 (libssl.so.3+0x64225) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
#5 _ssl__SSLSocket_read_impl /home/sgross/cpython/./Modules/_ssl.c:2958:18 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2b229) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
#6 _ssl__SSLSocket_read /home/sgross/cpython/./Modules/clinic/_ssl.c.h:723:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2b229)
#7 method_vectorcall_VARARGS /home/sgross/cpython/Objects/descrobject.c:325:24 (python+0x293161) (BuildId: 09bdcea4d998e789f62d66f1eea397f4c0970fe2)
Linked PRs
ActivityZeroIntensity commented on Jan 12, 2026 load_cert_chain looks fixable. We should acquire tstate_mutex (via PySSL_BEGIN_ALLOW_THREADS) around the SSL calls.
For test_thread_recv_while_main_thread_sends, you might be interested in the history here -- specifically #137583. In short, I tried to fix these races a while ago by adding locking around OpenSSL calls, but that ended up deadlocking libraries that attempted to write to a socket while another thread was reading from it. I can't think of any way to fix that without running into that problem again. colesbury commented on Jan 13, 2026 ContributorAuthor More actions Thanks for the pointers. I think that locking around the read and write calls is probably the right way to go. python-websockets will need to change their test code, but I think their code is currently broken. That's not the kind of change we should do in a bugfix release, but I think it's appropriate in a major release. cc @gpshead for his thoughts.
Separately, I don't think merging Py_BEGIN_ALLOW_THREADS with the mutex acquisition in SSL code makes sense. The mutex should be held across all of the OpenSSL calls in an operation, while Py_BEGIN/END_ALLOW_THREADS is only for the blocking calls. ZeroIntensity commented on Jan 13, 2026 I believe @aaugustin expressed the concern that it would be very difficult to implement websockets without the concurrent read/write problem.
The mutex should be held across all of the OpenSSL calls in an operation, while Py_BEGIN/END_ALLOW_THREADS is only for the blocking calls.
Yeah, you're right. I think my original impression was that OpenSSL calls are only done under Py_BEGIN_ALLOW_THREADS blocks, but that doesn't look correct in hindsight. colesbury commented on Jan 13, 2026 ContributorAuthor More actions I wonder if we can configure the OpenSSL BIO's to always be nonblocking and rely on PySSL_select for blocking. That way we could hold the mutex across the SSL_read_ex and SSL_write_ex calls, but not the PySSL_select calls. aaugustin commented on Jan 16, 2026
python-websockets will need to change their test code
It is not a test problem, it's a production problem.
Currently, it's possible to have a thread send a message (websocket.send(msg)) while another thread blocked waiting for the next message (msg = websocket.recv()). This translates directly to the same operations on a blocking socket — blocking on reads. I expect any other full-duplex protocol implemented with blocking reads to have the same problem.
What would be the code change if that becomes impossible?
For a single connection, I guess it would just be a matter of replacing a blocking recv by a non-blocking select followed by a non-blocking recv?
For a server managing several connections, I suppose the best way is to select (or equivalent) all connections rather than once per connection, but that's a more significant change that doesn't fit well into websockets' architecture. colesbury commented on Jan 16, 2026 ContributorAuthor More actions
It is not a test problem, it's a production problem.
Thanks for the clarification.
What would be the code change if that becomes impossible? For a single connection, I guess it would just be a matter of replacing a blocking recv by a non-blocking select followed by a non-blocking recv?
You shouldn't need to change any code. My current plan is for CPython to basically do the thing you described. We'll internally configure the socket to be non-blocking so that the OpenSSL SSL_read_ex/SSL_write_ex calls are non-blocking. The SSLSocket recv()/send() calls will still block (depending on settimeout() and setblocking()) using select/poll.
We only need the lock around the SSL_read_ex/SSL_write_ex calls, not select/poll.
I don't think it'll be a big change -- this is already how settimeout() is implemented -- although I'm a little worried about implementation details leaking through something like socket.fileno(). aaugustin commented on Jan 18, 2026
My current plan is for CPython to basically do the thing you described.
That would be amazing 🤩 divVerent commented on Jun 16, 2026 Just as an example, a repro to get a crash out of the SSL_read/SSL_write data race is here, tsan or asan not needed:
import os
import socket
import ssl
import threading
import time
os.system("openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -nodes -subj /CN=localhost")
def ssl_loop(func, stop_me):
while not stop_me.is_set():
try:
func()
except ssl.SSLError as e:
if e.errno in (ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE):
continue
break
except Exception:
break
def run_ssl_server(server_sock, stop_event):
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain(certfile="cert.pem", keyfile="key.pem")
conn, _ = server_sock.accept()
ssl_conn = context.wrap_socket(conn, server_side=True)
def server_step():
c.sendall(b"S" * 4096)
c.recv(4096)
ssl_loop(server_step, stop_event)
def try_to_crash():
server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_sock.bind(('127.0.0.1', 0))
server_sock.listen(5)
stop_event = threading.Event()
threading.Thread(target=run_ssl_server, args=(server_sock, stop_event), daemon=True).start()
client_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_sock.connect(server_sock.getsockname())
context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
ssl_client = context.wrap_socket(client_sock)
ssl_client.settimeout(0.001) # Can just retry if a short timeout is hit.
# Spawn 10 reader and writer threads against the same SSL client,
# and let them run for one second.
threads = []
for _ in range(10):
t_read = threading.Thread(target=lambda: ssl_loop(lambda: ssl_client.recv(4096), stop_event))
t_write = threading.Thread(target=lambda: ssl_loop(lambda: ssl_client.sendall(b"Y" * 4096), stop_event))
threads.extend([t_read, t_write])
t_read.start()
t_write.start()
time.sleep(1)
stop_event.set()
for t in threads:
t.join(timeout=0.1)
ssl_client.close()
server_sock.close()
for i in range(100):
print(i)
try_to_crash()
print('did not crash')
crashing with heap corruption:
* thread #9, name = 'python3', stop reason = signal SIGSEGV: sent by kernel (SI_KERNEL)
* frame #0: 0x00007ffff7ca497d libc.so.6`_int_malloc(av=0x00007fffc0000030, bytes=21) at malloc.c:4217:14
frame #1: 0x00007ffff7ca57f2 libc.so.6`__libc_malloc2(bytes=21) at malloc.c:3458:12
frame #2: 0x00007ffff6a613ee libcrypto.so.3`CRYPTO_malloc at mem.c:214:11
frame #3: 0x00007ffff6a06eb5 libcrypto.so.3`err_set_debug(es=0x00007fffc0005f60, i=1, file="../ssl/record/methods/tls_common.c", line=873, fn="tls_get_more_records") at err_local.h:69:33 [inlined]
frame #4: 0x00007ffff6a06e70 libcrypto.so.3`ERR_set_debug(file="../ssl/record/methods/tls_common.c", line=873, func="tls_get_more_records") at err_blocks.c:37:5
frame #5: 0x00007ffff70dfcf9 libssl.so.3`tls_get_more_records at tls_common.c:921:9
frame #6: 0x00007ffff70de29a libssl.so.3`tls_read_record at tls_common.c:1141:15
frame #7: 0x00007ffff70d6535 libssl.so.3`ssl3_read_bytes at rec_layer_s3.c:698:19
frame #8: 0x00007ffff706ac40 libssl.so.3`ssl3_read_internal.part.0 at s3_lib.c:5131:11
frame #9: 0x00007ffff7079f1d libssl.so.3`SSL_read_ex at ssl_lib.c:2388:15
frame #10: 0x00007ffff71b0c37 _ssl.cpython-313-x86_64-linux-gnu.so`_ssl__SSLSocket_read_impl(self=0x00007ffff674c6a0, len=4096, group_right_1=<unavailable>, buffer=0x00007fffda7fbaa0) at _ssl.c:2622:18
frame #11: 0x00007ffff71b0b51 _ssl.cpython-313-x86_64-linux-gnu.so`_ssl__SSLSocket_read(self=0x00007ffff674c6a0, args=<unavailable>) at _ssl.c.h:544:20
frame #12: 0x0000000000560d03 python3`method_vectorcall_VARARGS(func=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at descrobject.c:324:24
frame #13: 0x0000000000559723 python3`_PyObject_VectorcallTstate(tstate=0x0000000000e248c0, callable=0x00007ffff727cc70, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at pycore_call.h:168:11 [inlined]
frame #14: 0x0000000000559708 python3`PyObject_Vectorcall(callable=0x00007ffff727cc70, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:327:12
frame #15: 0x000000000056ea15 python3`_PyEval_EvalFrameDefault(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at generated_cases.c.h:1850:23
frame #16: 0x00000000005dd5ea python3`_PyObject_VectorcallTstate(tstate=0x0000000000e248c0, callable=0x00007ffff6755da0, args=0x00007fffda7fbdd8, nargsf=1, kwnames=0x0000000000000000) at pycore_call.h:168:11 [inlined]
frame #17: 0x00000000005dd5b7 python3`method_vectorcall(method=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at classobject.c:71:20
frame #18: 0x00000000006f7c10 python3`thread_run(boot_raw=0x0000000000d47a60) at _threadmodule.c:343:21
frame #19: 0x000000000069f908 python3`pythread_wrapper(arg=<unavailable>) at thread_pthread.h:242:5
frame #20: 0x00007ffff7c95dc9 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:448:8
frame #21: 0x00007ffff7d14d88 libc.so.6`__clone3 at clone3.S:78
As such, this probably should be fixed - I guess the idea above to basically do the SSL_read/SSL_write calls under the mutex, but nonblocking, instead waiting prior to them with select, should work - with the one caveat that select indicating there are bytes may not imply the SSL_read will actually read something, but it may just update internal state (so if the contract to the caller is that at least 1 byte must be read, it will have to loop).
Be aware that the mutex probably should be held up until the SSL_get_error call to ensure other threads can't interfere with the error value returned; so basically keep it with the `_PySSL_errno as a block, like it looks now. devdanzin commented on Aug 4, 2026 One thing worth putting on the record before a fix is written, because it rules out an approach that looks obvious: @critical_section provides no mutual exclusion across a GIL-released region.
Python/pystate.c:2323, in the detach path:
if (tstate->critical_section != 0) {
_PyCriticalSection_SuspendAll(tstate);
}So any critical section held on entry is suspended for the duration of a Py_BEGIN_ALLOW_THREADS / PySSL_BEGIN_ALLOW_THREADS block, and re-acquired on reattach. For _ssl that means the 88 @critical_section clinic directives in the module do not protect the SSL_* call they wrap — only the Python-level bookkeeping on either side of it.
Demonstrated without a sanitizer: two threads driven through a barrier end up simultaneously inside one SSL*, with OpenSSL reporting its own internal error.
This matches the direction the thread is already going — the mutex has to be held across the OpenSSL calls, and Py_BEGIN/END_ALLOW_THREADS is only for the blocking ones — but it is worth being explicit that a fix expressed as "add @critical_section to these methods" would compile, look right, and change nothing.
Same point applies to gh-150191.
Written with AI assistance; the pystate.c coordinate was read at 4f3be1b5777 and the two-thread demonstration was run.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|