Description

kumaraditya303

opened on Dec 26, 2024

Issue body actions

While working on #128002, I am seeing many tsan warning because of socket modules.
This issue tracks work for making it thread safe.

Linked PRs

• gh-128277: make globals variables thread safe in socket module #128286
• gh-128277: use relaxed atomics for sock_fd #128304
• gh-128277: remove unnecessary critical section from socket.close  #128305

Reacted by RUANG (James Roy)

Activity

1. kumaraditya303

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   topic-free-threading

   on Dec 26, 2024
2. picnixz

   added

   extension-modulesC modules in the Modules dirC modules in the Modules dir

   type-featureA feature request or enhancementA feature request or enhancement

   on Dec 27, 2024
3. WolframAlph commented on Dec 27, 2024

   

   WolframAlph

   on Dec 27, 2024

   Contributor

   More actions

   Hi @kumaraditya303 @picnixz , is this one free to pick up or someone is working on it?
4. picnixz commented on Dec 27, 2024

   

   picnixz

   on Dec 27, 2024

   Member

   More actions

   I think Kumar wants to do it. Since in-depth free-threading knowledge is needed and the task is likely to be long (namely, there will be probably multiple PRs which may need to be synchronized with #128002), we should first determine which parts are not thread-safe and how we can make them so.

   Reacted by RUANG (James Roy)
5. kumaraditya303 commented on Dec 27, 2024

   

   kumaraditya303

   on Dec 27, 2024

   ContributorAuthor

   More actions

   Yeah, I had already started to work on this and you are correct it is needs to be synchronized with asyncio work.
6. picnixz

   assigned

   kumaraditya303

   on Dec 27, 2024
7. picnixz commented on Dec 27, 2024

   

   picnixz

   on Dec 27, 2024

   Member

   More actions

   (Just assigning you to the issue so that users know this without having to read the thread)
8. kumaraditya303 commented on Dec 28, 2024

   

   kumaraditya303

   on Dec 28, 2024

   ContributorAuthor

   More actions

   Reproducer:

   from threading import Thread
   import socket
   
   def main():
       r = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
       def f():
           while True:
               r.detach()
   
       def g():
           while True:
               r.close()
       t1 = Thread(target=f)
       t1.start()
   
       t2 = Thread(target=g)
       t2.start()
       t1.join()
       t2.join()
   
   main()

   ❯ ./python main.py
   ==================
   WARNING: ThreadSanitizer: data race (pid=37380)
     Write of size 4 at 0x7fb00c8e0a40 by thread T1:
       #0 sock_detach Modules/socketmodule.c:3404 (_socket.cpython-314td-x86_64-linux-gnu.so+0xfbf3) (BuildId: 2d3cf6ab9784ec68dfd508837a206256d4f5da1b)
       #1 method_vectorcall_NOARGS Objects/descrobject.c:447 (python+0x181820) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #2 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x16a928) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #3 PyObject_Vectorcall Objects/call.c:327 (python+0x16aa71) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #4 _PyEval_EvalFrameDefault Python/generated_cases.c.h:960 (python+0x33a0ec) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #5 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x3681b1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #6 _PyEval_Vector Python/ceval.c:1911 (python+0x368437) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #7 _PyFunction_Vectorcall Objects/call.c:413 (python+0x16a4a1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #8 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x16ec05) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #9 method_vectorcall Objects/classobject.c:71 (python+0x16ef25) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #10 _PyVectorcall_Call Objects/call.c:273 (python+0x16c8e1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #11 _PyObject_Call Objects/call.c:348 (python+0x16cd2b) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #12 PyObject_Call Objects/call.c:373 (python+0x16cd8f) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #13 thread_run Modules/_threadmodule.c:346 (python+0x4d3151) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #14 pythread_wrapper Python/thread_pthread.h:242 (python+0x429d59) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
   
     Previous read of size 4 at 0x7fb00c8e0a40 by thread T2:
       #0 _socket_socket_close_impl Modules/socketmodule.c:3380 (_socket.cpython-314td-x86_64-linux-gnu.so+0x775b) (BuildId: 2d3cf6ab9784ec68dfd508837a206256d4f5da1b)
       #1 _socket_socket_close Modules/clinic/socketmodule.c.h:32 (_socket.cpython-314td-x86_64-linux-gnu.so+0x1037a) (BuildId: 2d3cf6ab9784ec68dfd508837a206256d4f5da1b)
       #2 method_vectorcall_NOARGS Objects/descrobject.c:447 (python+0x181820) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #3 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x16a928) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #4 PyObject_Vectorcall Objects/call.c:327 (python+0x16aa71) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #5 _PyEval_EvalFrameDefault Python/generated_cases.c.h:960 (python+0x33a0ec) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #6 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x3681b1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #7 _PyEval_Vector Python/ceval.c:1911 (python+0x368437) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #8 _PyFunction_Vectorcall Objects/call.c:413 (python+0x16a4a1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #9 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x16ec05) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #10 method_vectorcall Objects/classobject.c:71 (python+0x16ef25) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #11 _PyVectorcall_Call Objects/call.c:273 (python+0x16c8e1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #12 _PyObject_Call Objects/call.c:348 (python+0x16cd2b) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #13 PyObject_Call Objects/call.c:373 (python+0x16cd8f) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #14 thread_run Modules/_threadmodule.c:346 (python+0x4d3151) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #15 pythread_wrapper Python/thread_pthread.h:242 (python+0x429d59) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
   
     Thread T1 'Thread-1 (f)' (tid=37392, running) created by main thread at:
       #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
       #1 do_start_joinable_thread Python/thread_pthread.h:289 (python+0x42a1aa) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #2 PyThread_start_joinable_thread Python/thread_pthread.h:313 (python+0x42a3a0) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #3 ThreadHandle_start Modules/_threadmodule.c:431 (python+0x4d2ae6) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #4 do_start_new_thread Modules/_threadmodule.c:1794 (python+0x4d2cef) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #5 thread_PyThread_start_joinable_thread Modules/_threadmodule.c:1917 (python+0x4d2f5d) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #6 cfunction_call Objects/methodobject.c:551 (python+0x1ee9e6) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #7 _PyObject_MakeTpCall Objects/call.c:242 (python+0x16a6fc) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #8 _PyObject_VectorcallTstate Include/internal/pycore_call.h:165 (python+0x16a9b9) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #9 PyObject_Vectorcall Objects/call.c:327 (python+0x16aa71) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #10 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1993 (python+0x340549) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #11 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x3681b1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #12 _PyEval_Vector Python/ceval.c:1911 (python+0x368437) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #13 PyEval_EvalCode Python/ceval.c:658 (python+0x3685ac) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #14 run_eval_code_obj Python/pythonrun.c:1338 (python+0x40a462) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #15 run_mod Python/pythonrun.c:1423 (python+0x40a74d) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #16 pyrun_file Python/pythonrun.c:1256 (python+0x40b1a7) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #17 _PyRun_SimpleFileObject Python/pythonrun.c:491 (python+0x40d040) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #18 _PyRun_AnyFileObject Python/pythonrun.c:78 (python+0x40d240) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #19 pymain_run_file_obj Modules/main.c:410 (python+0x44544c) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #20 pymain_run_file Modules/main.c:429 (python+0x4455c0) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #21 pymain_run_python Modules/main.c:697 (python+0x44644c) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #22 Py_RunMain Modules/main.c:776 (python+0x446790) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #23 pymain_main Modules/main.c:806 (python+0x446847) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #24 Py_BytesMain Modules/main.c:830 (python+0x4469a6) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #25 main Programs/python.c:15 (python+0x84aeb) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
   
     Thread T2 'Thread-2 (g)' (tid=37393, running) created by main thread at:
       #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
       #1 do_start_joinable_thread Python/thread_pthread.h:289 (python+0x42a1aa) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #2 PyThread_start_joinable_thread Python/thread_pthread.h:313 (python+0x42a3a0) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #3 ThreadHandle_start Modules/_threadmodule.c:431 (python+0x4d2ae6) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #4 do_start_new_thread Modules/_threadmodule.c:1794 (python+0x4d2cef) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #5 thread_PyThread_start_joinable_thread Modules/_threadmodule.c:1917 (python+0x4d2f5d) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #6 cfunction_call Objects/methodobject.c:551 (python+0x1ee9e6) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #7 _PyObject_MakeTpCall Objects/call.c:242 (python+0x16a6fc) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #8 _PyObject_VectorcallTstate Include/internal/pycore_call.h:165 (python+0x16a9b9) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #9 PyObject_Vectorcall Objects/call.c:327 (python+0x16aa71) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #10 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2195 (python+0x341ad5) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #11 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x3681b1) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #12 _PyEval_Vector Python/ceval.c:1911 (python+0x368437) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #13 PyEval_EvalCode Python/ceval.c:658 (python+0x3685ac) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #14 run_eval_code_obj Python/pythonrun.c:1338 (python+0x40a462) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #15 run_mod Python/pythonrun.c:1423 (python+0x40a74d) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #16 pyrun_file Python/pythonrun.c:1256 (python+0x40b1a7) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #17 _PyRun_SimpleFileObject Python/pythonrun.c:491 (python+0x40d040) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #18 _PyRun_AnyFileObject Python/pythonrun.c:78 (python+0x40d240) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #19 pymain_run_file_obj Modules/main.c:410 (python+0x44544c) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #20 pymain_run_file Modules/main.c:429 (python+0x4455c0) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #21 pymain_run_python Modules/main.c:697 (python+0x44644c) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #22 Py_RunMain Modules/main.c:776 (python+0x446790) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #23 pymain_main Modules/main.c:806 (python+0x446847) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #24 Py_BytesMain Modules/main.c:830 (python+0x4469a6) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
       #25 main Programs/python.c:15 (python+0x84aeb) (BuildId: d9db3c5867c977522c3be92604eba81f48c82275)
   
   SUMMARY: ThreadSanitizer: data race Modules/socketmodule.c:3404 in sock_detach

   The sock_fd member of PySocketSockObject needs to be accessed by atomics because it can be accessed and modified concurrently by thread which may even not be holding the GIL. See also #116912

   I think the fix would be make it thread safe by adding using relaxed atomics for sock_fd member.
9. kumaraditya303

   added 2 commits that reference this issue on Dec 31, 2024

   gh-128277: use relaxed atomics for `sock_fd` (#128304)

   7c72c1f

   gh-128277: make globals variables thread safe in socket module (#128286)

   e389d6c
10. WolframAlph

    added a commit that references this issue on Dec 31, 2024

    pythongh-128277: make globals variables thread safe in socket module (p…

    51a2015
11. kumaraditya303

    added a commit that references this issue on Jan 1, 2025

    gh-128277: remove unnecessary critical section from `socket.close` (#…

    bb9d955
12. kumaraditya303

    closed this as completedon Jan 1, 2025
13. srinivasreddy

    added 3 commits that reference this issue on Jan 8, 2025

    pythongh-128277: use relaxed atomics for `sock_fd` (python#128304)

    595693f

    pythongh-128277: make globals variables thread safe in socket module (p…

    6850cab

    pythongh-128277: remove unnecessary critical section from `socket.clo…

    bc07e75

Issue actions