GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — gh-108342: Make ssl TestPreHandshakeClose more reliable by vstinner · Pull Request #108370 · python/cpython Conversation
|
|
Non-Linux platforms have this issue because of the Without this change, test_ssl fails with "env changed" on Windows (on an idle machine): With this change, the test pass. I tested my change by stressing my Windows VM:
I cannot reproduce the issue with this method on Windows anymore with this change. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
This issue also impacts Linux machines, see buildbot failures starting at this comment: #108344 (comment) |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
I plan to schedule buildbot jobs on this PR to see if my fix works as expected. I tried to write the smallest change to fix the issue, without putting |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
On Windows x64, test_ssl failed with ENV CHANGED, but it's a different test (unrelated to this PR): |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
🤖 New build scheduled with the buildbot fleet by @vstinner for commit ff74fd7 🤖 If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Oh, on "AMD64 Fedora Stable Clang Installed PR", the bug is still there :-( Oh. But here, the bug occurred in test_https_client_non_tls_response_ignored(). |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
* In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue pythongh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * Replace socket.send() with socket.sendall() * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0].
|
It also still fails on Windows: |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
ff74fd7 to
51e1e09
Compare
|
Maybe let's not merge this until it really fixes the issue now? |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Yeah, I didn't touch test_https_client_non_tls_response_ignored() whereas this test was also fragile :-/ I rewrote my PR to make the 3 tests of TestPreHandshakeClose more reliable. The overall change is more complicated than expected, so I fixed multiple issues (more or less important). |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
I stress-tested the updated PR (commit 51e1e09) on the 3 tests on Linux on Windows. Linux: Windows: Results:
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
I don't plan to merge before I'm sure that the test is very reliable :-) |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| server_responding.set() | ||
| return True # Tell the server to stop. | ||
|
|
||
| timeout = 2.0 |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Since this is set to 2.0, all the changes to generalize the timeout in SingleConnectionTestServerThread aren't really used?
Can we at least set this to, say, 5.0 to give it some breathing room compared to the previous value?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Since this is set to 2.0, all the changes to generalize the timeout in SingleConnectionTestServerThread aren't really used?
I increased the timeout from 2 seconds to SHORT_TIMEOUT (at least 30 seconds) in the 2 other tests to make these tests more reliable (2 seconds may be too short on a busy system).
Can we at least set this to, say, 5.0 to give it some breathing room compared to the previous value?
On Windows, the test takes timeout * 2 seconds (4 seconds) to complete :-( I don't understand why the client doesn't fail with a timeout error as soon as the server closes its listener connection!? Right now, I prefer to fix the known issues of dangling threads before attempting to change this fragile timeout.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
it's also fair to just skip the test on windows if reliability is platform specific. the primary reproducer only happens on Linux anyways.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
The behavior on Windows is very different than on Linux. On Windows, test_https_client_non_tls_response_ignored() takes 4 seconds (server times out, then client times out). On Linux, it completes in 100 ms.
I didn't spend too much time to try to understand why/how. I prefer to continue running the test on Windows, unless there is a good reason to no do so. With my change, the test is reliable on Windows.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
The challenge with these is that having an environment that the failure can be reliably reproduced in is hard. So "fixed" isn't well defined. I'm satisfied that this PR improves things and that Victor has found reproducers that appear to be healthier. But a concrete "fixed" can really only be declared after weeks of diverse buildbot data even when we're 99% confident "this is the last one for real this time". It'd be fine to do the releases without this or the related preceeding test refleak fix PR in FWIW. Our refleaks checks are pedantry we impose upon ourselves that also catch things that are not actual CPython bugs as is the case here. I found and fixed a one reference cycle from the test that I saw come up during initial development. I had no way to notice or find others given I couldn't use public buildbots and I don't maintain a pile of unusually overloaded diverse OS VMs. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
For me, it was quite easy to reproduce the 2 issues on Windows, in a few seconds:
Moreover, on Linux, using |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Right now, there are many buildbots failing on these 3 test_ssl tests because of "dangling threads" (need my fix)! It's hard to notice that a test is not reliable in advance. In my experience, only shipping a fix and launching the heavy buildbot farm discovers race conditions in tests. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
I updated my PR to address @gpshead's review. @gpshead @ambv: I propose that you review and approve it, and then merge it, rather than waiting for a 3rd batch of buildbot jobs on this PR. We can use the regular buildbot process: merge the PR and then watch buildbots. Then only add backport labels to this PR once enough buildbots completed successfully. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
OK, let me just run this PR branch without Greg's original fix to see if the changed tests still trigger the issue. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Yup, it still works: |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
If the https cllient test still proves flaky, disabling or removing it is fine by me. (it wasn't a regression test)
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Buildbots are more happy with this change. Only one Windows Refleak buildbot still fails, just because it didn't run with the new code yet. The build 494 is running and it looks good as well ("test_ssl passed")! |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…ythonGH-108370) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue pythongh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
|
GH-108404 is a backport of this pull request to the 3.12 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…ythonGH-108370) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue pythongh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
|
GH-108405 is a backport of this pull request to the 3.11 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…ythonGH-108370) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue pythongh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
|
GH-108406 is a backport of this pull request to the 3.10 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…ythonGH-108370) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue pythongh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
|
GH-108407 is a backport of this pull request to the 3.9 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…ythonGH-108370) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue pythongh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
|
GH-108408 is a backport of this pull request to the 3.8 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…8370) (#108404) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue gh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
…8370) (#108405) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue gh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
…8370) (#108406) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue gh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
) (#108407) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue gh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
) (#108408) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue gh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb)
…ythonGH-108370) (python#108407) * In preauth tests of test_ssl, explicitly break reference cycles invoving SingleConnectionTestServerThread to make sure that the thread is deleted. Otherwise, the test marks the environment as altered because the threading module sees a "dangling thread" (SingleConnectionTestServerThread). This test leak was introduced by the test added for the fix of issue pythongh-108310. * Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds timeout. * SingleConnectionTestServerThread.run() catchs TimeoutError * Fix a race condition (missing synchronization) in test_preauth_data_to_tls_client(): the server now waits until the client connect() completed in call_after_accept(). * test_https_client_non_tls_response_ignored() calls server.join() explicitly. * Replace "localhost" with server.listener.getsockname()[0]. (cherry picked from commit 592bacb) Co-authored-by: Victor Stinner <[email protected]>
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.