GeezerVison is showing github.com — test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler? · Issue #109096 · python/cpython
test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler? #109096DescriptionBug report
Every test class in test_httpservers starts a thread to run a server. CGIHTTPServerTestCase tests trigger htt.server.CGIHTTPRequestHandler.run_cgi() which executes a CGI script in a separate process. Using os.fork() if possible. But os.fork() now emits a DeprecationWarning if used in multi-threaded environment, and in future it will be an error.
$ ./python -Wa -m test -v test_httpservers -m CGIHTTPServerTestCase
...
test_accept (test.test_httpservers.CGIHTTPServerTestCase.test_accept) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
test_authorization (test.test_httpservers.CGIHTTPServerTestCase.test_authorization) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
test_cgi_path_in_sub_directories (test.test_httpservers.CGIHTTPServerTestCase.test_cgi_path_in_sub_directories) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
...
We should do something with this. First, the tests are polluted with presumably false alerts. Second, these alerts may be not false. Third, these tests will be broken in future, when the warning is turned into error.
Also, what actions are expected from users of http.server when they see a warning? How they can solve the problem on their side? If http.server is deprecated, it should be documented. If it remains, it should not produce scary warnings.
cc @gpshead
Linked PRs
Activitygpshead commented on Sep 12, 2023 At a minimum we should at least silence this particular warning in this particular CGIHTTPServerTestCase class. If this part of the test suite starts hanging on particular platform configurations due to the technically not allowed by posix issue, we can add explicit skips for those. To fix the underlying issue and not silence it:
The real bug exposed is that http.server.CGIHTTPRequestHandler has never been safe to use on a multithreaded server. This code - https://github.com/python/cpython/blob/main/Lib/http/server.py#L1172-L1192 - where the warning comes from needs to be rewritten to use subprocess.
But given that http.server is not considered a good idea to use in serious applications (aka "production") per the warning at the top of its docs, nobody has been paying attention to that kind of maintenance on it. It does not see much use. https://docs.python.org/3/library/http.server.html serhiy-storchaka commented on Sep 13, 2023 We should not simply silence deprecation warnings. We should make os.fork() only be used if it is safe (only in single-threaded program) or allow the user to control this by an option. We should rewrite tests and test the fork-using code without threads, and subprocess-using code with threads. We should document the limitations of using CGI in http.server. gpshead commented on Sep 13, 2023 I wonder if we need CGIHTTPRequestHandler at all? It isn't the '90s any more, CGI is... extremely dated and often problematic. Deprecating it could be another option. gpshead commented on Sep 13, 2023 the cgi and cgitb modules were removed in 3.13 for https://peps.python.org/pep-0594/#deprecated-modules.
Searching internally (which covers a large PyPI/github third_party corpus as well) I only found a single use of CGIHTTPRequestHandler in actual code in an internal demo test server. It was a misunderstanding, they didn't need to use that class and should've used SimpleHTTPRequestHandler as they implemented their own do_POST method. changed the title [-]test_httpservers produces DeprecationWarning[/-] [+]test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler?[/+] on Sep 13, 2023gpshead commented on Sep 13, 2023 If anyone can find actual uses of CGIHTTPRequestHandler in real world code, it'd be good to see what still uses it and why. gpshead commented on Sep 15, 2023 The deprecation is in for 3.13. I'll work up a small 3.12 specific test-only change to silence the warning on the CGIHTTPRequestHandler test there.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|