GeezerVison is showing github.com — Directory listing in SimpleHTTPRequestHandler does not work well in non-UTF-8 locale · Issue #133889 · python/cpython
Directory listing in SimpleHTTPRequestHandler does not work well in non-UTF-8 locale #133889DescriptionFor directory, SimpleHTTPRequestHandler generates an index.html page containing a list of files. It uses the filesystem encoding for the page, which is reasonable, because file names are encoded with that encoding. The problem is that the directory patch, included in the title, can contain a query part of the URL, which may be not encodable with the filesystem encoding.
This causes test failure when running in non-UTF8 locale:
$ LC_ALL=uk_UA ./python -m test -vuall test_httpservers -m test_undecodable_parameter
...
test_undecodable_parameter (test.test_httpservers.SimpleHTTPServerTestCase.test_undecodable_parameter) ... ----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 48062)
Traceback (most recent call last):
File "/home/serhiy/py/cpython/Lib/socketserver.py", line 318, in _handle_request_noblock
self.process_request(request, client_address)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/serhiy/py/cpython/Lib/socketserver.py", line 349, in process_request
self.finish_request(request, client_address)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/serhiy/py/cpython/Lib/socketserver.py", line 362, in finish_request
self.RequestHandlerClass(request, client_address, self)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/serhiy/py/cpython/Lib/http/server.py", line 721, in __init__
super().__init__(*args, **kwargs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/serhiy/py/cpython/Lib/socketserver.py", line 766, in __init__
self.handle()
~~~~~~~~~~~^^
File "/home/serhiy/py/cpython/Lib/http/server.py", line 485, in handle
self.handle_one_request()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/serhiy/py/cpython/Lib/http/server.py", line 473, in handle_one_request
method()
~~~~~~^^
File "/home/serhiy/py/cpython/Lib/http/server.py", line 725, in do_GET
f = self.send_head()
File "/home/serhiy/py/cpython/Lib/http/server.py", line 769, in send_head
return self.list_directory(path)
~~~~~~~~~~~~~~~~~~~^^^^^^
File "/home/serhiy/py/cpython/Lib/http/server.py", line 874, in list_directory
encoded = '\n'.join(r).encode(enc, 'surrogateescape')
File "/home/serhiy/py/cpython/Lib/encodings/koi8_u.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_table)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in position 178: character maps to <undefined>
encoding with 'koi8-u' codec failed
----------------------------------------
ERROR
======================================================================
ERROR: test_undecodable_parameter (test.test_httpservers.SimpleHTTPServerTestCase.test_undecodable_parameter)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/serhiy/py/cpython/Lib/test/test_httpservers.py", line 559, in test_undecodable_parameter
response = self.request(self.base_url + '/?x=%bb').read()
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/serhiy/py/cpython/Lib/test/test_httpservers.py", line 131, in request
return self.connection.getresponse()
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/serhiy/py/cpython/Lib/http/client.py", line 1430, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "/home/serhiy/py/cpython/Lib/http/client.py", line 331, in begin
version, status, reason = self._read_status()
~~~~~~~~~~~~~~~~~^^
File "/home/serhiy/py/cpython/Lib/http/client.py", line 300, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
" response")
http.client.RemoteDisconnected: Remote end closed connection without response
----------------------------------------------------------------------
I suspect that there may also be issues if some files in the directory have non-decodable or the path of the directory is non-decodable, but I have not tested this yet.
Linked PRs
ActivityStanFromIreland commented on May 13, 2025 There is no good way that guarantees it will work with simple locale, so why not encode in utf-8 instead? It is becoming default in Python anyway, and I believe is the default in the majority of web browsers. We could make it optional to use system encoding, and default to the web standard? serhiy-storchaka commented on May 17, 2025 Always using the UTF-8 encoding, as well as the "xmlcharrefreplace" errors handler will fix the failure. But it will also change the page representation. In long run it may be good, but note that currently you get the same binary representation of file names independently from the locale of the server.
On other hand, I think that applying urllib.parse.unquote() to the whole URL is incorrect. The path can contain ? or # (percent-encoded in the URL), after unquoting it will look as wrong URL. Since query and fragment are ignored in any case, I think that they should not be shown in the page title. Then we will not have a problem of encoding non-encodable characters in a page. added 3 commits that reference this issue on May 17, 2025gh-133889: Only show the path of the URL in the SimpleHTTPRequestHand…
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|