GeezerVison is showing github.com — socket.getfqdn() UnicodeDecodeError depending on LANG variable · Issue #93251 · python/cpython
socket.getfqdn() UnicodeDecodeError depending on LANG variable #93251DescriptionBug report
This code:
import locale
import socket
locale.setlocale(locale.LC_ALL, '')
socket.getfqdn()
Raise an exception if running it like this:
LANG=ru_RU.CP1251 /opt/Python-3.9.2/bin/python3 bug.py
Note the LANG. I haven't checked for which "LANG" this works or fails.
⚠️ : to exercise the problematic code (see comments for details on the problematic code path) the hostname should not be resolvable (so not in /etc/hosts, not resolvable via DNS or other methods up to /etc/nsswitch.conf hosts settings). The hostname, to reproduce the problem, can be changed on Linux via sudo hostname something-that-does-not-exist.
Traceback (most recent call last):
File "/root/t/prova.py", line 7, in <module>
socket.getfqdn()
File "/opt/Python-3.9.2/lib/python3.9/socket.py", line 791, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcd in position 0: invalid continuation byte
Your environment
Tested this on a Debian 11 bullseye with the the following Python interpreters:
- Packaged Python 3.9.2
- Compiled from source Python 3.9.2
- Compiled from source Python 3.9.13
- Compiled from source Python 3.10.4
I've encountered this bug in two independent Debian installations (with different locale settings) and in a CI system (also Debian based but unrelated settings).
Only tested in x64 systems.
Linked PRs
Activitycpina commented on May 26, 2022 In case that it helps, the stacktrace before hits the line:
errmsg = "invalid continuation byte";
In Objects/unicodeobject.c function unicode_decode_utf8.
Backtrace:
#0 unicode_decode_utf8 (s=0x555555a2e8e0 "����������� ��� ��� ������", size=26, error_handler=_Py_ERROR_UNKNOWN, errors=0x0, consumed=0x0)
at Objects/unicodeobject.c:5069
#1 0x00005555556348c4 in PyUnicode_DecodeUTF8Stateful (s=0x555555a2e8e0 "����������� ��� ��� ������", size=26, errors=0x0, consumed=0x0)
at Objects/unicodeobject.c:5141
#2 0x0000555555629dae in PyUnicode_FromStringAndSize (u=0x555555a2e8e0 "����������� ��� ��� ������", size=26) at Objects/unicodeobject.c:2267
#3 0x00005555556a0064 in do_mkvalue (p_format=0x7fffffff73b8, p_va=0x7fffffff73a0, flags=1) at Python/modsupport.c:423
#4 0x000055555569f5cd in do_mktuple (p_format=0x7fffffff73b8, p_va=0x7fffffff73a0, endchar=41 ')', n=2, flags=1) at Python/modsupport.c:264
#5 0x000055555569f737 in do_mkvalue (p_format=0x7fffffff73b8, p_va=0x7fffffff73a0, flags=1) at Python/modsupport.c:289
#6 0x00005555556a06ac in va_build_value (format=0x7ffff79bf942 "(is)", va=0x7fffffff73f0, flags=1) at Python/modsupport.c:562
#7 0x00005555556a05b0 in _Py_BuildValue_SizeT (format=0x7ffff79bf942 "(is)") at Python/modsupport.c:530
#8 0x00007ffff79b3a91 in set_gaierror (error=-2) at /root/python/Python-3.9.2/Modules/socketmodule.c:680
#9 0x00007ffff79b43b2 in setipaddr (name=0x7ffff7b6bb90 "reprotest-capture-hostname", addr_ret=0x7fffffffb600, addr_ret_size=128, af=0)
at /root/python/Python-3.9.2/Modules/socketmodule.c:1211
#10 0x00007ffff79bada7 in socket_gethostbyaddr (self=0x7ffff79de220, args=0x7ffff7b64940) at /root/python/Python-3.9.2/Modules/socketmodule.c:5822
Ignore the line numbers - In some files I had added some debug information.
I wonder (but I cannot reproduce outside Python) if the handling of the result of set_gaierror is what is causing errors depending on the locale settings. cpina commented on May 26, 2022 If it helps, gai_strerror is called (in set_gaierror) and might return a localised error:
root@reprotest-capture-hostname:~/t# cat bug.py
import locale
import socket
locale.setlocale(locale.LC_ALL, '')
print('test')
socket.getfqdn()
root@reprotest-capture-hostname:~/t# ./a.out
test
gai_strerror: Name or service not known
root@reprotest-capture-hostname:~/t# LANG=ru_RU.CP1251 ./a.out
test
gai_strerror: ����������� ��� ��� ������
root@reprotest-capture-hostname:~/t#
In set_gaierror there is:
v = Py_BuildValue("(is)", error, gai_strerror(error));
With the russian locale (and I suspect that other locales) it seems that when using PyUnicode_FromString via Py_BuildValue it cannot create the PyUnicode (see the original post) and it all fails.
Hopefully this helps to find the error. goeranu commented on Sep 19, 2024 The problem remains in the prerelease of Python 3.13 coming with Fedora 41 beta currently.
We just hit this using the Latin-1 Swedish locale sv_SE and a call of socket.gethostbyaddr with an argument that replies with an error code. That results in a UnicodeDecodeError exception rather than the expected socket.herror exception.
Our analysis came to the same conclusion as above, with the added little detail that the format string s according to the documentation does indeed interpret the string as UTF-8. But that is not what gai_strerror returns. added a commit that references this issue on Jul 26, 2026gh-93251: Decode localized socket error messages from the locale enco…
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|