GeezerVison is showing github.com — [CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets · Issue #103848 · python/cpython
[CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets #103848DescriptionBackground
RFC 3986 defines a host as follows
host = IP-literal / IPv4address / reg-name
Where
IP-literal = "[" ( IPv6address / IPvFuture ) "]"
reg-name = *( unreserved / pct-encoded / sub-delims )
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet
WhatWG says that "A valid host string must be a valid domain string, a valid IPv4-address string, or: U+005B ([), followed by a valid IPv6-address string, followed by U+005D (])."
The Bug
This is code from Lib/urllib/parse.py:196-208 used for retrieving the hostname from the netloc
@property
def _hostinfo(self):
netloc = self.netloc
_, _, hostinfo = netloc.rpartition('@')
_, have_open_br, bracketed = hostinfo.partition('[')
if have_open_br:
hostname, _, port = bracketed.partition(']')
_, _, port = port.partition(':')
else:
hostname, _, port = hostinfo.partition(':')
if not port:
port = None
return hostname, portIt will incorrectly retrieve IPv4 addresses and regular name hosts from inside brackets. This is in violation of both specifications.
Minimally reproducible example:
from urllib.parse import urlsplit
parsedURL = urlsplit('scheme://user@[regname]/Path')
print(parsedURL.hostname) # Prints 'regname'Your environment
- CPython versions tested on:
- Operating system and architecture:
Linked PRs
Activityadded a commit that references this issue on May 10, 2023gh-103848: Adds checks to ensure that bracketed hosts found by urlspl… mjpieters commented on Jun 6, 2023 As implemented, the URL scheme://foobar.[::1]/ is considered correct, but results in the hostname value of ::1. The hostname portion of the netloc is invalid however, as it doesn't start with [. Shouldn't the parser at least validate that the [ bracket is the first character or directly following a @? MarcoGlauser commented on Jun 12, 2023 This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.
Running this code:
from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")In 3.11.3 it works as expected:
ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')
In 3.11.4 it throws an exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.11/urllib/parse.py", line 395, in urlparse
splitresult = urlsplit(url, scheme, allow_fragments)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/urllib/parse.py", line 500, in urlsplit
_check_bracketed_host(bracketed_host)
File "/usr/local/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host
ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: '@host.com' does not appear to be an IPv4 or IPv6 addressmjpieters commented on Jun 12, 2023
This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.
Running this code:
from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")In 3.11.3 it works as expected:
ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')
That this worked before was a bug. urlparse() was violating the various and miriad URL standards before; [ and ] are reserved characters with meaning in the netloc portion of a URL (the part between the scheme:// and /path) and must be URL encoded. See the WhatWG living URL standard percent-encoding section; for the userinfo part (username + password), you need to encode:
- U+002F (
/), U+003A (:), U+003B (;), U+003D (=), U+0040 (@), U+005B ([) to U+005E (^), inclusive, and U+007C (|).
- U+003F (
?), U+0060 (`), U+007B ({), and U+007D (}) (path percent-encode set).
- U+0020 SPACE, U+0022 (
"), U+0023 (#), U+003C (<), and U+003E (>) (query percent-encode set).
- U+0000 NULL to U+001F INFORMATION SEPARATOR ONE, inclusive, and all codepoints past U+007E (
~) (C0 control percent-encode set)
So this works:
>>> from urllib.parse import urlparse
>>> urlparse("https://user:some%5Dpassword%[email protected]")
ParseResult(scheme='https', netloc='user:some%5Dpassword%[email protected]', path='', params='', query='', fragment='')
>>> _.password
'some%5Dpassword%5B'
The password is still URL encoded, use urllib.parse.unquote() for that.
Further references:
Reacted by Marco Glauser, Gregory P. Smith and RacerZ MarcoGlauser commented on Jun 12, 2023 Thank you for the clarification! This makes a lot of sense :) Jc2k commented on Jun 28, 2023 I'm also hitting "ValueError: 'xxxxx' does not appear to be an IPv4 or IPv6 address" on code that was working.
In my case I have no control over the input data, it's background noise from the internet that i'm parsing and analyzing. Some of the crap in my dataset is very much not standards compliant. None the less, as it was working and now isn't i figured i should 👍 before i start vendoring old versions of the stdlib into my project 😆 Beuc commented on Mar 7, 2025
My backport of fix for Python 3.6 (which with minimal changes applies to 3.4 as well, and with a lot of changes, including injecting ipaddress from https://github.com/phihag/ipaddress into the Python standard library, into 2.7) is at openSUSE-Python@4f2496b
Hi @mcepl. I'm currently evaluating how to fix this in older Debian dists for 2.7, and FYI it seems the version from OpenSUSE Leap 15 (150000.74.1) doesn't fully fix it:
$ PYTHONPATH=/root/rpmbuild/BUILD/Python-2.7.18/Lib \
python2 -c 'import urlparse; print(urlparse.urlparse("https://user:some]password[@host.com"))'
ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')
(expected: ValueError)
It seems CVE-2024-11168-validation-IPv6-addrs.patch fixes the "http://" special case but not the general case (which handles "httpS://"). I didn't check the 3.6 patch closely but it seems it's the opposite situation there.
After fixing the above I also get a couple test_urlparse errors, related to passing non-unicode data to the ipaddress backport. Casting to Unicode ipaddress.ip_address(unicode(hostname)) appears to tame the testsuite. mcepl commented on Mar 8, 2025
(expected: ValueError) It seems CVE-2024-11168-validation-IPv6-addrs.patch fixes the "http://" special case but not the general case (which handles "httpS://"). I didn't check the 3.6 patch closely but it seems it's the opposite situation there.
After fixing the above I also get a couple test_urlparse errors, related to passing non-unicode data to the ipaddress backport. Casting to Unicode ipaddress.ip_address(unicode(hostname)) appears to tame the testsuite.
Do you have a corrected patch? I am all ears! Beuc commented on Mar 8, 2025 mcepl commented on Mar 13, 2025
Thank you. In the end I went just with the modification of the original patch, I like how similar the patch is on all versions of the interpreter.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|