GeezerVison is showing github.com — Correct urllib.parse functions dropping the delimiters of empty URI components · Issue #82150 · python/cpython
Correct urllib.parse functions dropping the delimiters of empty URI components #82150Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-08-28.14:54:51.541>
labels = ['3.7', 'type-bug', 'library']
title = 'Correct urllib.parse functions dropping the delimiters of empty URI components'
updated_at = <Date 2020-06-10.11:21:47.483>
user = 'https://github.com/maggyero' bugs.python.org fields:
activity = <Date 2020-06-10.11:21:47.483>
actor = 'op368'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-08-28.14:54:51.541>
creator = 'maggyero'
dependencies = []
files = []
hgrepos = []
issue_num = 37969
keywords = ['patch']
message_count = 4.0
messages = ['350663', '350687', '351043', '371180']
nosy_count = 5.0
nosy_names = ['orsenthil', 'Jeremy.Hylton', 'nicktimko', 'maggyero', 'op368']
pr_nums = ['15642']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37969'
versions = ['Python 3.7'] Activitymaggyero commented on Aug 28, 2019 MannequinAuthor More actions The Python library documentation of the urllib.parse.urlunparse <https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlunparse>_ and urllib.parse.urlunsplit <https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlunsplit>_ functions states:
This may result in a slightly different, but equivalent URL, if the URL that was parsed originally had unnecessary delimiters (for example, a ? with an empty query; the RFC states that these are equivalent).
So with the <http://example.com/?\> URI::
>>> import urllib.parse
>>> urllib.parse.urlunparse(urllib.parse.urlparse("http://example.com/?"))
'http://example.com/'
>>> urllib.parse.urlunsplit(urllib.parse.urlsplit("http://example.com/?"))
'http://example.com/'But RFC 3986 <https://tools.ietf.org/html/rfc3986?#section-6.2.3>_ states the exact opposite:
Normalization should not remove delimiters when their associated component is empty unless licensed to do so by the scheme specification. For example, the URI "http://example.com/?" cannot be assumed to be equivalent to any of the examples above. Likewise, the presence or absence of delimiters within a userinfo subcomponent is usually significant to its interpretation. The fragment component is not subject to any scheme-based normalization; thus, two URIs that differ only by the suffix "#" are considered different regardless of the scheme.
So maybe urllib.parse.urlunparse ∘ urllib.parse.urlparse and urllib.parse.urlunsplit ∘ urllib.parse.urlsplit are not supposed to be used for syntax-based normalization <https://tools.ietf.org/html/rfc3986?#section-6>_ of URIs. But still, both urllib.parse.urlparse or urllib.parse.urlsplit lose the "delimiter + empty component" information of the URI string, so they report false equivalent URIs::
>>> import urllib.parse
>>> urllib.parse.urlparse("http://example.com/?") == urllib.parse.urlparse("http://example.com/")
True
>>> urllib.parse.urlsplit("http://example.com/?") == urllib.parse.urlsplit("http://example.com/")
TrueP.-S. — Is there a syntax-based normalization function of URIs in the Python library? nicktimko commented on Aug 28, 2019 maggyero commented on Sep 2, 2019 MannequinAuthor More actions @nicktimko Thanks for the historical track.
Here is a patch that solves this issue by updating the urlsplit and urlunsplit functions of the urllib.parse module to keep the '?' and '#' delimiters in URIs if present, even if their associated component is empty, as required by RFC 3986: #15642
That way we get the correct behavior:
>>> import urllib.parse
>>> urllib.parse.urlunsplit(urllib.parse.urlsplit("http://example.com/?"))
'http://example.com/?'
>>> urllib.parse.urlunsplit(urllib.parse.urlsplit("http://example.com/#"))
'http://example.com/#'Any feedback welcome. maggyero changed the title [-]urllib.parse functions reporting false equivalent URIs[/-] [+]Correct urllib.parse functions dropping the delimiters of empty URI components[/+] on Sep 11, 2019openandclose commented on Jun 10, 2020 This is a duplicate of bpo-22852 ('urllib.parse wrongly strips empty #fragment, ?query, //netloc').
Also note that three alternative solutions have already proposed.
(1) Add 'None' type to Result objects members like this one.
But it is considering not only query and fragment, but also netloc,
which may solve many other issues.
(2) Add 'has_netloc', 'has_query' and 'has_fragment' attribute.
(3) like (1), but conditional on 'allow_none' argument (similar to 'allow_fragments') erlend-aasland commented on Dec 3, 2022
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|