GeezerVison is showing github.com — http.client._MAXHEADERS = 100 limit no longer sufficient · Issue #131724 · python/cpython
http.client._MAXHEADERS = 100 limit no longer sufficient #131724DescriptionBug report
Bug description:
This hard-coded sanity check for HTTP response headers is no longer sufficient to fetch a Microsoft 365 page.
I do have a case open with Microsoft Support (TrackingID#2503130010002871), but it's not getting much traction as it's not causing problems with full web browsers.
Steps to reproduce:
import http.client
con = http.client.HTTPSConnection('outlook.office365.com')
con.request("GET", "/owa/example.edu") # any domain seems to trigger
r = con.getresponse()And that throws a HTTPException
>>> con = http.client.HTTPSConnection('outlook.office365.com')
>>> con.request("GET", "/owa/foo.bar")
>>> r = con.getresponse()
Traceback (most recent call last):
File "<python-input-8>", line 1, in <module>
r = con.getresponse()
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 1428, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 350, in begin
self.headers = self.msg = parse_headers(self.fp)
~~~~~~~~~~~~~^^^^^^^^^
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 248, in parse_headers
headers = _read_headers(fp)
File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 226, in _read_headers
raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers
>>>
It seems to be just spilling over with 101 headers. Though, not consistently. Presumably it depends upon which load balancer node is responding.
$ curl --silent -D - 'https://outlook.office365.com/owa/example.edu' | grep -E "^[a-zA-Z-]+: " | wc -l returns with 96, 99, 101, etc. headers, depending on Microsoft's mood unknown factors.
For background, it's common to use https://outlook.com/example.edu as a domain hint ("smart link") to go directly to a tenant's identity provider and avoid the "Please provide your email address" step. We have a nagios check for that, which broke recently as the number of Set-Cookie: OpenIdConnect.token.[...] variants continues to grow.
CPython versions tested on:
3.13, 3.9, 3.11
Operating systems tested on:
Linux, Windows
Linked PRs
ActivityStanFromIreland commented on Mar 25, 2025 This seems to only restrict HTTPSConnection and HTTPConnection is not affected. jmacdone commented on Mar 28, 2025 Just to help clarify, picking a new arbitrary limit allows getresponse() to complete without an exception.
>>> import http.client
>>> http.client._MAXHEADERS = 120
>>> con = http.client.HTTPSConnection('outlook.office365.com')
>>> con.request("GET", "/owa/example.edu") # any domain seems to trigger
>>> r = con.getresponse()
>>> r.read()
b'<html><head><title>Object moved</title> [...]'@StanFromIreland I don't have an explicit test case, but I think this would affect both classes. HTTPSConnection.getresponse() is inherited from HTTPConnection.getresponse()
|
class HTTPSConnection(HTTPConnection): |
vstinner commented on Mar 28, 2025 Maybe we should add a parameter to customize this limit? gostak-dd commented on May 20, 2025 I tried with both 3.11 and 3.15 and did not get an exception jmacdone commented on May 20, 2025
I tried with both 3.11 and 3.15 and did not get an exception
@GGyll Confirmed. It does indeed appear to be just under the limit today
And, for completeness, Microsoft closed my ticket a few weeks ago: "working as intended". They're happy they're setting 60+ cookies.
For the time being, my workaround is setting the module's "private" variable as show in #131724 (comment)
I suppose that could be a patch option. Rename _MAXHEADERS to MAXHEADERS and document it as part of the module's public interface. Or, maybe it's not worth bothering with just to accommodate one page from one vendor? vstinner commented on May 20, 2025
Maybe we should add a parameter to customize this limit?
I mean adding a parameter to HTTPConnection and HTTPSConnection constructors. If not specified, the default value would be http.client._MAXHEADERS. aeurielesn commented on Jul 19, 2025 Adding a parameter sounds like a good approach. I can have a look at this. added a commit that references this issue on Jul 20, 2025gh-131724: Add a new max_response_headers param to HTTP/HTTPSConnecti… encukou commented on Jul 22, 2025 Reacted by Alexander Urieles and jmacdone Reacted by Victor Stinner
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|