GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-69152: Add _proxy_response_headers attribute to HTTPConnection by nametkin · Pull Request #26152 · python/cpython

Conversation

@nametkin

@nametkin nametkin commented May 15, 2021 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

It would be nice to be able to access the proxy response headers after tunneling through the proxy. Now these headers can only be obtained in debug mode.
This is necessary both for the case of a successful connection, and for the case of problems with establishing a connection (for example, in order to see the type of authentication required). Later, having access to the header Proxy-Authenticate, it will be possible to simplify the authentication on the proxy server in the dependent libraries urllib3, requests. There are problems with this now (Digest Proxy Auth, NTLM Proxy Auth, Kerberos Proxy Auth)

My proposed version is based on the idea proposed by Thomas Belhalfaoui in issue 24964. And also on the approach used in the library http.rb (here)

https://bugs.python.org/issue24964

  • Issue: Add tunnel CONNECT response headers to httplib / http.client #69152

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added the awaiting review label May 15, 2021
@the-knights-who-say-ni

the-knights-who-say-ni commented May 15, 2021

Copy link
Copy Markdown

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@OneMoreZanuda

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@the-knights-who-say-ni the-knights-who-say-ni added the CLA not signed label May 15, 2021
@terryjreedy
terryjreedy requested a review from vadmium May 15, 2021 20:54
@the-knights-who-say-ni the-knights-who-say-ni added CLA signed and removed CLA not signed labels May 16, 2021
@github-actions

github-actions Bot commented Jun 16, 2021

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Jun 16, 2021
MaxwellDupre
MaxwellDupre approved these changes Mar 24, 2022
View reviewed changes

@MaxwellDupre MaxwellDupre left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting review labels Mar 24, 2022
@nametkin

nametkin commented Mar 25, 2022

Copy link
Copy Markdown
Contributor Author

@vadmium, could you take a look at this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nametkin

nametkin commented Mar 29, 2022

Copy link
Copy Markdown
Contributor Author

@MaxwellDupre, thanks for the review!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ezio-melotti ezio-melotti removed the CLA signed label Jul 13, 2022
@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label Aug 8, 2022
@nametkin
nametkin force-pushed the add_proxy_response_headers branch from efa9212 to 3258e0a Compare October 11, 2022 09:05
@ghost

ghost commented Oct 11, 2022 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.
CLA signed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nametkin
nametkin force-pushed the add_proxy_response_headers branch 2 times, most recently from 15e31c8 to ef3244d Compare April 23, 2023 11:02
nametkin
nametkin commented Apr 23, 2023
View reviewed changes
Comment thread Lib/http/client.py Outdated
response = self.response_class(self.sock, method=self._method)
(version, code, message) = response._read_status()

self._proxy_response_headers = parse_headers(response.fp)

@nametkin nametkin Apr 23, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to how it is done in lines 337-341

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nametkin

nametkin commented Apr 23, 2023

Copy link
Copy Markdown
Contributor Author

Hi @orsenthil ! Could you take a look at this request? I see that you have worked a lot with module http/client.py. And I also found out that you were a reviewer for another PR with changes to the http/client.py. Or maybe you can recommend someone who could look at this code?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@orsenthil

orsenthil commented Apr 23, 2023

Copy link
Copy Markdown
Member

@nametkin, sure. I will.

nametkin reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@orsenthil orsenthil self-assigned this Apr 23, 2023
@arhadthedev arhadthedev changed the title bpo-24964: Add _proxy_response_headers attribute to HTTPConnection gh-69152: Add _proxy_response_headers attribute to HTTPConnection Apr 24, 2023
@nametkin
nametkin force-pushed the add_proxy_response_headers branch 6 times, most recently from 8d1ea4d to 3e0cdf9 Compare April 28, 2023 21:56
nametkin added 3 commits April 29, 2023 20:20
@nametkin
add_test
367893d
@nametkin
refactoring test
48d9119
@nametkin
add record to NEWS
ec7f577
@nametkin
nametkin force-pushed the add_proxy_response_headers branch from 3e0cdf9 to ec7f577 Compare April 29, 2023 17:20
orsenthil
orsenthil reviewed Apr 30, 2023
View reviewed changes
Comment thread Lib/http/client.py Outdated
line = response.fp.readline(_MAXLINE + 1)
if len(line) > _MAXLINE:
raise LineTooLong("header line")
if not line:

@orsenthil orsenthil Apr 30, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition if not line: isn't captured in parse_headers call.
Additional check of _MAXHEADERS is verified.
So, this isn't a strict 1:1 replacement.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@orsenthil orsenthil Apr 30, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add the if not line condition in _reader_headers method for equivalence.

However, given that _tunnel method already prints headers in line 960 in print('header:', line.decode()), I am trying to understand, what additional benefit this patch brings

The discussion in #69152 seems adding additional states to the _tunnel method, and adding debug response to those states.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@nametkin nametkin Apr 30, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orsenthil, thanks for the review!
Regarding condition if not line. It seems to me that this is now a useless part of the code. Because the only variant of the line value in which this condition will be true, if I'm not mistaken, is line = b" (because the method readline can't return None or just an empty string). And for the equality of line to the value of b'', we check a little below.

I looked at the history, these lines were added when there was no equality check for the empty binary object below. That is, at some point they made sense, but after this commit, they ceased to be necessary.

But we can add this check to the _read_headers method as a precaution. Do you think it should be done?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@nametkin nametkin Apr 30, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. As for the benefits of the changes being made. Firstly, headers are needed in the situations described in the comment Add tunnel CONNECT response headers to httplib / http.client #69152 (comment). And secondly, now headers gets into the debug-logs only if we were able to go beyond this line: https://github.com/python/cpython/blob/main/Lib/http/client.py#L948.
    But if we did not pass the authorization headers we need, then we will get an OSError without information about what happened, we will not be able to find out what authentication data is required. With my PR, I would like to take the first step towards solving the problems described here Add support for digest authentication with an HTTP proxy psf/requests#2526, Allow custom authentication (in particular NTLM) to proxies  psf/requests#1582, http proxy negotiate/gssapi authentication? requests/requests-kerberos#83

Maybe then it would be possible to start returning not OSError, but some custom error that can be catched in the library code (urllib3, requests) and automatically prepare data for authentication based on the information contained in _proxy_response_headers. Similar to how it is done in https://github.com/requests/requests-kerberos/pull/149/files.

Maybe you know ways to solve these problems in a different way, I will be glad of any ideas and suggestions.
How @vadmium's proposal will solve the problem of the lack of any information about what went wrong when creating the tunnel, I did not understand.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
orsenthil
orsenthil requested changes Apr 30, 2023
View reviewed changes

@orsenthil orsenthil left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might atleast want to make parse_headers equivalent to the lines of code being removed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot removed the awaiting core review label Apr 30, 2023
@bedevere-bot

bedevere-bot commented Apr 30, 2023

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added the awaiting changes label Apr 30, 2023
@nametkin

nametkin commented May 5, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@orsenthil, please look at the two comments above (in conversation). Can we leave the PR in its current state? Or just in case, we should add if not line condition?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nametkin
Merge branch 'main' into add_proxy_response_headers
6d59c02
@arhadthedev arhadthedev added the stdlib Standard Library Python modules in the Lib/ directory label May 5, 2023
@arhadthedev

arhadthedev commented May 5, 2023

Copy link
Copy Markdown
Member

@nametkin Could you sign the new CLA by clicking not signed button in the cpython-cla-bot's message, please? The message is above, posted Oct 11, 2022.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nametkin

nametkin commented May 5, 2023

Copy link
Copy Markdown
Contributor Author

@nametkin Could you sign the new CLA by clicking not signed button in the cpython-cla-bot's message, please? The message is above, posted Oct 11, 2022.

Done.

arhadthedev reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@arhadthedev
arhadthedev requested a review from orsenthil May 5, 2023 07:31
orsenthil
orsenthil approved these changes May 5, 2023
View reviewed changes

@orsenthil orsenthil left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me. Thanks, @nametkin

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nametkin reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting changes labels May 5, 2023
@orsenthil
Merge branch 'main' into add_proxy_response_headers
232240e
@orsenthil

orsenthil commented May 5, 2023

Copy link
Copy Markdown
Member

@gpshead - fyi.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@orsenthil
orsenthil enabled auto-merge (squash) May 5, 2023 18:29
@orsenthil
orsenthil merged commit 1afe0e0 into python:main May 5, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 5, 2023
@nametkin
nametkin deleted the add_proxy_response_headers branch May 5, 2023 19:02
gpshead
gpshead reviewed May 5, 2023
View reviewed changes
Comment thread Lib/http/client.py
try:
(version, code, message) = response._read_status()

self._proxy_response_headers = parse_headers(response.fp)

@gpshead gpshead May 5, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This always does complicated parsing of the data despite these headers not being used in 99.999% of use cases.

It'd be better to just call _read_headers(response.fp) and store that raw data privately, and add a get_proxy_response_headers() public method that triggers the parsing into data structures.

The debuglevel > 0 print loop should just loop over the raw unparsed headers.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
carljm added a commit to carljm/cpython that referenced this pull request May 5, 2023
@carljm
Merge branch 'main' into inlinecomp2
baacf5f
* main:
  pythongh-99113: Add PyInterpreterConfig.own_gil (pythongh-104204)
  pythongh-104146: Remove unused var 'parser_body_declarations' from clinic.py (python#104214)
  pythongh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (pythongh-104205)
  pythongh-104108: Add the Py_mod_multiple_interpreters Module Def Slot (pythongh-104148)
  pythongh-99113: Share the GIL via PyInterpreterState.ceval.gil (pythongh-104203)
  pythonGH-100479: Add `pathlib.PurePath.with_segments()` (pythonGH-103975)
  pythongh-69152: Add _proxy_response_headers attribute to HTTPConnection (python#26152)
  pythongh-103533: Use PEP 669 APIs for cprofile (pythonGH-103534)
  pythonGH-96803: Add three C-API functions to make _PyInterpreterFrame less opaque for users of PEP 523. (pythonGH-96849)
@nametkin nametkin mentioned this pull request May 6, 2023
gh-69152: add method get_proxy_response_headers to HTTPConnection class #104248
Merged
@thomasbelhalfaoui thomasbelhalfaoui mannequin mentioned this pull request Apr 10, 2022
Add tunnel CONNECT response headers to httplib / http.client #69152
Closed
jbower-fb pushed a commit to jbower-fb/cpython that referenced this pull request May 8, 2023
@nametkin @orsenthil @jbower-fb
pythongh-69152: Add _proxy_response_headers attribute to HTTPConnecti…
ef989ab
…on (python#26152)

Add _proxy_response_headers attribute to HTTPConnection (python#26152)

---------

Co-authored-by: Senthil Kumaran <[email protected]>
gpshead added a commit that referenced this pull request May 16, 2023
@nametkin @blurb-it @gpshead
gh-69152: add method get_proxy_response_headers to HTTPConnection cla…
85ec192
…ss (#104248)

Add http.client.HTTPConnection method get_proxy_response_headers() - this is a followup to #26152 which added it as a non-public attribute. This way we don't pre-compute a headers dictionary that most users will never access. The new method is properly public and documented and triggers full proxy header parsing into a dict only when actually called.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead left review comments
@orsenthil orsenthil orsenthil approved these changes
@vadmium vadmium Awaiting requested review from vadmium
+1 more reviewer
@MaxwellDupre MaxwellDupre MaxwellDupre approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@orsenthil orsenthil

Labels

stdlib Standard Library Python modules in the Lib/ directory

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@nametkin @the-knights-who-say-ni @orsenthil @bedevere-bot @arhadthedev @gpshead @MaxwellDupre @ezio-melotti
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.