GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-91539: improve performance of get_proxies_environment by eendebakpt · Pull Request #91566 · python/cpython

Conversation

eendebakpt commented Apr 15, 2022 •

edited by bedevere-bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Improve performance of get_proxies_environment when there are many environment variables. Improvements depend on the number of environment variables, but the method is several times faster in this PR.

Fixes #91539

Performance test details

Results:

0.1634683609008789
0.024187326431274414

with

import os
import time
import urllib.request

if 0:
    os.environ={ f'{ii}': 1 for ii in range(8000)}
    os.environ.update({ f'{ii}_proxy': 1 for ii in range(30)})

def getproxies_environment():
    """Return a dictionary of scheme -> proxy server URL mappings.
    Scan the environment for variables named <scheme>_proxy;
    this seems to be the standard convention.  If you need a
    different way, you can pass a proxies dictionary to the
    [Fancy]URLopener constructor.
    """
    # in order to prefer lowercase variables, process environment in
    # two passes: first matches any, second pass matches lowercase only

    # select only environment variables which end in (after making lowercase) _proxy 
    candidate_names = [name for name in os.environ.keys() if len(name)>5 and name[-6]=='_'] # fast selection of candidates
    environment = [(name, os.environ[name], name.lower()) for name in candidate_names if name[-6:].lower()=='_proxy'] 

    proxies = {}
    for name, value, name_lower in environment:
        if value and name_lower[-6:] == '_proxy':
            proxies[name_lower[:-6]] = value
    # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
    # (non-all-lowercase) as it may be set from the web server by a "Proxy:"
    # header from the client
    # If "proxy" is lowercase, it will still be used thanks to the next block
    if 'REQUEST_METHOD' in os.environ:
        proxies.pop('http', None)
    for name, value, name_lower in environment:
        if name[-6:] == '_proxy':
            if value:
                proxies[name_lower[:-6]] = value
            else:
                proxies.pop(name_lower[:-6], None)
    return proxies

nn=400
t0=time.time()
for ii in range(nn):
    urllib.request.getproxies_environment()
dt=time.time()-t0
print(dt)

t0=time.time()
for ii in range(nn):
    getproxies_environment()
dt=time.time()-t0
print(dt)

Issue: speed up urllib.request.getproxies_environment #91539

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hroncok reacted with heart emoji

All reactions

❤️ 1 reaction


          improve performance of get_proxies_environment when there are many en…

42118e3

…vironment variables

bedevere-bot added the awaiting review label

Apr 15, 2022


          📜🤖 Added by blurb_it.

37ff8d9

eendebakpt marked this pull request as draft

April 15, 2022 11:31

eendebakpt added 4 commits

April 15, 2022 13:34


          fix case of short env name

b8de962


          Merge branch 'performance/getproxies_environment' of githubeendebakpt…

0f90945

…:eendebakpt/cpython into performance/getproxies_environment


          fix formatting

336da07


          fix whitespace

cabeb47

eendebakpt marked this pull request as ready for review

April 15, 2022 12:18

eendebakpt added 4 commits

April 15, 2022 14:41


          whitespace

c581748


          Merge branch 'main' into performance/getproxies_environment


          Merge branch 'main' into performance/getproxies_environment


          Merge branch 'main' into performance/getproxies_environment

869fa9a

carljm reviewed

May 17, 2022

View reviewed changes

Comment thread

Lib/urllib/request.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/urllib/request.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/urllib/request.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/urllib/request.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt and others added 4 commits

May 18, 2022 10:03


          Update Lib/urllib/request.py

dba0414

Co-authored-by: Carl Meyer <[email protected]>


          Update Lib/urllib/request.py

76f16ca

Co-authored-by: Carl Meyer <[email protected]>


          Update Lib/urllib/request.py

d3feb07

Co-authored-by: Carl Meyer <[email protected]>


          Update Lib/urllib/request.py

b9ab037

Co-authored-by: Carl Meyer <[email protected]>

eendebakpt force-pushed the performance/getproxies_environment branch 2 times, most recently from 0f694c9 to aeb96ea Compare

May 18, 2022 08:14


          whitespace

f961505

eendebakpt force-pushed the performance/getproxies_environment branch from aeb96ea to f961505 Compare

May 18, 2022 08:52

eendebakpt commented May 18, 2022

Copy link

Copy Markdown

Contributor Author

@carljm Thanks for the suggestion. Benchmarks show it is just as fast, and much cleaner code.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

carljm requested changes

May 18, 2022

View reviewed changes

carljm left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I missed this in first review, but the name of the function has a typo (extra underscore) in the NEWS entry.

Also suggested an added comment. (Wouldn't have bothered with this if it was the only thing, but may be worth it if you are doing one more update to fix NEWS anyway.)

Thanks for the improvements to this function!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/urllib/request.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot added awaiting core review and removed awaiting review labels

May 18, 2022

eendebakpt and others added 2 commits

May 18, 2022 18:48


          Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7W…

36e5497

…gVuA.rst

Co-authored-by: Carl Meyer <[email protected]>


          Update Lib/urllib/request.py

ca4fbd4

Co-authored-by: Carl Meyer <[email protected]>

carljm approved these changes

May 18, 2022

View reviewed changes


          Merge branch 'main' into performance/getproxies_environment

77cceab

eendebakpt commented Aug 10, 2022

Copy link

Copy Markdown

Contributor Author

@ambv As the latest core dev touching Lib/urllib/request.py, would you be able to review this PR?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt mentioned this pull request

Aug 31, 2022

Triage the open "performance" issues on cpython repo faster-cpython/ideas#440

Closed

iritkatriel requested a review from orsenthil

August 31, 2022 20:47

iritkatriel added performance

Performance or resource usage

3.12 only security fixes labels

Aug 31, 2022

eendebakpt added 2 commits

September 11, 2022 14:04


          Merge branch 'main' into performance/getproxies_environment

41b72c6


          Merge branch 'main' into performance/getproxies_environment

9db1103

orsenthil self-assigned this

Oct 5, 2022

orsenthil added needs backport to 3.10 needs backport to 3.11 only security fixes labels

Oct 5, 2022

orsenthil approved these changes

Oct 5, 2022

View reviewed changes

orsenthil left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt reacted with heart emoji

All reactions

❤️ 1 reaction

bedevere-bot added awaiting merge and removed awaiting core review labels

Oct 5, 2022

orsenthil merged commit aeb28f5 into python:main

Oct 5, 2022

miss-islington commented Oct 5, 2022

Copy link

Copy Markdown

Contributor

Thanks @eendebakpt for the PR, and @orsenthil for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot removed the awaiting merge label

Oct 5, 2022

bedevere-bot commented Oct 5, 2022

Copy link

Copy Markdown

GH-97918 is a backport of this pull request to the 3.11 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot removed the needs backport to 3.11 only security fixes label

Oct 5, 2022

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

Oct 5, 2022


          pythongh-91539: improve performance of get_proxies_environment (pytho…

24d2650

…nGH-91566)

* improve performance of get_proxies_environment when there are many environment variables

* 📜🤖 Added by blurb_it.

* fix case of short env name

* fix formatting

* fix whitespace

* whitespace

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* whitespace

* Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Carl Meyer <[email protected]>
(cherry picked from commit aeb28f5)

Co-authored-by: Pieter Eendebak <[email protected]>

bedevere-bot removed the needs backport to 3.10 label

Oct 5, 2022

bedevere-bot commented Oct 5, 2022

Copy link

Copy Markdown

GH-97919 is a backport of this pull request to the 3.10 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

Oct 5, 2022


          pythongh-91539: improve performance of get_proxies_environment (pytho…

cbc69af

…nGH-91566)

* improve performance of get_proxies_environment when there are many environment variables

* 📜🤖 Added by blurb_it.

* fix case of short env name

* fix formatting

* fix whitespace

* whitespace

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* whitespace

* Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Carl Meyer <[email protected]>
(cherry picked from commit aeb28f5)

Co-authored-by: Pieter Eendebak <[email protected]>

miss-islington added a commit that referenced this pull request

Oct 5, 2022


          gh-91539: improve performance of get_proxies_environment (GH-91566)

ac2427e

* improve performance of get_proxies_environment when there are many environment variables

* 📜🤖 Added by blurb_it.

* fix case of short env name

* fix formatting

* fix whitespace

* whitespace

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* whitespace

* Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Carl Meyer <[email protected]>
(cherry picked from commit aeb28f5)

Co-authored-by: Pieter Eendebak <[email protected]>

eendebakpt deleted the performance/getproxies_environment branch

October 5, 2022 19:23

eendebakpt restored the performance/getproxies_environment branch

October 5, 2022 19:23

eendebakpt deleted the performance/getproxies_environment branch

October 5, 2022 19:23

eendebakpt restored the performance/getproxies_environment branch

October 5, 2022 19:24

carljm added a commit to carljm/cpython that referenced this pull request

Oct 6, 2022


          Merge branch 'main' into dictwatch

4bb5d9d

* main: (66 commits)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896)
  pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383)
  pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759)
  pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251)
  pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662)
  pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644)
  pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756)
  pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778)
  pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826)
  Add re.VERBOSE flag documentation example (python#97678)
  ...

carljm added a commit to carljm/cpython that referenced this pull request

Oct 8, 2022


          Merge branch 'main' into typewatch

1af2a98

* main: (53 commits)
  pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)
  pythonGH-88050: fix race in closing subprocess pipe in asyncio  (python#97951)
  pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962)
  pythongh-95986: Fix the example using match keyword (python#95989)
  pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944)
  pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929)
  pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610)
  pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703)
  pythonGH-88968: Add notes about socket ownership transfers (python#97936)
  pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  ...

mpage pushed a commit to mpage/cpython that referenced this pull request

Oct 11, 2022


          pythongh-91539: improve performance of get_proxies_environment (pytho…

f200192

…n#91566)

* improve performance of get_proxies_environment when there are many environment variables

* 📜🤖 Added by blurb_it.

* fix case of short env name

* fix formatting

* fix whitespace

* whitespace

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

* whitespace

* Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst

Co-authored-by: Carl Meyer <[email protected]>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <[email protected]>

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Carl Meyer <[email protected]>

Reviewers

carljm

carljm approved these changes

orsenthil

orsenthil approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

orsenthil

Labels

3.12

only security fixes

performance

Performance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

speed up urllib.request.getproxies_environment

Uh oh!

There was an error while loading. Please reload this page.

6 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

eendebakpt commented Apr 15, 2022 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eendebakpt commented May 18, 2022

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eendebakpt commented Aug 10, 2022

Uh oh!

orsenthil left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Oct 5, 2022

Uh oh!

bedevere-bot commented Oct 5, 2022

Uh oh!

bedevere-bot commented Oct 5, 2022

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

eendebakpt commented Apr 15, 2022 •

edited by bedevere-bot

Loading

Uh oh!

There was an error while loading. Please reload this page.