Description

bohea

opened on Apr 14, 2022

Issue body actions

in module urllib.request， function getproxies_environment （https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L2493）

iterate over all environment variables twice

I made a test:

1. create 2500 environment variables
2. call and timing getproxies_environment()
3. the result is: getproxies_environment() take more than 50ms, which is a big overhead

so if there are many environment variables(like 10 thousands) in a machine, packages depend on function urllib.request.getproxies_environment (like requests does) shall be very slow

is 10 thousands environment variables reasonable？

yes it is, for example, in k8s cluster，kubelete may add a set of environment variables all Services(could be 10 thousands even 100 thousands), in purpose of service discovery

so would it be ok that getproxies_environment don't iterate over all environment variables?

Linked PRs

• gh-91539: Small performance improvement of urrlib.request.getproxies_environment() #108771

Activity

1. AlexWaygood

   added

   performancePerformance or resource usagePerformance or resource usage

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Apr 14, 2022
2. eendebakpt commented on Apr 15, 2022

   

   eendebakpt

   on Apr 15, 2022

   Contributor

   More actions

   @bohea The following code reduces the computation time by a factor 3 or more on my system.

   def getproxies_environment():
       """Return a dictionary of scheme -> proxy server URL mappings.
       Scan the environment for variables named <scheme>_proxy;
       this seems to be the standard convention.  If you need a
       different way, you can pass a proxies dictionary to the
       [Fancy]URLopener constructor.
       """
       # in order to prefer lowercase variables, process environment in
       # two passes: first matches any, second pass matches lowercase only
   
       # select only environment variables which end in (after making lowercase) _proxy 
       candidate_names = [name for name in os.environ.keys() if name[-6:]=='_'] # fast selection of candidates
       environment = [(name, os.environ[name], name.lower()) for name in candidate_names if name[-6:].lower()=='_proxy'] 
       
       proxies = {}
       for name, value, name_lower in environment:
           if value and name_lower[-6:] == '_proxy':
               proxies[name_lower[:-6]] = value
       # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
       # (non-all-lowercase) as it may be set from the web server by a "Proxy:"
       # header from the client
       # If "proxy" is lowercase, it will still be used thanks to the next block
       if 'REQUEST_METHOD' in os.environ:
           proxies.pop('http', None)
       for name, value, name_lower in environment:
           if name[-6:] == '_proxy':
               if value:
                   proxies[name_lower[:-6]] = value
               else:
                   proxies.pop(name_lower[:-6], None)
       return proxies
   

   Does this work for you? If so, then I will make a PR
3. iritkatriel

   added

   3.12only security fixesonly security fixes

   on Sep 1, 2022
4. orsenthil

   closed this as completedin #91566on Oct 5, 2022
5. orsenthil

   added a commit that references this issue on Oct 5, 2022

   gh-91539: improve performance of get_proxies_environment (#91566)

   aeb28f5
6. miss-islington

   added 3 commits that reference this issue on Oct 5, 2022

   pythongh-91539: improve performance of get_proxies_environment (pytho…

   24d2650

   pythongh-91539: improve performance of get_proxies_environment (pytho…

   cbc69af

   gh-91539: improve performance of get_proxies_environment (GH-91566)

   ac2427e
7. carljm

   added 2 commits that reference this issue on Oct 6, 2022

   Merge branch 'main' into dictwatch

   4bb5d9d

   Merge branch 'main' into typewatch

   1af2a98
8. mpage

   added a commit that references this issue on Oct 11, 2022

   pythongh-91539: improve performance of get_proxies_environment (pytho…

   f200192
9. hauntsaninja

   added a commit that references this issue on Jan 15, 2024

   gh-91539: Small performance improvement of urrlib.request.getproxies_…

   5094690
10. kulikjak

    added a commit that references this issue on Jan 22, 2024

    pythongh-91539: Small performance improvement of urrlib.request.getpr…

    ed5889d
11. aisk

    added a commit that references this issue on Feb 11, 2024

    pythongh-91539: Small performance improvement of urrlib.request.getpr…

    9717187
12. Glyphack

    added a commit that references this issue on Sep 2, 2024

    pythongh-91539: Small performance improvement of urrlib.request.getpr…

    e72ae11

Issue actions