GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-118761: Improve import time of `mimetypes` by hugovk · Pull Request #126979 · python/cpython

Conversation

hugovk commented Nov 18, 2024 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Makes import time 11 to 16 times as fast. Measured with a PGO and LTO non-debug build on macOS.

The slowest import in mimetypes is urllib.parse (taking 5,174 of mimetypes 5,448 μs = 95%).
After deferring urllib.parse, import time is 480 μs. That's 11.35 times as fast.
We could stop here. The other imports are easy enough to defer as well and have some benefit when running with -S to not import site on initialisation: 7,540 μs -> 469 μs = 16.08 times as fast.

`-X importtime`

`python.exe`

total import time: 0.010s -> 0.006s -> 0.005s

main	defer `urllib.parse`	defer rest

`mimetypes` import time: 0.005s -> 0.000s -> 0.000s

main	defer `urllib.parse`	defer rest

`python.exe -S`

total import time: 0.013s -> 0.009s -> 0.006s

main	defer `urllib.parse`	defer rest

`mimetypes` import time: 0.013s -> 0.004s -> 0.001s

main	defer `urllib.parse`	defer rest

hyperfine

`python.exe`: 24.2 ms -> 11.2 ms

`main`

❯ hyperfine --warmup 8 "./python.exe -c 'import mimetypes'"
Benchmark 1: ./python.exe -c 'import mimetypes'
  Time (mean ± σ):      24.2 ms ±  11.3 ms    [User: 15.2 ms, System: 5.7 ms]
  Range (min … max):    16.2 ms …  59.6 ms    133 runs

PR

❯ hyperfine --warmup 8 "./python.exe -c 'import mimetypes'"
Benchmark 1: ./python.exe -c 'import mimetypes'
  Time (mean ± σ):      11.2 ms ±   0.7 ms    [User: 8.1 ms, System: 2.6 ms]
  Range (min … max):    10.3 ms …  14.2 ms    170 runs

`python.exe -S`: 15.1 ms -> 8.4 ms

`main`

❯ hyperfine --warmup 8 "./python.exe -S -c 'import mimetypes'"
Benchmark 1: ./python.exe -S -c 'import mimetypes'
  Time (mean ± σ):      15.1 ms ±   0.7 ms    [User: 11.4 ms, System: 3.1 ms]
  Range (min … max):    14.3 ms …  19.2 ms    133 runs

PR

❯ hyperfine --warmup 8 "./python.exe -S -c 'import mimetypes'"
Benchmark 1: ./python.exe -S -c 'import mimetypes'
  Time (mean ± σ):       8.4 ms ±   0.4 ms    [User: 5.7 ms, System: 2.2 ms]
  Range (min … max):     7.9 ms …  12.3 ms    216 runs

Issue: Improve import time of various stdlib modules #118761

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

danielhollas reacted with rocket emoji

All reactions

🚀 1 reaction

hugovk added 4 commits

November 18, 2024 21:23


          Improve import time of mimetypes: defer urllib.parse


          Update link redirect

12fc4fe


          Improve import time of mimetypes: defer os, sys, posixpath

e6e4752


          Add blurb

0f2ccf0

hugovk requested a review from a team as a code owner

November 18, 2024 20:03

bedevere-app Bot added the awaiting core review label

Nov 18, 2024

hugovk changed the title ~~Improve import time of mimetypes~~ gh-118761: Improve import time of mimetypes

Nov 18, 2024

bedevere-app Bot mentioned this pull request

Nov 18, 2024

Improve import time of various stdlib modules #118761

Closed

asvetlov reviewed

Nov 18, 2024

View reviewed changes

asvetlov left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import time improvement looks great!
My question is: how do these import mod lines in a function body affect the function execution time itself?
IIRC import statement accuires a lock, isn't it?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

danielhollas reviewed

Nov 19, 2024

View reviewed changes

danielhollas left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 🎉

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/mimetypes.py

		read_mime_types(file) -- parse one file, return a dictionary or None
		"""

		import os

danielhollas Nov 19, 2024

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if lazy import of os module is worth it here, it is needed three times and the improvement is small and only in a special case.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

vstinner Nov 19, 2024

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python always imports os and sys at startup. Moving import sys makes sense, it's only used by _main(). But I'm not sure about moving os.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

danielhollas Nov 19, 2024

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the OP, @hugovk provides timings for when the python is run without the site.py module (python -S), in which case os and sys are not loaded I believe?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

hugovk Nov 21, 2024

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it makes a difference with python -S, compare the last three images above.

But I'm fine reverting any of these, because I expect with site.py is by far the most usual thing.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

hugovk commented Nov 21, 2024

Copy link

Copy Markdown

Member Author

Import time improvement looks great! My question is: how do these import mod lines in a function body affect the function execution time itself? IIRC import statement accuires a lock, isn't it?

They do affect execution time but in a much smaller way. Let's time these scripts that call a function a million times:

# 1.py
import urllib.parse


def thing():
    urllib.parse.urlparse("https://example.com")


for _ in range(1_000_000):
    thing()

# 2.py
def thing():
    import urllib.parse

    urllib.parse.urlparse("https://example.com")


for _ in range(1_000_000):
    thing()

Importing outside the loop is 1.06 times faster, but this is with a million loops:

❯ hyperfine --warmup 1 "python3.14 1.py" "python3.14 2.py"
Benchmark 1: python3.14 1.py
  Time (mean ± σ):      1.403 s ±  0.014 s    [User: 1.380 s, System: 0.019 s]
  Range (min … max):    1.392 s …  1.441 s    10 runs

Benchmark 2: python3.14 2.py
  Time (mean ± σ):      1.486 s ±  0.006 s    [User: 1.464 s, System: 0.020 s]
  Range (min … max):    1.477 s …  1.498 s    10 runs

Summary
  python3.14 1.py ran
    1.06 ± 0.01 times faster than python3.14 2.py

Looping 1,000 times shows no difference:

❯ hyperfine --warmup 3 "python3.14 1.py" "python3.14 2.py"
Benchmark 1: python3.14 1.py
  Time (mean ± σ):      17.5 ms ±   1.5 ms    [User: 14.1 ms, System: 2.8 ms]
  Range (min … max):    16.7 ms …  34.1 ms    152 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: python3.14 2.py
  Time (mean ± σ):      17.5 ms ±   1.0 ms    [User: 14.1 ms, System: 2.7 ms]
  Range (min … max):    16.9 ms …  28.2 ms    150 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  python3.14 2.py ran
    1.00 ± 0.10 times faster than python3.14 1.py

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

asvetlov approved these changes

Nov 21, 2024

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels

Nov 21, 2024

asvetlov commented Nov 21, 2024

Copy link

Copy Markdown

Contributor

6% slower in the worst case and no difference in a more realistic scenario sounds good to me.
Thanks!

hugovk reacted with rocket emoji

All reactions

🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner approved these changes

Nov 21, 2024

View reviewed changes

vstinner left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk reacted with rocket emoji

All reactions

🚀 1 reaction

hugovk commented Nov 21, 2024

Copy link

Copy Markdown

Member Author

Thanks for the reviews!

vstinner reacted with hooray emoji

All reactions

🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk merged commit dc7a2b6 into python:main

Nov 21, 2024

bedevere-app Bot removed the awaiting merge label

Nov 21, 2024

hugovk deleted the mimetypes-defer-imports branch

November 21, 2024 14:55

ebonnal pushed a commit to ebonnal/cpython that referenced this pull request

Jan 12, 2025


          pythongh-118761: Improve import time of mimetypes (python#126979)

883457a

Reviewers

vstinner

vstinner approved these changes

asvetlov

asvetlov approved these changes

+1 more reviewer

danielhollas

danielhollas left review comments

Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

hugovk commented Nov 18, 2024 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

-X importtime

python.exe

total import time: 0.010s -> 0.006s -> 0.005s

mimetypes import time: 0.005s -> 0.000s -> 0.000s

python.exe -S

total import time: 0.013s -> 0.009s -> 0.006s

mimetypes import time: 0.013s -> 0.004s -> 0.001s

hyperfine

python.exe: 24.2 ms -> 11.2 ms

main

PR

python.exe -S: 15.1 ms -> 8.4 ms

main

PR

Uh oh!

asvetlov left a comment

Choose a reason for hiding this comment

Uh oh!

danielhollas left a comment

Choose a reason for hiding this comment

Uh oh!

danielhollas Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

vstinner Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

danielhollas Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

hugovk Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

hugovk commented Nov 21, 2024

Uh oh!

asvetlov commented Nov 21, 2024

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

hugovk commented Nov 21, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hugovk commented Nov 18, 2024 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

`-X importtime`

`python.exe`

`mimetypes` import time: 0.005s -> 0.000s -> 0.000s

`python.exe -S`

`mimetypes` import time: 0.013s -> 0.004s -> 0.001s

`python.exe`: 24.2 ms -> 11.2 ms

`main`

`python.exe -S`: 15.1 ms -> 8.4 ms

`main`