Description

jaraco

opened on Jan 28, 2024

Issue body actions

Bug description:

In #114664, I've stumbled on an unexpected behavior in the timeit module.

According to the docs, the "setup" function is run once before repeating the statement under test, but that's not what I'm observing:

 ~ @ py -m timeit --setup 'import sys; assert "csv" not in sys.modules' 'import csv'
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 330, in main
    number, _ = t.autorange(callback)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 226, in autorange
    time_taken = self.timeit(number)
                 ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 180, in timeit
    timing = self.inner(it, self.timer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<timeit-src>", line 3, in inner
    import sys; assert "csv" not in sys.modules
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Compared with

 ~ @ py -m timeit --setup 'import sys; assert "csv" not in sys.modules' pass
100000000 loops, best of 5: 3.89 nsec per loop

In the first, the assertion fails in the setup. But why should it fail? If it's run before the import csv statement, then the assertion should pass. How is it that "csv" is in sys.modules during setup?

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Activity

1. jaraco

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jan 28, 2024
2. hauntsaninja commented on Jan 28, 2024

   

   hauntsaninja

   on Jan 28, 2024

   Contributor

   More actions

   I responded with specifics about what timeit is doing here in #114664 (comment) , but maybe the documentation could be improved

   Reacted by Serhiy Storchaka
3. AlexWaygood commented on Jan 28, 2024

   

   AlexWaygood

   on Jan 28, 2024

   Member

   More actions

   FWIW @jaraco, this is a script I have hanging around locally for measuring import times. It avoids the pitfalls you've correctly identified with using timeit for this task. You might find it helpful :) (Or you can use an external tool like hyperfine, as @hauntsaninja did in #114664 (comment))

   If you save it as bench_import.py, you execute it with e.g. python bench_import.py typing from the command line to measure the import time of typing.

   import sys
   import time
   import statistics
   import subprocess
   
   module = sys.argv[1]
   times = []
   
   for _ in range(500):
       ret = subprocess.run(
           [sys.executable, "-c", f"import time; t0 = time.perf_counter(); import {module}; print(time.perf_counter() - t0)"],
           check=True,
           capture_output=True,
           text=True
       )
       times.append(float(ret.stdout.strip()))
   
   print(statistics.mean(times))

   Note that Python also has a special mode builtin for profiling import time: you can run python -X importtime "import typing" to create a log file that shows exactly what's making a module slow to import. https://docs.python.org/3/using/cmdline.html#cmdoption-X
4. StanFromIreland

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Sep 6, 2025

Issue actions