GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-137855: Improve import time of `sqlite3` by Wulian233 · Pull Request #131796 · python/cpython

Conversation

Wulian233 commented Mar 27, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Improve import time of sqlite3 by 1.5x faster

Benchmark on Windows10, CPython 3.14.0b1

D:\Python314>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'
Benchmark 1: ./python -c 'from sqlite3 import *'
  Time (mean ± σ):     176.8 µs ± 179.3 µs    [User: 1985.8 µs, System: 2886.4 µs]
  Range (min … max):     0.0 µs … 1018.1 µs    406 runs

Benchmark 2: ./python -c 'from sqlite3_new import *'
  Time (mean ± σ):     245.4 µs ± 200.7 µs    [User: 914.9 µs, System: 2002.8 µs]
  Range (min … max):     0.0 µs … 1787.5 µs    401 runs

Summary
  ./python -c 'from sqlite3 import *' ran
    1.39 ± 1.81 times faster than ./python -c 'from sqlite3_new import *'

I just realized today that I accidentally deleted an unmerged branch last month, which resulted in closing #129118 . My apologies for the oversight. I've now recreated the pull request - the content remains exactly the same as before

Module	sqlite3 self(us)	sqlite3 cumulative(us)	sqlite3_new self(us)	sqlite3_new cumulative(us)
`_sqlite3`	2658	6641	2455	6462
`sqlite3.dbapi2`	6431	31714	5607	28860
`sqlite3` (main)	8592	40306	4213	35121

The time taken by _sqlite3 is not much different.
The time for sqlite3.dbapi2 decreased significantly from 6431 to 5607
The time for the main sqlite3 module dropped noticeably from 8592 to 4213

Issue: Improve import time of various stdlib modules #137855

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

ok

4b04bdf

Wulian233 requested review from berkerpeksag and erlend-aasland as code owners

March 27, 2025 12:17

bedevere-app Bot added the awaiting review label

Mar 27, 2025

bedevere-app Bot mentioned this pull request

Mar 27, 2025

Improve import time of various stdlib modules #118761

Closed

Wulian233 mentioned this pull request

Mar 27, 2025

gh-118761: Improve import time of sqlite3 #129118

Closed

eendebakpt commented Mar 27, 2025

Copy link

Copy Markdown

Contributor

@Wulian233 In your benchmark the sqlite3_new is slower (takes more time to import). Based on the name I would expect it to be the faster one.

Also time is a builtin module. Is that one really a bottleneck?

erlend-aasland reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brianschubert reviewed

Mar 27, 2025

View reviewed changes

Comment thread

Lib/sqlite3/dbapi2.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

brianschubert commented Mar 27, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

I'm also a bit surprised that time would be a bottleneck. When I run ./python -X importtime -c 'import sqlite3' on my system, I get

import time: self [us] | cumulative | imported package
[...]
import time:        80 |         80 |   time
[...]
import time:      2011 |       8096 | sqlite3

Which, if I'm reading it right, says that importing time can only account for only a small fraction sqlite3's import time?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Wulian233 commented Mar 27, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor Author

@Wulian233 In your benchmark the sqlite3_new is slower (takes more time to import). Based on the name I would expect it to be the faster one.

Also time is a builtin module. Is that one really a bottleneck?

I think I got the name wrong at the time, it was a copy of the previous PR content, and here is the test I just run

D:\Python313>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'"
Benchmark 1: ./python -c 'from sqlite3 import *'
  Time (mean ± σ):     697.1 µs ± 1059.7 µs    [User: 2577.5 µs, System: 3063.5 µs]
  Range (min … max):     0.0 µs … 10960.7 µs    391 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: Ignoring non-zero exit code.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: ./python -c 'from sqlite3_new import *'
  Time (mean ± σ):     635.0 µs ± 755.4 µs    [User: 1821.0 µs, System: 3691.3 µs]
  Range (min … max):     0.0 µs … 3622.5 µs    399 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: Ignoring non-zero exit code.

Summary
  ./python -c 'from sqlite3_new import *' ran
    1.10 ± 2.12 times faster than ./python -c 'from sqlite3 import *'

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

StanFromIreland reviewed

Mar 27, 2025

View reviewed changes

Comment thread

Lib/sqlite3/dbapi2.py Outdated

		# 3. This notice may not be removed or altered from any source distribution.

		import datetime
		import time

StanFromIreland Mar 27, 2025

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the point of the PR if the import is not removed? Lazy imports are therefore not used, then why should any speedup be expected, a slowdown is more likely?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

eli-schwartz Mar 27, 2025

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already pointed out in #131796 (comment) :)

(But the PR author self-reviewed the review comment by declaring the review comment to be resolved, which is a controversial feature for github to even allow.)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

picnixz commented Mar 29, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

To be precise, it's not importing sqlite3 that is faster, it's importing sqlite3 and loading its content in the global namespace (the benchmarks are done for from sqlite3 import * not for a plain import sqlite3 statement).

Also, the benchmarks are quite unstable. The standard deviation is much higher than the mean itself! In practice, I think we don't gain anything else than stability (the stdev becomes smaller with the new implementation but again this could be noise).

Can you check if import sqlite3 has less noise?

And to be precise, we're not gaining a 2x speed-up. On average, we're only gaining a 1.1x speed-up and the standard deviation of this speed-up is $\pm2$, which is again not really indicative =/

And yes, I'm also surprised that we're gaining "that much" when we just make import time local. Considering it's a built-in module (though not sure if it's present at startup), I don't think we need to do this change (we're maybe gaining a small speed-up but maybe not much; and sqlite3 is already is a heavy module to import). So benchmarks using -X importtime here are much more important IMO compared to interpreter's startup as well.

erlend-aasland and serhiy-storchaka reacted with thumbs up emoji

All reactions

👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity added the topic-sqlite3 label

Apr 21, 2025

erlend-aasland commented Jun 5, 2025

Copy link

Copy Markdown

Contributor

@Wulian233, can you address @picnixz's last remark?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland added the pending The issue will be closed if no feedback is provided label

Jun 5, 2025


          Improve the import time of sqlite3 by 1.5x faster.

d3cf658

Wulian233 commented Jun 6, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor Author

I've reworked another version and it's going to be faster now

I ran the command D:\Python314>python -X importtime -c "import sqlite3" and obtained the following output. Below is a summary table comparing sqlite3 and sqlite3_new import times (GPT helped):

Module	sqlite3 self(us)	sqlite3 cumulative(us)	sqlite3_new self(us)	sqlite3_new cumulative(us)
`_sqlite3`	2658	6641	2455	6462
`sqlite3.dbapi2`	6431	31714	5607	28860
`sqlite3` (main)	8592	40306	4213	35121

_sqlite3 is not much different.
sqlite3.dbapi2 decreased from 6431 -> 5607
sqlite3 main dropped very noticeably from 8592 -> 4213

Raw output:

Details

import time: self [us] | cumulative | imported package import time: 296 | 296 | winreg import time: 248 | 248 | _io import time: 70 | 70 | marshal import time: 312 | 312 | nt import time: 1637 | 2266 | _frozen_importlib_external import time: 1103 | 1103 | time import time: 1089 | 2191 | zipimport import time: 75 | 75 | _codecs import time: 793 | 868 | codecs import time: 1926 | 1926 | encodings.aliases import time: 797 | 797 | encodings._win_cp_codecs import time: 3400 | 6989 | encodings import time: 960 | 960 | encodings.utf_8 import time: 75 | 75 | _codecs_cn import time: 126 | 126 | _multibytecodec import time: 1389 | 1590 | encodings.gbk import time: 86 | 86 | _signal import time: 59 | 59 | _abc import time: 323 | 382 | abc import time: 104 | 104 | _stat import time: 527 | 630 | stat import time: 1745 | 1745 | _collections_abc import time: 126 | 126 | genericpath import time: 229 | 229 | _winapi import time: 3109 | 3463 | ntpath import time: 2543 | 8760 | os import time: 584 | 584 | _sitebuiltins import time: 1004 | 1004 | sitecustomize import time: 563 | 563 | usercustomize import time: 2043 | 12953 | site import time: 1104 | 1104 | linecache import time: 787 | 787 | _datetime import time: 1003 | 1790 | datetime import time: 704 | 704 | itertools import time: 1147 | 1147 | keyword import time: 128 | 128 | _operator import time: 1412 | 1540 | operator import time: 2293 | 2293 | reprlib import time: 164 | 164 | _collections import time: 4104 | 9950 | collections import time: 2917 | 12866 | collections.abc import time: 64 | 64 | _types import time: 1134 | 1197 | types import time: 136 | 136 | _functools import time: 2652 | 3984 | functools import time: 2658 | 6641 | _sqlite3 import time: 74 | 74 | _contextvars import time: 1406 | 1479 | _py_warnings import time: 2508 | 3987 | warnings import time: 6431 | 31714 | sqlite3.dbapi2 import time: 8592 | 40306 | sqlite3

D:\Python314>python -X importtime -c "import sqlite3_new"
import time: self [us] | cumulative | imported package
import time: 142 | 142 | winreg
import time: 284 | 284 | _io
import time: 119 | 119 | marshal
import time: 461 | 461 | nt
import time: 2051 | 2914 | _frozen_importlib_external
import time: 849 | 849 | time
import time: 651 | 1500 | zipimport
import time: 131 | 131 | _codecs
import time: 4036 | 4166 | codecs
import time: 1493 | 1493 | encodings.aliases
import time: 1159 | 1159 | encodings._win_cp_codecs
import time: 6495 | 13311 | encodings
import time: 1028 | 1028 | encodings.utf_8
import time: 72 | 72 | _codecs_cn
import time: 124 | 124 | _multibytecodec
import time: 1305 | 1500 | encodings.gbk
import time: 119 | 119 | _signal
import time: 58 | 58 | _abc
import time: 366 | 424 | abc
import time: 100 | 100 | _stat
import time: 2987 | 3086 | stat
import time: 1215 | 1215 | _collections_abc
import time: 135 | 135 | genericpath
import time: 238 | 238 | _winapi
import time: 673 | 1045 | ntpath
import time: 2186 | 7956 | os
import time: 1146 | 1146 | _sitebuiltins
import time: 1021 | 1021 | sitecustomize
import time: 545 | 545 | usercustomize
import time: 1849 | 12515 | site
import time: 1417 | 1417 | linecache
import time: 163 | 163 | _datetime
import time: 809 | 972 | datetime
import time: 570 | 570 | itertools
import time: 1219 | 1219 | keyword
import time: 154 | 154 | _operator
import time: 1384 | 1538 | operator
import time: 1604 | 1604 | reprlib
import time: 151 | 151 | _collections
import time: 3696 | 8776 | collections
import time: 2846 | 11621 | collections.abc
import time: 68 | 68 | _types
import time: 1052 | 1119 | types
import time: 174 | 174 | _functools
import time: 2716 | 4007 | functools
import time: 2455 | 6462 | _sqlite3
import time: 77 | 77 | _contextvars
import time: 2803 | 2879 | _py_warnings
import time: 1321 | 4200 | warnings
import time: 5607 | 28860 | sqlite3.dbapi2
import time: 1298 | 30158 | sqlite3
import time: 751 | 30908 | sqlite3.dbapi2
import time: 4213 | 35121 | sqlite3_new

hyperfine test:

D:\Python314>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'
Benchmark 1: ./python -c 'from sqlite3 import *'
  Time (mean ± σ):     176.8 µs ± 179.3 µs    [User: 1985.8 µs, System: 2886.4 µs]
  Range (min … max):     0.0 µs … 1018.1 µs    406 runs

Benchmark 2: ./python -c 'from sqlite3_new import *'
  Time (mean ± σ):     245.4 µs ± 200.7 µs    [User: 914.9 µs, System: 2002.8 µs]
  Range (min … max):     0.0 µs … 1787.5 µs    401 runs

Summary
  ./python -c 'from sqlite3 import *' ran
    1.39 ± 1.81 times faster than ./python -c 'from sqlite3_new import *'

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Wulian233 added 4 commits

June 6, 2025 10:33


          Merge branch 'main' into sqlite

4373d47


          fix & finish

def1974


          Update dbapi2.py

04eedab

fix

ccf3575

Wulian233 mentioned this pull request

Jun 6, 2025

gh-134036: Improve error messages for invalid raise statements #134077

Merged

eli-schwartz suggested changes

Jun 6, 2025

View reviewed changes

Comment thread

Lib/sqlite3/__main__.py

Comment on lines 66 to +69

		def main(*args):
		from argparse import ArgumentParser
		from textwrap import dedent

eli-schwartz Jun 6, 2025

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not see a valid reason to do code churn and delay imports for the sake of speeding up import sqlite3.__main__ while assuming that sqlite3.__main__.main() is never called.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka reacted with thumbs up emoji

All reactions

👍 1 reaction

terryjreedy Aug 16, 2025

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume the following: execute is used by the REPL in addition to CLI; user code could import to use SQLiteInteractiveConsole. Whether worth it, not my call.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

eli-schwartz Aug 17, 2025

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The REPL is of course part of calling main(), which makes my point quite well. :)

User code certainly could import SQLiteInteractiveConsole, as it lies within their technical capability. I think any user code importing an interactive console from a module named __main__ ought to be prepared to use it without worrying about the startup time of importing argparse and textwrap. A module called __main__ feels rather special to me in that sense -- one ought not casually resort to importing it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka reacted with thumbs up emoji

All reactions

👍 1 reaction

serhiy-storchaka Aug 17, 2025

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @eli-schwartz. This change looks meaningless.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/sqlite3/__main__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

bedevere-app Bot added awaiting core review and removed awaiting review labels

Jun 6, 2025

ofek mentioned this pull request

Aug 16, 2025

Improve import time of various stdlib modules #137855

Closed

hugovk changed the title ~~gh-118761: Improve import time of sqlite3~~ gh-137855: Improve import time of sqlite3

Aug 16, 2025

Wulian233 added 2 commits

August 17, 2025 08:43


          Merge branch 'python:main' into sqlite

4c138d8


          Rename pythongh-137855

f434be3

serhiy-storchaka commented Aug 17, 2025

Copy link

Copy Markdown

Member

I agree with @picnixz and @eli-schwartz. The effect of moving import time looks not more than a noise. And it does not make much sense to optimize "import" time of sqlite3.__main__ -- it does not affect the total time.

erlend-aasland reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Wulian233 closed this

Aug 17, 2025

Wulian233 deleted the sqlite branch

August 17, 2025 06:34

Reviewers

serhiy-storchaka

serhiy-storchaka left review comments

terryjreedy

terryjreedy left review comments

brianschubert

brianschubert left review comments

StanFromIreland

StanFromIreland left review comments

berkerpeksag

Awaiting requested review from berkerpeksag

erlend-aasland

Awaiting requested review from erlend-aasland

erlend-aasland is a code owner

+1 more reviewer

eli-schwartz

eli-schwartz requested changes

Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review pending

The issue will be closed if no feedback is provided

topic-sqlite3

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

10 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

Wulian233 commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eendebakpt commented Mar 27, 2025

Uh oh!

Uh oh!

brianschubert commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Wulian233 commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StanFromIreland Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

picnixz commented Mar 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erlend-aasland commented Jun 5, 2025

Uh oh!

Wulian233 commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eli-schwartz Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

terryjreedy Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

serhiy-storchaka commented Aug 17, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Wulian233 commented Mar 27, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

brianschubert commented Mar 27, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Wulian233 commented Mar 27, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

picnixz commented Mar 29, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Wulian233 commented Jun 6, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.