Description

jtcave

opened on Jul 23, 2023

Issue body actions

Bug report

Calling the clear method on a shelve.Shelf object takes a very long time on databases that have thousands of entries.

It can be seen in this script, which creates a database with 10,000 entries and immediately clears it.

import os
import shelve
import tempfile

with tempfile.TemporaryDirectory() as tempdir:
    filename = os.path.join(tempdir, "test-shelf")
    with shelve.open(filename) as db:
        items = {str(x):x for x in range(10000)}
        db.update(items)
        db.clear()
print("ok")

On my M2 Mac Mini, this script takes about 2.1 seconds.

james@iris ~ % time python shelve-clear-test.py
ok
python shelve-clear-test.py  1.18s user 0.92s system 99% cpu 2.100 total

On a Debian VPS:

james@asteroid:~$ time python3 shelve-clear-test.py 
ok

real	0m43.665s
user	0m34.330s
sys	0m9.107s

Your environment

• CPython versions tested on
  • 3.11.2
  • 3.11.4
  • cpython/main branch
• Operating system and architecture
  • macOS/arm64 13.4.1
  • Debian/x86_64 12.1

Linked PRs

• gh-107089: Improve Shelf.clear method performance #107090

Activity

1. jtcave

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jul 23, 2023
2. jtcave

   added a commit that references this issue on Jul 23, 2023

   pythongh-107089: Improve Shelf.clear method performance

   6cbb066
3. pochmann commented on Jul 23, 2023

   

   pochmann

   on Jul 23, 2023

   Contributor

   More actions

   How much of the 2.1 seconds is the clear method?
4. jtcave commented on Jul 23, 2023

   

   jtcave

   on Jul 23, 2023

   ContributorAuthor

   More actions

   How much of the 2.1 seconds is the clear method?

   Virtually all of it. The example script running on my branch takes milliseconds.

   james@iris cpython-build % time ./python.exe ~/shelve-clear-test.py
   ok
   ./python.exe ~/shelve-clear-test.py  0.05s user 0.02s system 97% cpu 0.069 total
   
5. jtcave

   added 9 commits that reference this issue on Jul 23, 2023

   pythongh-107089: Improve Shelf.clear method performance

   76bea89

   pythongh-107089: Improve backwards compatibility

   4987883

   pythongh-107089: call dbm.clear method

   726b3dc

   pythongh-107089: Improve Shelf.clear method performance

   139dc7c

   pythongh-107089: call dbm.clear method

   81e5c15

   pythongh-107089: Improve Shelf.clear method performance

   05ea943

   pythongh-107089: call dbm.clear method

   345b714

   pythongh-107089: Improve Shelf.clear method performance

   2b5cc11

   pythongh-107089: call dbm.clear method

   0e3f387
6. corona10

   added

   performancePerformance or resource usagePerformance or resource usage

   and removed

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jul 29, 2023
7. corona10

   closed this as completedon Jul 29, 2023
8. corona10

   added a commit that references this issue on Jul 29, 2023

   gh-107089: Improve Shelf.clear method performance (gh-107090)

   810d5d8
9. erlend-aasland

   added this to dbm and shelve issueson Jan 25, 2024
10. erlend-aasland

    moved this to Done in dbm and shelve issueson Jan 25, 2024

Issue actions