File system implementation of the key-value store client.
This client persists data to the file system, making it suitable for scenarios where data needs to survive process restarts. Keys are mapped to file paths in a directory structure following the pattern:
{STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}
Binary data is stored as-is, while JSON and text data are stored in human-readable format. The implementation automatically handles serialization based on the content type and maintains metadata about each record.
This implementation is ideal for long-running crawlers where persistence is important and for development environments where you want to easily inspect the stored data between runs.
Hierarchy
- KeyValueStoreClient
- FileSystemKeyValueStoreClient
Index
Methods
__init__
- __init__(*, metadata, path_to_kvs, lock): None
Initialize a new instance.
Preferably use the
FileSystemKeyValueStoreClient.openclass method to create a new instance.Parameters
keyword-onlymetadata: KeyValueStoreMetadata
keyword-onlypath_to_kvs: Path
keyword-onlylock: asyncio.Lock
Returns None
delete_value
- async delete_value(*, key): None
Delete a value from the key-value store by its key.
The backend method for the
KeyValueStore.delete_valuecall.Parameters
keyword-onlykey: str
Returns None
drop
- async drop(): None
Drop the whole key-value store and remove all its values.
The backend method for the
KeyValueStore.dropcall.Returns None
get_metadata
- async get_metadata(): KeyValueStoreMetadata
Get the metadata of the key-value store.
Returns KeyValueStoreMetadata
get_public_url
- async get_public_url(*, key): str
Return a file:// URL for the given key.
Parameters
keyword-onlykey: str
The key to get the public URL for.
Returns str
A file:// URL pointing to the file on the local filesystem.
get_value
- async get_value(*, key): KeyValueStoreRecord | None
Retrieve the given record from the key-value store.
The backend method for the
KeyValueStore.get_valuecall.Parameters
keyword-onlykey: str
Returns KeyValueStoreRecord | None
iterate_entries
- async iterate_entries(*, exclusive_start_key, limit): AsyncIterator[KeyValueStoreRecord]
Iterate over all the existing records in the key-value store, including their values.
The backend method for the
KeyValueStore.iterate_entriesandKeyValueStore.iterate_valuescalls.The default implementation lists the keys with
iterate_keysand reads each value separately withget_valueas the iteration advances, so only a single value is held in memory at a time. Whether a record deleted while the iteration is in progress is yielded depends on the implementation. Backends that can read the keys together with their values more efficiently should override this method; such an implementation must still keep the memory usage bounded, e.g. by streaming or by batching on the known record sizes.Parameters
optionalkeyword-onlyexclusive_start_key: str | None = None
optionalkeyword-onlylimit: int | None = None
Returns AsyncIterator[KeyValueStoreRecord]
iterate_keys
- async iterate_keys(*, exclusive_start_key, limit): AsyncIterator[KeyValueStoreRecordMetadata]
Iterate over all the existing keys in the key-value store.
The backend method for the
KeyValueStore.iterate_keyscall.Parameters
optionalkeyword-onlyexclusive_start_key: str | None = None
optionalkeyword-onlylimit: int | None = None
Returns AsyncIterator[KeyValueStoreRecordMetadata]
open
- async open(*, id, name, alias, configuration): Self
Open or create a file system key-value store client.
This method attempts to open an existing key-value store from the file system. If a KVS with the specified ID or name exists, it loads the metadata from the stored files. If no existing store is found, a new one is created.
Parameters
keyword-onlyid: str | None
The ID of the key-value store to open. If provided, searches for existing store by ID.
keyword-onlyname: str | None
The name of the key-value store for named (global scope) storages.
keyword-onlyalias: str | None
The alias of the key-value store for unnamed (run scope) storages.
keyword-onlyconfiguration: Configuration
The configuration object containing storage directory settings.
Returns Self
An instance for the opened or created storage client.
purge
- async purge(): None
Purge all items from the key-value store.
The backend method for the
KeyValueStore.purgecall.Returns None
record_exists
- async record_exists(*, key): bool
Check if a record with the given key exists in the key-value store.
Parameters
keyword-onlykey: str
The key to check for existence.
Returns bool
True if a record with the given key exists, False otherwise.
set_value
- async set_value(*, key, value, content_type): None
Set a value in the key-value store by its key.
The backend method for the
KeyValueStore.set_valuecall.Parameters
keyword-onlykey: str
keyword-onlyvalue: Any
optionalkeyword-onlycontent_type: str | None = None
Returns None
Properties
path_to_kvs
The full path to the key-value store directory.
path_to_metadata
The full path to the key-value store metadata file.