Base class for storage clients.
The StorageClient serves as an abstract base class that defines the interface for accessing Crawlee's
storage types: datasets, key-value stores, and request queues. It provides methods to open clients for
each of these storage types and handles common functionality.
Storage clients implementations can be provided for various backends (file system, memory, databases, various cloud providers, etc.) to support different use cases from development to production environments.
Each storage client implementation is responsible for ensuring proper initialization, data persistence (where applicable), and consistent access patterns across all storage types it supports.
Hierarchy
Index
Methods
create_dataset_client
- async create_dataset_client(*, id, name, alias, configuration): DatasetClient
Create a dataset client.
Parameters
optionalkeyword-onlyid: str | None = None
optionalkeyword-onlyname: str | None = None
optionalkeyword-onlyalias: str | None = None
optionalkeyword-onlyconfiguration: Configuration | None = None
Returns DatasetClient
create_kvs_client
- async create_kvs_client(*, id, name, alias, configuration): KeyValueStoreClient
Create a key-value store client.
Parameters
optionalkeyword-onlyid: str | None = None
optionalkeyword-onlyname: str | None = None
optionalkeyword-onlyalias: str | None = None
optionalkeyword-onlyconfiguration: Configuration | None = None
Returns KeyValueStoreClient
create_rq_client
- async create_rq_client(*, id, name, alias, configuration): RequestQueueClient
Create a request queue client.
Parameters
optionalkeyword-onlyid: str | None = None
optionalkeyword-onlyname: str | None = None
optionalkeyword-onlyalias: str | None = None
optionalkeyword-onlyconfiguration: Configuration | None = None
Returns RequestQueueClient
get_rate_limit_errors
- get_rate_limit_errors(): dict[int, int]
Return statistics about rate limit errors encountered by the HTTP client in storage client.
Returns dict[int, int]
get_storage_client_cache_key
- get_storage_client_cache_key(configuration): Hashable
Return a cache key that can differentiate between different storages of this and other clients.
Can be based on configuration or on the client itself. By default, returns a module and name of the client class.
Parameters
configuration: Configuration
Returns Hashable