A class for collecting, tracking, and logging runtime statistics for requests.
It is designed to record information such as request durations, retries, successes, and failures, enabling
analysis of crawler performance. The collected statistics are persisted to a KeyValueStore, ensuring they
remain available across crawler migrations, abortions, and restarts. This persistence allows for tracking
and evaluation of crawler behavior over its lifecycle.
Hierarchy
- Statistics
Index
Methods
__aenter__
- async __aenter__(): Self
Subscribe to events and start collecting statistics.
Returns Self
__aexit__
- async __aexit__(exc_type, exc_value, exc_traceback): None
Stop collecting statistics.
Parameters
exc_type: type[BaseException] | None
exc_value: BaseException | None
exc_traceback: TracebackType | None
Returns None
__init__
- __init__(*, persistence_enabled, persist_state_kvs_name, persist_state_key, persist_state_kvs_factory, log_message, periodic_message_logger, log_interval, state_model, statistics_log_format, save_error_snapshots): None
Parameters
optionalkeyword-onlypersistence_enabled: bool | Literal[explicit_only] = False
optionalkeyword-onlypersist_state_kvs_name: str | None = None
optionalkeyword-onlypersist_state_key: str | None = None
optionalkeyword-onlypersist_state_kvs_factory: Callable[[], Coroutine[None, None, KeyValueStore]] | None = None
optionalkeyword-onlylog_message: str = 'Statistics'
optionalkeyword-onlyperiodic_message_logger: Logger | None = None
optionalkeyword-onlylog_interval: timedelta = timedelta(minutes=1)
keyword-onlystate_model: type[TStatisticsState]
optionalkeyword-onlystatistics_log_format: Literal[table, inline] = 'table'
optionalkeyword-onlysave_error_snapshots: bool = False
Returns None
calculate
- calculate(): FinalStatistics
Calculate the current statistics.
Returns FinalStatistics
record_request_processing_deferral
- record_request_processing_deferral(request_id_or_key): None
Mark a request as deferred, so this attempt doesn't count as a retry.
Parameters
request_id_or_key: str
Returns None
record_request_processing_failure
- record_request_processing_failure(request_id_or_key): None
Mark a request as failed.
Parameters
request_id_or_key: str
Returns None
record_request_processing_finish
- record_request_processing_finish(request_id_or_key): None
Mark a request as finished.
Parameters
request_id_or_key: str
Returns None
record_request_processing_start
- record_request_processing_start(request_id_or_key): None
Mark a request as started.
Parameters
request_id_or_key: str
Returns None
register_status_code
- register_status_code(code): None
Increment the number of times a status code has been received.
Parameters
code: int
Returns None
replace_state_model
- replace_state_model(state_model): Statistics[TNewStatisticsState]
Create near copy of the
Statisticswith replacedstate_model.Parameters
state_model: type[TNewStatisticsState]
Returns Statistics[TNewStatisticsState]
reset
- async reset(): None
Reset the statistics to their defaults and remove any persistent state.
Returns None
with_default_state
- with_default_state(*, persistence_enabled, persist_state_kvs_name, persist_state_key, persist_state_kvs_factory, log_message, periodic_message_logger, log_interval, statistics_log_format, save_error_snapshots): Statistics[StatisticsState]
Initialize a new instance with default state model
StatisticsState.Parameters
optionalkeyword-onlypersistence_enabled: bool = False
optionalkeyword-onlypersist_state_kvs_name: str | None = None
optionalkeyword-onlypersist_state_key: str | None = None
optionalkeyword-onlypersist_state_kvs_factory: Callable[[], Coroutine[None, None, KeyValueStore]] | None = None
optionalkeyword-onlylog_message: str = 'Statistics'
optionalkeyword-onlyperiodic_message_logger: Logger | None = None
optionalkeyword-onlylog_interval: timedelta = timedelta(minutes=1)
optionalkeyword-onlystatistics_log_format: Literal[table, inline] = 'table'
optionalkeyword-onlysave_error_snapshots: bool = False
Returns Statistics[StatisticsState]
Properties
active
Indicate whether the context is active.