@@ -31,6 +31,10 @@ When the server processes a `get_online_features()` call, it groups the requeste |
31 | 31 |
|
32 | 32 | **Guideline:** For features that share the same entity key and are frequently requested together, consolidate them into a **single feature view**. This reduces the number of store round-trips per request. Split feature views only when features have different entities, different materialization schedules, or different data source update frequencies. |
33 | 33 |
|
| 34 | +{% hint style="info" %} |
| 35 | +**Redis exception:** The Redis online store overrides `get_online_features()` to batch all `HMGET` commands across every feature view into a **single pipeline execution**. Because all feature views for the same entity share one Redis hash key, the number of Redis round trips is always **1**, regardless of how many feature views the request touches. This means the "fewer feature views" guideline is less critical for Redis than for other stores — but consolidating feature views still reduces serialization and protobuf overhead at the application layer. |
| 36 | +{% endhint %} |
| 37 | + |
34 | 38 | ### Feature services are free |
35 | 39 |
|
36 | 40 | A [Feature Service](../getting-started/concepts/feature-retrieval.md) is a named collection of feature references — it's a convenience grouping, not a separate storage or execution unit. Using a feature service adds only a registry lookup (cached) compared to listing features individually. There is no performance penalty for using feature services, and they are the recommended way to define stable, versioned feature sets for production models. |
@@ -229,7 +233,7 @@ The online store is the single largest factor in `get_online_features()` latency |
229 | 233 |
|
230 | 234 | | Store | Typical p50 latency | Async read | Best for | Key trade-off | |
231 | 235 | | ----- | ------------------- | ---------- | -------- | ------------- | |
232 | | -| **Redis / Dragonfly** | < 1 ms | No (threadpool) | Ultra-low latency, high throughput | Requires in-memory capacity for your dataset | |
| 236 | +| **Redis / Dragonfly** | < 1 ms | No (threadpool) | Ultra-low latency, high throughput; all FV reads batched into 1 pipeline | Requires in-memory capacity for your dataset | |
233 | 237 | | **DynamoDB** | 2–5 ms | Yes | Serverless, auto-scaling on AWS | Pay-per-request cost; batch API limits (100 items) | |
234 | 238 | | **PostgreSQL** | 3–10 ms | No (threadpool) | Teams with existing Postgres infra | Connection pooling needed at scale | |
235 | 239 | | **MongoDB** | 2–5 ms | Yes | Flexible schema, async-native | Requires index tuning for large datasets | |
@@ -259,7 +263,7 @@ The feature server can read from the online store using either an **async** or * |
259 | 263 | | **DynamoDB** | Yes | Yes | Uses `aiobotocore` for non-blocking I/O | |
260 | 264 | | **MongoDB** | Yes | Yes | Uses `motor` (async MongoDB driver) | |
261 | 265 | | **PostgreSQL** | Implemented | No | Has `online_read_async` but does not yet advertise via `async_supported`; uses sync/threadpool path | |
262 | | -| **Redis** | Implemented | No | Has `online_read_async` but does not yet advertise via `async_supported`; uses sync/threadpool path | |
| 266 | +| **Redis** | Implemented | **Yes** | `online_read_async` and `online_write_batch_async` both implemented; uses sync/threadpool path for `get_online_features` (overridden with batched single pipeline) | |
263 | 267 | | All others | No | No | Fall back to sync with `run_in_threadpool()` | |
264 | 268 |
|
265 | 269 | **When async matters most:** |
@@ -329,12 +333,39 @@ online_store: |
329 | 333 | connection_string: "redis-cluster.internal:6379,ssl=true" |
330 | 334 | redis_type: redis_cluster |
331 | 335 | key_ttl_seconds: 604800 |
| 336 | + skip_dedup: false # set true for initial bulk loads to halve write round trips |
332 | 337 | ``` |
333 | 338 |
|
334 | 339 | - Use `redis_cluster` for horizontal partitioning across shards. |
335 | | -- Set `key_ttl_seconds` to auto-expire stale feature data, keeping memory usage bounded. |
| 340 | +- Set `key_ttl_seconds` to auto-expire stale feature data, keeping memory usage bounded. This is a **key-level** TTL — it expires the entire entity hash (all feature views for that entity) together. |
336 | 341 | - Ensure the Redis instance is in the **same availability zone** as the feature server pods to minimize network round-trips. |
337 | 342 |
|
| 343 | +#### Batched multi-feature-view reads |
| 344 | + |
| 345 | +The Redis online store overrides `get_online_features()` to issue all `HMGET` commands — across every feature view in the request — in a **single pipeline execution**. This reduces Redis round trips from `N` (one per feature view) to `1` regardless of request size. |
| 346 | + |
| 347 | +| Feature views | Round trips (other stores) | Round trips (Redis) | |
| 348 | +| :---: | :---: | :---: | |
| 349 | +| 1 | 1 | 1 | |
| 350 | +| 5 | 5 | **1** | |
| 351 | +| 10 | 10 | **1** | |
| 352 | +| 20 | 20 | **1** | |
| 353 | + |
| 354 | +This means the latency cost of adding feature views to a Redis-backed request is primarily **serialization and protobuf overhead** at the application layer, not Redis network latency. |
| 355 | + |
| 356 | +#### Write throughput: `skip_dedup` |
| 357 | + |
| 358 | +For initial bulk loads or append-only materialization pipelines: |
| 359 | + |
| 360 | +```yaml |
| 361 | +online_store: |
| 362 | + type: redis |
| 363 | + connection_string: "localhost:6379" |
| 364 | + skip_dedup: true |
| 365 | +``` |
| 366 | + |
| 367 | +With `skip_dedup: true`, `online_write_batch()` skips the existing-timestamp read pipeline and writes all values directly in a single pass, halving write round trips. Use only when you can guarantee write ordering — under concurrent writers, an older record can overwrite a newer one. |
| 368 | + |
338 | 369 | ### Cassandra / ScyllaDB tuning |
339 | 370 |
|
340 | 371 | Cassandra and ScyllaDB share the same Feast connector. The key read-path knobs are concurrency and data-center-aware routing: |
@@ -421,7 +452,7 @@ Different online stores have different optimal batch sizes for `get_online_featu |
421 | 452 | | Store | Default batch size | Max batch size | Recommendation | |
422 | 453 | | ----- | ------------------ | -------------- | -------------- | |
423 | 454 | | DynamoDB | 100 | 100 (API limit) | Keep at 100; tune `max_read_workers` for parallelism | |
424 | | -| Redis | N/A (pipelined) | N/A | Redis pipelines all keys in one round-trip; no batch tuning needed | |
| 455 | +| Redis | N/A (pipelined) | N/A | All entity keys **and** all feature views are batched into a single pipeline; no batch tuning needed | |
425 | 456 | | PostgreSQL | N/A (single query) | N/A | Single `SELECT ... WHERE key IN (...)` query; tune connection pool instead | |
426 | 457 |
|
427 | 458 | Profile your workload by measuring `feast_feature_server_online_store_read_duration_seconds` (see [Metrics setup](#metrics-setup-prometheus--opentelemetry)) across different entity counts to find the sweet spot. |
@@ -980,12 +1011,79 @@ registry: |
980 | 1011 |
|
981 | 1012 | --- |
982 | 1013 |
|
| 1014 | +## Materialization write performance |
| 1015 | + |
| 1016 | +Materialization (`feast materialize` / `feast materialize-incremental`) reads features from the offline store and writes them to the online store. For large feature views, two common bottlenecks arise: **memory exhaustion** during proto conversion and **write throughput** to the online store. |
| 1017 | + |
| 1018 | +### Memory: `online_write_batch_size` |
| 1019 | + |
| 1020 | +By default, Feast converts the entire Arrow table returned by the offline store into Python protobuf objects in a single pass before writing to the online store. For datasets with millions of rows this can consume tens of gigabytes of memory on the materialization worker. |
| 1021 | + |
| 1022 | +Set `online_write_batch_size` in `feature_store.yaml` to break the write into manageable chunks: |
| 1023 | + |
| 1024 | +```yaml |
| 1025 | +materialization: |
| 1026 | + online_write_batch_size: 10000 # rows per write batch |
| 1027 | +``` |
| 1028 | + |
| 1029 | +Each chunk is independently converted and written, keeping peak memory proportional to the batch size rather than the full dataset. This is supported by the **local, Spark, and Ray** compute engines. |
| 1030 | + |
| 1031 | +| Dataset size | Without batching | With `online_write_batch_size: 10000` | |
| 1032 | +| --- | --- | --- | |
| 1033 | +| 1 M rows (100 bytes/row) | ~100 MB peak | ~1 MB peak | |
| 1034 | +| 10 M rows | ~1 GB peak | ~1 MB peak | |
| 1035 | +| 100 M rows | OOM / swap | ~1 MB peak | |
| 1036 | + |
| 1037 | +**Choosing a value:** |
| 1038 | + |
| 1039 | +- **Larger batches** (50 000+): fewer write calls to the online store, lower overhead per row — good when worker memory allows. |
| 1040 | +- **Smaller batches** (1 000–5 000): lower peak memory — necessary for memory-constrained workers or very wide feature views (many features per row). |
| 1041 | +- For **Redis**: pipeline overhead per batch is negligible; a batch size of 10 000–50 000 is a good starting point. |
| 1042 | +- For **DynamoDB**: each batch maps to one or more `BatchWriteItem` calls (max 25 items per call); a larger `online_write_batch_size` amortizes the per-call overhead but doesn't change the 25-item DynamoDB limit. |
| 1043 | + |
| 1044 | +See the [feature-store-yaml reference](../reference/feature-repository/feature-store-yaml.md#online_write_batch_size) for the complete option documentation. |
| 1045 | + |
| 1046 | +### Throughput: parallel feature view materialization |
| 1047 | + |
| 1048 | +Each feature view in a `feast materialize` call is materialized sequentially by the local engine. To materialize multiple feature views in parallel, use a job orchestrator (Airflow, Kubernetes Jobs) and materialize one feature view per job: |
| 1049 | + |
| 1050 | +```bash |
| 1051 | +# Airflow / cron: one task per feature view |
| 1052 | +feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S") \ |
| 1053 | + --views driver_stats |
| 1054 | +``` |
| 1055 | + |
| 1056 | +Alternatively, use the **Spark** or **Ray** compute engines which distribute the work across a cluster. |
| 1057 | + |
| 1058 | +### Redis: combine with `skip_dedup` for bulk reloads |
| 1059 | + |
| 1060 | +When performing a full historical reload into Redis (not an incremental update), combine `online_write_batch_size` with `skip_dedup` for maximum throughput: |
| 1061 | + |
| 1062 | +```yaml |
| 1063 | +materialization: |
| 1064 | + online_write_batch_size: 50000 # large chunks — memory is bounded |
| 1065 | +
|
| 1066 | +online_store: |
| 1067 | + type: redis |
| 1068 | + connection_string: "redis-cluster.internal:6379" |
| 1069 | + skip_dedup: true # skip per-row timestamp check — halves write round trips |
| 1070 | +``` |
| 1071 | + |
| 1072 | +`skip_dedup: true` eliminates the timestamp-read pipeline before each write (see [Redis tuning](#redis-tuning)), while `online_write_batch_size` prevents the write worker from converting the entire dataset into memory at once. |
| 1073 | + |
| 1074 | +{% hint style="warning" %} |
| 1075 | +Reset `skip_dedup` to `false` (or remove it) after the bulk reload. Under normal incremental materialization, deduplication prevents older feature values from overwriting newer ones. |
| 1076 | +{% endhint %} |
| 1077 | + |
| 1078 | +--- |
| 1079 | + |
983 | 1080 | ## Further reading |
984 | 1081 |
|
985 | 1082 | - [Scaling Feast](./scaling-feast.md) — Horizontal scaling, HPA, KEDA, and HA in detail |
986 | 1083 | - [Python Feature Server](../reference/feature-servers/python-feature-server.md) — CLI flags, metrics reference, and API endpoints |
987 | 1084 | - [OpenTelemetry Integration](../getting-started/components/open-telemetry.md) — Full OTEL setup with Prometheus Operator |
988 | 1085 | - [DynamoDB Online Store](../reference/online-stores/dynamodb.md) — Store-specific configuration and performance tuning |
989 | 1086 | - [PostgreSQL Online Store](../reference/online-stores/postgres.md) — Connection pooling and SSL configuration |
990 | | -- [Redis Online Store](../reference/online-stores/redis.md) — Cluster mode, Sentinel, and TTL configuration |
| 1087 | +- [Redis Online Store](../reference/online-stores/redis.md) — Cluster mode, Sentinel, TTL configuration, and batched reads |
991 | 1088 | - [On Demand Feature Views](../reference/beta-on-demand-feature-view.md) — Transformation modes and write-time transforms |
| 1089 | +- [feature_store.yaml reference](../reference/feature-repository/feature-store-yaml.md) — Full configuration reference including `materialization` options |