GeezerVison is showing github.com — Slow materialization · Issue #5707 · feast-dev/feast
Slow materialization #5707DescriptionExpected Behavior
Feature materialization used to call "get latest feature values" function and return only the latest values - this was much faster
Current Behavior
Materializatio now pulls all features now and deduplicates in pandas - this makes materialization much slower (almost unusable for some cases)
Steps to reproduce
Try materialize
Specifications
- Version: 0.56
- Platform:
- Subsystem:
offline_store: Clickhouse, online_store: Redis
Possible Solution
Give ability to pull only latest values by introducing flag/env variable Activitymiroslawas commented on Oct 31, 2025 ContributorAuthor More actions miroslawas commented on Oct 31, 2025 ContributorAuthor More actions astronautas commented on Oct 31, 2025 We could contribute this param ourselves, no issue at all, just wanted to see what maintainers think. franciscojavierarceo commented on Oct 31, 2025 was this always the case or recently changed? It'd be helpful to know what exposed this. If some users like this behavior than probably the exercise is for us to make this configurable. As a general point, I do agree with you that having pandas execute deduplication is wildly expensive. astronautas commented on Oct 31, 2025
was this always the case or recently changed? It'd be helpful to know what exposed this. If some users like this behavior than probably the exercise is for us to make this configurable. As a general point, I do agree with you that having pandas execute deduplication is wildly expensive.
Seems from 0.50.0, the version which broke Clickhouse offline store , since the core had started calling get_all_... instead of get_latest_... from offline store (which eventually requires deduplication taken care by online store e.g. Redis does this in pure Python / Pandas). haoxu0 commented on Nov 1, 2025 It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only astronautas commented on Nov 3, 2025
It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only
@HaoXuAI I don't think it's a bug. It's a feature. Functionally, it does the job - online store receives only the latest values. The issue is not about this.
Non-functionally, it's much slower. That is the issue. Why? It dedups just before pushing to the online store. Pure Python is used for that. For large periods between start_ts and end_ts, that pulls in a significant amount of data from offline store to be deduped in later downstream. Before it would dedup using SQL via get_latest, so it was multiple times faster and used many times less memory. Basically, it did predicate pushdown, in some sense.
See here. Pretty sure before 0.50.0 it used to call pull_latest..., which instead delegated deduplication (via window functions) to offline store, which would pull-in significantly lower amount of data into materialization.
Does this make sense? haoxu0 commented on Nov 3, 2025
It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only
@HaoXuAI I don't think it's a bug. It's a feature. Functionally, it does the job - online store receives only the latest values. The issue is not about this.
Non-functionally, it's much slower. That is the issue. Why? It dedups just before pushing to the online store. Pure Python is used for that. For large periods between start_ts and end_ts, that pulls in a significant amount of data from offline store to be deduped in later downstream. Before it would dedup using SQL via get_latest, so it was multiple times faster and used many times less memory. Basically, it did predicate pushdown, in some sense.
Does this make sense?
If using pandas that's right. For engines like polars or spark lazy execution it should run the transformation in an optimal way.
I think we could arrange the dedup at early order if not aggregation or transformation provided for compute without optimization such as pandas astronautas commented on Nov 3, 2025
It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only
@HaoXuAI I don't think it's a bug. It's a feature. Functionally, it does the job - online store receives only the latest values. The issue is not about this.
Non-functionally, it's much slower. That is the issue. Why? It dedups just before pushing to the online store. Pure Python is used for that. For large periods between start_ts and end_ts, that pulls in a significant amount of data from offline store to be deduped in later downstream. Before it would dedup using SQL via get_latest, so it was multiple times faster and used many times less memory. Basically, it did predicate pushdown, in some sense.
Does this make sense?
If using pandas that's right. For engines like polars or spark lazy execution it should run the transformation in an optimal way. I think we could arrange the dedup at early order if not aggregation or transformation provided for compute without optimization such as pandas
Any way to switch to Polars? Could you give a YAML reference? I would give it a shot at least. Anything similar - local compute engine without an external cluster - would be also fine to test out :).
Local compute engine - 404. astronautas commented on Nov 3, 2025 astronautas commented on Nov 3, 2025 All right, I just tried this in feature_store.yaml:
...
batch_engine:
type: local
backend: polars
Unfortunately, the total materialization duration got reduced insignificantly :/ I can observe increased CPU usage (good, it parallelizes something), but sadly e2e it's not as fast as it used before when it delegated deduplication to offline store (Clickhouse, in our case). So no, it does not run the materialization in optimal way (optimal meaning compared to < 0.50.0).
Deduplication delegated to offlne store in our case works fast because we set partitioning on the timestamp column. Likely the case for every other analytical store like Snowflake. This significantly increases the performance of materialization.
I propose we introduce some sort of a flag - pushdown_deduplication_to_offline_store - for folks who know what they're doing to be pushed down to offline store to deduplicate early. Maybe part of RepoConfig, as an experimental flag? Or offline_config: Any = Field(None, alias="offline_store")?
We're more than OK to raise a PR, if that would make sense :). astronautas commented on Nov 11, 2025
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|