GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat: Add deduplicate pushdown to clickhouse - improve materialize performance by astronautas · Pull Request #5709 · feast-dev/feast

feat: Add deduplicate pushdown to clickhouse - improve materialize performance #5709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

astronautas wants to merge 7 commits into feast-dev:master from astronautas:feat/add-deduplicate-pushdown-to-clickhouse

Uh oh!

There was an error while loading. Please reload this page.

Closed

feat: Add deduplicate pushdown to clickhouse - improve materialize performance #5709

Show file tree

Hide file tree

Changes from 1 commit

Commits

Show all changes

7 commits

Select commit Hold shift + click to select a range

1752a4f

add pull_all_from_table_or_query for clickhouse, to align with new ma…

astronautas Oct 15, 2025

a6fcf5e

Merge branch 'feast-dev:master' into master

Oct 29, 2025

f7b7d37

add possibility to pushdown clickhouse deduplication to offline store

astronautas Nov 3, 2025

7579962

add possibility to pushdown clickhouse deduplication to offline store

astronautas Nov 3, 2025

762de72

Merge branch 'feast-dev:master' into feat/add-deduplicate-pushdown-to…

Nov 3, 2025

50d84d1

use info, since it's an intended path too

astronautas Nov 3, 2025

cfaf9a6

make timestamps mandatory, filtering won't work otherwise

astronautas Nov 3, 2025

File filter

Filter by extension .py (2) All 1 file type selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

Previous commit

Next commit

add possibility to pushdown clickhouse deduplication to offline store

Signed-off-by: lukas.valatka <[email protected]>

Loading branch information

Uh oh!

There was an error while loading. Please reload this page.

astronautas committed Nov 3, 2025

commit f7b7d3722cfbf1db20b01d748664e9ba69dacd31

There are no files selected for viewing

26 changes: 26 additions & 0 deletions 26 sdk/python/feast/infra/offline_stores/contrib/clickhouse_offline_store/clickhouse.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
		@@ -1,4 +1,5 @@
		import contextlib
		import logging
		import re
		from dataclasses import asdict
		from datetime import datetime
Expand Down Expand Up		@@ -32,6 +33,8 @@
		from feast.infra.utils.clickhouse.connection_utils import get_client
		from feast.saved_dataset import SavedDatasetStorage

		logger = logging.getLogger(__name__)


		class ClickhouseOfflineStoreConfig(ClickhouseConfig):
		type: Literal["clickhouse"] = "clickhouse"
Expand Down Expand Up		@@ -205,6 +208,29 @@ def pull_all_from_table_or_query(
		assert isinstance(config.offline_store, ClickhouseOfflineStoreConfig)
		assert isinstance(data_source, ClickhouseSource)

		# https://github.com/feast-dev/feast/issues/5707
		# Not an ideal solution, but least invasion into existing codebase
		if config.offline_store.deduplicate_pushdown:
		logger.warning(
		"""
		deduplicate_pushdown optimization is set to True in ClickhouseOfflineStoreConfig.
		Calling pull_latest_from_table_or_query instead of pull_all_from_table_or_query.
		This results in multiple times more efficient materialization jobs for large datasets.
		However, it comes with a caveat - can't compute historical features (just latest values available) with Feast compute engines
		Use with caution and ensure your use case aligns with this behavior.
		"""
		)
		return ClickhouseOfflineStore.pull_latest_from_table_or_query(
		config=config,
		data_source=data_source,
		join_key_columns=join_key_columns,
		feature_name_columns=feature_name_columns,
		timestamp_field=timestamp_field,
		created_timestamp_column=created_timestamp_column,
		start_date=start_date,
		end_date=end_date,
		)

		from_expression = data_source.get_table_query_string()

		timestamp_fields = [timestamp_field]
Expand Down

5 changes: 5 additions & 0 deletions 5 sdk/python/feast/infra/utils/clickhouse/clickhouse_config.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -11,4 +11,9 @@ class ClickhouseConfig(FeastConfigBaseModel):
		password: StrictStr
		use_temporary_tables_for_entity_df: bool = True

		# https://github.com/feast-dev/feast/issues/5707
		# We observed that for large materialization jobs, it's multiple times more efficient
		# pushdown deduplication to ClickHouse side rather than doing it in Feast (no matter compute engine used)
		deduplicate_pushdown: bool = False

		model_config = ConfigDict(frozen=True)

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.