GeezerVison is showing github.com — DeltaLake Table Source offline store implementation · Issue #1533 · feast-dev/feast
DeltaLake Table Source offline store implementation #1533DescriptionIs your feature request related to a problem? Please describe.
Scalable data stores—Data warehouse and modern Data Lakes—are common today for offline store implementations or as a source of data ingestion. These stores are optimized and support ACID functionality where clean data can reside, and new data constantly can be updated and merged, accommodating data changes and observing schema evolutions. This ability to ingest data from or provide an offline store as a feast.data_source.FileSource from these stores extends
Feast's ecosystem to modern data lakes such as Delta Lake, Apache Hudi, or Apache Iceberg.
A similar feature request for HudiTableSource has been filed by @blvp
Describe the solution you'd like
Extend feast.data_source.FileSource(...) to take table names and locations to read from, for both local or remote sources.
Describe alternatives you've considered
I would have to save my Delta Lake tables as a single parquet file and use that as FileSource, which may defeat the purpose of being able to ingest point-in-time data from these modern data lake sources. Activitywoop commented on Jul 5, 2021 Just swinging back here @dmatrix. We'd love to support the sources you've laid out above. I think the biggest challenge from our side is just how many storage implementations we should support out of the box, and how many should be community contributed. We're trying to strike a balance here. Efforts like increased pluggability will be our short term solution, but I'd love to figure out if there are low hanging fruit that we haven't identified. dmatrix commented on Jul 6, 2021 ContributorAuthor More actions @woop Yes, it's a matter of striking a balance and compromise: support a data source out of the box that's has wider adoption, reliable vendor support, and large user or contributing community. Yet we cannot overlook the growing adoption of modern data lakes, based on open file formats, as enterprises' central repository for all their data. So it may make sense to support one of these data sources.
As part of an easily extensible Feast ecosystem, the idea of making data sources as pluggable architecture is viable and attractive short term solution. Perhaps, we can solicit community help by labeling and tagging this issue as "Community Help Needed" or "Need Community Help." Reacted by Francisco Javier Arceo Data-drone commented on Sep 30, 2021 Happy to take a crack at this. I have had a little dive into the code source.
I can see that there is a redshift and redshift_source file and same with bigquery. Would I just need to reimplement that? creativedutchmen commented on Jun 4, 2022 Looking at the source code for the Spark offline store, support for delta should be as easy as listing it in the SparkSourceFormat. Reading the table is done (as it should) with df = spark_session.read.format(self.file_format).load(self.path). Reading a delta table is a matter of having .format("delta") and Spark will take care of the rest. franciscojavierarceo commented on Feb 26, 2026 @HaoXuAI i think this was added, right?
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|