GeezerVison is showing github.com — Move DataSource validation to OfflineStore · Issue #4186 · feast-dev/feast
Move DataSource validation to OfflineStore #4186
Description
DataSource abstract interface contains methods like validate and get_table_column_names_and_types for validation and schema extraction respectively. This doesn't make too much sense when you consider the separation of concerns between DataSource and OfflineStore. DataSource is supposed to be a static description of source dataset, while OfflineStore is an engine that knows how to read one or more data source types. Having these methods in DataSource classes means data sources should also be able to somehow access the underlying sources.
I propose to move validate method to OfflineStore abstract class as a validate_data_source. This also makes sense for scenarios when a single source can be read by multiple offline stores. For example, FileSource (which can be read by dask, duckdb and probably spark in the future) is right now validated with pyarrow instead of leaving it up to the OfflineStores to choose how to validate the sources).
Reacted by Francisco Javier Arceo
Activity
franciscojavierarceo commented on May 9, 2024
Can you point me to the code you're referring to?
I want to confirm if it's used for Data Quality Monitoring or not.
tokoko commented on May 9, 2024
CollaboratorAuthor
More actions
I think it has nothing to do with dqm. validation here refers to validating whether the data source exists or not. Here is the abstract method I'm referring to. The only place where it's called from is this code segment in repo_operations.py (meaning it's called during feast apply).
franciscojavierarceo commented on May 9, 2024
Yeah, you're right. I just took a quick looks. It looks like it's referenced in feature logging but doesn't appear to be used and is mostly used for the offline store. I agree with your take here. 👍
Metadata
Metadata
Development
No branches or pull requestsIssue actions