GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Move DataSource validation to OfflineStore · Issue #4186 · feast-dev/feast

Move DataSource validation to OfflineStore #4186

New issue
Copy link
New issue
Copy link
Closed
#4197
Closed
Move DataSource validation to OfflineStore#4186
#4197
Copy link

Description

@tokoko
tokoko
opened on May 9, 2024
Issue body actions

DataSource abstract interface contains methods like validate and get_table_column_names_and_types for validation and schema extraction respectively. This doesn't make too much sense when you consider the separation of concerns between DataSource and OfflineStore. DataSource is supposed to be a static description of source dataset, while OfflineStore is an engine that knows how to read one or more data source types. Having these methods in DataSource classes means data sources should also be able to somehow access the underlying sources.

I propose to move validate method to OfflineStore abstract class as a validate_data_source. This also makes sense for scenarios when a single source can be read by multiple offline stores. For example, FileSource (which can be read by dask, duckdb and probably spark in the future) is right now validated with pyarrow instead of leaving it up to the OfflineStores to choose how to validate the sources).

Reacted by Francisco Javier Arceo

Activity

  1. franciscojavierarceo commented on May 9, 2024

    @franciscojavierarceo
    franciscojavierarceo
    on May 9, 2024
    Member
    More actions

    Can you point me to the code you're referring to?

    I want to confirm if it's used for Data Quality Monitoring or not.

  2. tokoko commented on May 9, 2024

    @tokoko
    tokoko
    on May 9, 2024
    CollaboratorAuthor
    More actions

    I think it has nothing to do with dqm. validation here refers to validating whether the data source exists or not. Here is the abstract method I'm referring to. The only place where it's called from is this code segment in repo_operations.py (meaning it's called during feast apply).

  3. franciscojavierarceo commented on May 9, 2024

    @franciscojavierarceo
    franciscojavierarceo
    on May 9, 2024
    Member
    More actions

    Yeah, you're right. I just took a quick looks. It looks like it's referenced in feature logging but doesn't appear to be used and is mostly used for the offline store. I agree with your take here. 👍

  4. jeremyary
    closed this as completedin #4197on May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.