GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: add ibis-based duckdb offline store by tokoko · Pull Request #3822 · feast-dev/feast

Conversation

@tokoko

@tokoko tokoko commented Oct 29, 2023

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
This PR adds two offline store implementations:

  • ibis - a base offline store implementation meant to be used by other offline stores.
  • duckdb - a thin layer on ibis offline store meant to be tested and used by users just like other offline stores.

This is a draft PR to garner feedback from the community regarding the idea to converge offline store implementations to a single abstract base implementation that will then be extended by concrete offline stores. Right now each feast offline store has to more or less reimplement the same data transformation logic (mainly point-in-time join) in their respective technologies. The goal of this PR is to suggest a possible alternative approach where feast would only implement abstract transformations and offload technology translation details to another library, namely ibis.

Ibis is a lightweight python library that offers a universal dataframe API with pluggable query engine backends. As of now it can interface with 15+ engines with varying level of API coverage, including BigQuery, Snowflake, Pyspark, PosgreSQL, Trino and others. I think adopting ibis as a common backend for at least some offline store implementations (contrib stores for starters) would simplify development effort and ensure feature parity.

The implementation in this PR is very rough and far from merge-ready. The main goal is to find out it the community thinks this is a good long-term approach for feast offline stores in general.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

MRocholl reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@tokoko
draft implementation for registry service
a1dc714
@achals
achals requested review from achals and kevjumba October 31, 2023 05:53
@achals achals added the ok-to-test label Oct 31, 2023
@tokoko tokoko mentioned this pull request Dec 11, 2023
Add deltalake backend #3865
Closed
@tokoko
tokoko force-pushed the feast-ibis branch from 91e16f0 to a1dc714 Compare February 23, 2024 01:14
@tokoko
feat: add ibis offline store
88cbed3
Signed-off-by: tokoko <[email protected]>
@tokoko tokoko closed this Mar 1, 2024
@tokoko tokoko mentioned this pull request Mar 2, 2024
feat: Add duckdb offline store #3981
Merged
@tokoko
tokoko deleted the feast-ibis branch April 4, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kevjumba kevjumba Awaiting requested review from kevjumba
@achals achals Awaiting requested review from achals
@sudohainguyen sudohainguyen Awaiting requested review from sudohainguyen sudohainguyen will be requested when the pull request is marked ready for review sudohainguyen is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@tokoko @achals
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.