GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Add gRPC Registry Server by tokoko · Pull Request #3924 · feast-dev/feast

Conversation

@tokoko

@tokoko tokoko commented Feb 1, 2024

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
Implements a read-only python grpc server that wraps calls to the registry configured in feature store. Adds a cli command to start registry server from feast cli. Primary motivation behind this feature is to enable non-python feast applications (java feature server) to easily communicate with feast registry.

Which issue(s) this PR fixes:
Fixes #3901

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sudohainguyen reacted with eyes emoji
All reactions
  • 👀 1 reaction
tokoko added 3 commits February 1, 2024 21:22
@tokoko
feat: Add grpc registry server
9841637
Signed-off-by: tokoko <[email protected]>
@tokoko
fix: Add allow_cache to base registry get_request_feature_view
cf0c441
Signed-off-by: tokoko <[email protected]>
@tokoko
fix: registry server linter issues
cfe1585
Signed-off-by: tokoko <[email protected]>
@sudohainguyen

sudohainguyen commented Feb 2, 2024

Copy link
Copy Markdown
Collaborator

do you mind adding a quick guide how to define registry in feature_store.yaml within the PR description? so that we can update documentation accordingly in another PR

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sudohainguyen
sudohainguyen self-requested a review February 2, 2024 03:43
@tokoko

tokoko commented Feb 2, 2024

Copy link
Copy Markdown
Collaborator Author

Thanks for asking that. I'm trying to split the changes into smaller PRs, so the short answer right now is that you can't. This PR enables users to call feast serve_registry with a normal feature store configuration that will start the registry server on a specified port. I'm thinking of adding two different clients in subsequent PRs, a python client that will extend BaseRegistry and a java client for java feature server.

I'm not sure yet how feature_store.yaml should look like, I'm interested what others think about that. I think the cleanest approach would be to let the users use feature_store.yaml as a way to configure their clients. So, in cases when they need to access the registry directly (including the container from where feast serve_registry will be run), the config file will look something like this:

project: <your project name>
...
registry:
    registry_type: sql
    path: postgresql://postgres:[email protected]:55001/feast
    cache_ttl_seconds: 60

For applications where clients (java or python) connect to registry server, it might look something like this (this is still unimplemented!!!):

project: <your project name>
...
registry:
    registry_type: server
    url: grpc://127.0.0.1:8888

The biggest downside obviously is that you will need to have two different configurations depending on how you want to configure your client, but imho that's more of a feature than a bug. For example if a feast admin wants to force users to exclusively use registry server rather than connect to underlying implementation, they would be able to only share server information and hide implementation details. The same approach could in future be extrapolated to online store configuration, if you would like your store.get_online_features call to query feature server rather than an underlying online store, you would configure your store something like this:

project: <your project name>
...
online_store:
  type: server
  url: http://127.0.0.1:80 // or grpc://127.0.0.1:80

I definitely want to know what you guys think before delving into the implementations too much.

shuchu reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sudohainguyen

sudohainguyen commented Feb 2, 2024

Copy link
Copy Markdown
Collaborator

@tokoko got your ideas, and I believe this is a pretty great one, which reduces headaches connecting components

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sudohainguyen
sudohainguyen reviewed Feb 2, 2024
View reviewed changes
Comment thread sdk/python/feast/cli.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@sudohainguyen sudohainguyen added approved ok-to-test labels Feb 3, 2024
sudohainguyen
sudohainguyen approved these changes Feb 4, 2024
View reviewed changes

@sudohainguyen sudohainguyen left a comment

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
haoxu0
haoxu0 approved these changes Feb 6, 2024
View reviewed changes

@haoxu0 haoxu0 left a comment

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Realy great work. Nothing to comment on.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@haoxu0
haoxu0 merged commit 373e624 into feast-dev:master Feb 6, 2024
zseta pushed a commit to zseta/feast that referenced this pull request Feb 7, 2024
@tokoko @zseta
feat: Add gRPC Registry Server (feast-dev#3924)
ce94503
Signed-off-by: Attila Toth <[email protected]>
@tokoko
tokoko deleted the registry-server branch February 7, 2024 23:44
@tokoko tokoko mentioned this pull request Feb 28, 2024
Feature Request: InfluxDB ReadOnly On/Offline, use Waitress instead of Gunicorn, and integrate Feast.client into repo #3973
Closed
tqtensor pushed a commit to tqtensor/feast that referenced this pull request Mar 11, 2024
@tokoko @tqtensor
feat: Add gRPC Registry Server (feast-dev#3924)
e3f1a4c
@tokoko tokoko mentioned this pull request Mar 19, 2024
Feast UI: URL scheme "postgresql" is not supported when specifying SQL registry in projects-list.json #3974
Closed
franciscojavierarceo pushed a commit that referenced this pull request Apr 16, 2024
@feast-ci-bot
chore(release): release 0.36.0
474eb62
# [0.36.0](v0.35.0...v0.36.0) (2024-04-16)

### Bug Fixes

* Add __eq__, __hash__ to SparkSource for correct comparison ([#4028](#4028)) ([e703b40](e703b40))
* Add conn.commit() to Postgresonline_write_batch.online_write_batch ([#3904](#3904)) ([7d75fc5](7d75fc5))
* Add missing __init__.py to embedded_go ([#4051](#4051)) ([6bb4c73](6bb4c73))
* Add missing init files in infra utils ([#4067](#4067)) ([54910a1](54910a1))
* Added registryPath parameter documentation in WebUI reference ([#3983](#3983)) ([5e0af8f](5e0af8f)), closes [#3974](#3974) [#3974](#3974)
* Adding missing init files in materialization modules ([#4052](#4052)) ([df05253](df05253))
* Allow trancated timestamps when converting ([#3861](#3861)) ([bdd7dfb](bdd7dfb))
* Azure blob storage support in Java feature server ([#2319](#2319)) ([#4014](#4014)) ([b9aabbd](b9aabbd))
* Bugfix for grabbing historical data from Snowflake with array type features. ([#3964](#3964)) ([1cc94f2](1cc94f2))
* Bytewax materialization engine fails when loading feature_store.yaml ([#3912](#3912)) ([987f0fd](987f0fd))
* CI unittest warnings ([#4006](#4006)) ([0441b8b](0441b8b))
* Correct the returning class proto type of StreamFeatureView to StreamFeatureViewProto instead of FeatureViewProto. ([#3843](#3843)) ([86d6221](86d6221))
* Create index only if not exists during MySQL online store update ([#3905](#3905)) ([2f99a61](2f99a61))
* Disable minio tests in workflows on master and nightly ([#4072](#4072)) ([c06dda8](c06dda8))
* Disable the Feast Usage feature by default. ([#4090](#4090)) ([b5a7013](b5a7013))
* Dump repo_config by alias ([#4063](#4063)) ([e4bef67](e4bef67))
* Extend SQL registry config with a sqlalchemy_config_kwargs key ([#3997](#3997)) ([21931d5](21931d5))
* Feature Server image startup in OpenShift clusters ([#4096](#4096)) ([9efb243](9efb243))
* Fix copy method for StreamFeatureView ([#3951](#3951)) ([cf06704](cf06704))
* Fix for materializing entityless feature views in Snowflake ([#3961](#3961)) ([1e64c77](1e64c77))
* Fix type mapping spark ([#4071](#4071)) ([3afa78e](3afa78e))
* Fix typo as the cli does not support shortcut-f option. ([#3954](#3954)) ([dd79dbb](dd79dbb))
* Get container host addresses from testcontainers ([#3946](#3946)) ([2cf1a0f](2cf1a0f))
* Handle ComplexFeastType to None comparison ([#3876](#3876)) ([fa8492d](fa8492d))
* Hashlib md5 errors in FIPS for python 3.9+ ([#4019](#4019)) ([6d9156b](6d9156b))
* Making the query_timeout variable as optional int because upstream is considered to be optional ([#4092](#4092)) ([fd5b620](fd5b620))
* Move gRPC dependencies to an extra ([#3900](#3900)) ([f93c5fd](f93c5fd))
* Prevent spamming pull busybox from dockerhub ([#3923](#3923)) ([7153cad](7153cad))
* Quickstart notebook example ([#3976](#3976)) ([b023aa5](b023aa5))
* Raise error when not able read of file source spark source ([#4005](#4005)) ([34cabfb](34cabfb))
* remove not use input parameter in spark source ([#3980](#3980)) ([7c90882](7c90882))
* Remove parentheses in pull_latest_from_table_or_query ([#4026](#4026)) ([dc4671e](dc4671e))
* Remove proto-plus imports ([#4044](#4044)) ([ad8f572](ad8f572))
* Remove unnecessary dependency on mysqlclient ([#3925](#3925)) ([f494f02](f494f02))
* Restore label check for all actions using pull_request_target ([#3978](#3978)) ([591ba4e](591ba4e))
* Revert mypy config ([#3952](#3952)) ([6b8e96c](6b8e96c))
* Rewrite Spark materialization engine to use mapInPandas ([#3936](#3936)) ([dbb59ba](dbb59ba))
* Run feature server w/o gunicorn on windows ([#4024](#4024)) ([584e9b1](584e9b1))
* SqlRegistry _apply_object update statement ([#4042](#4042)) ([ef62def](ef62def))
* Substrait ODFVs for online ([#4064](#4064)) ([26391b0](26391b0))
* Swap security label check on the PR title validation job to explicit permissions instead ([#3987](#3987)) ([f604af9](f604af9))
* Transformation server doesn't generate files from proto ([#3902](#3902)) ([d3a2a45](d3a2a45))
* Trino as an OfflineStore Access Denied when BasicAuthenticaion ([#3898](#3898)) ([49d2988](49d2988))
* Trying to import pyspark lazily to avoid the dependency on the library ([#4091](#4091)) ([a05cdbc](a05cdbc))
* Typo Correction in Feast UI Readme ([#3939](#3939)) ([c16e5af](c16e5af))
* Update actions/setup-python from v3 to v4 ([#4003](#4003)) ([ee4c4f1](ee4c4f1))
* Update typeguard version to >=4.0.0 ([#3837](#3837)) ([dd96150](dd96150))
* Upgrade sqlalchemy from 1.x to 2.x regarding PVE-2022-51668. ([#4065](#4065)) ([ec4c15c](ec4c15c))
* Use CopyFrom() instead of __deepycopy__() for creating a copy of protobuf object. ([#3999](#3999)) ([5561b30](5561b30))
* Using version args to install the correct feast version ([#3953](#3953)) ([b83a702](b83a702))
* Verify the existence of Registry tables in snowflake before calling CREATE sql command. Allow read-only user to call feast apply. ([#3851](#3851)) ([9a3590e](9a3590e))

### Features

* Add duckdb offline store ([#3981](#3981)) ([161547b](161547b))
* Add Entity df in format of a Spark Dataframe instead of just pd.DataFrame or string for SparkOfflineStore ([#3988](#3988)) ([43b2c28](43b2c28))
* Add gRPC Registry Server ([#3924](#3924)) ([373e624](373e624))
* Add local tests for s3 registry using minio ([#4029](#4029)) ([d82d1ec](d82d1ec))
* Add python bytes to array type conversion support proto ([#3874](#3874)) ([8688acd](8688acd))
* Add python client for remote registry server ([#3941](#3941)) ([42a7b81](42a7b81))
* Add Substrait-based ODFV transformation ([#3969](#3969)) ([9e58bd4](9e58bd4))
* Add support for arrays in snowflake ([#3769](#3769)) ([8d6bec8](8d6bec8))
* Added delete_table to redis online store ([#3857](#3857)) ([03dae13](03dae13))
* Adding support for Native Python feature transformations for ODFVs ([#4045](#4045)) ([73bc853](73bc853))
* Bumping requirements ([#4079](#4079)) ([1943056](1943056))
* Decouple transformation types from ODFVs ([#3949](#3949)) ([0a9fae8](0a9fae8))
* Dropping Python 3.8 from local integration tests and integration tests ([#3994](#3994)) ([817995c](817995c))
* Dropping python 3.8 requirements files from the project. ([#4021](#4021)) ([f09c612](f09c612))
* Dropping the support for python 3.8 version from feast ([#4010](#4010)) ([a0f7472](a0f7472))
* Dropping unit tests for Python 3.8 ([#3989](#3989)) ([60f24f9](60f24f9))
* Enable Arrow-based columnar data transfers  ([#3996](#3996)) ([d8d7567](d8d7567))
* Enable Vector database and retrieve_online_documents API ([#4061](#4061)) ([ec19036](ec19036))
* Kubernetes materialization engine written based on bytewax ([#4087](#4087)) ([7617bdb](7617bdb))
* Lint with ruff ([#4043](#4043)) ([7f1557b](7f1557b))
* Make arrow primary interchange for offline ODFV execution ([#4083](#4083)) ([9ed0a09](9ed0a09))
* Pandas v2 compatibility ([#3957](#3957)) ([64459ad](64459ad))
* Pull duckdb from contribs, add to CI ([#4059](#4059)) ([318a2b8](318a2b8))
* Refactor ODFV schema inference ([#4076](#4076)) ([c50a9ff](c50a9ff))
* Refactor registry caching logic into a separate class ([#3943](#3943)) ([924f944](924f944))
* Rename OnDemandTransformations to Transformations ([#4038](#4038)) ([9b98eaf](9b98eaf))
* Revert updating dependencies so that feast can be run on 3.11. ([#3968](#3968)) ([d3c68fb](d3c68fb)), closes [#3958](#3958)
* Rewrite ibis point-in-time-join w/o feast abstractions ([#4023](#4023)) ([3980e0c](3980e0c))
* Support s3gov schema by snowflake offline store during materialization ([#3891](#3891)) ([ea8ad17](ea8ad17))
* Update odfv test ([#4054](#4054)) ([afd52b8](afd52b8))
* Update pyproject.toml to use Python 3.9 as default ([#4011](#4011)) ([277b891](277b891))
* Update the Pydantic from v1 to v2 ([#3948](#3948)) ([ec11a7c](ec11a7c))
* Updating dependencies so that feast can be run on 3.11. ([#3958](#3958)) ([59639db](59639db))
* Updating protos to separate transformation ([#4018](#4018)) ([c58ef74](c58ef74))

### Reverts

* Reverting bumping requirements ([#4081](#4081)) ([1ba65b4](1ba65b4)), closes [#4079](#4079)
* Verify the existence of Registry tables in snowflake… ([#3907](#3907)) ([c0d358a](c0d358a)), closes [#3851](#3851)
@tokoko tokoko mentioned this pull request May 2, 2024
Remote offline feature server deployment #4032
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@haoxu0 haoxu0 haoxu0 approved these changes
@sudohainguyen sudohainguyen sudohainguyen approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

approved ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement Python Registry Server

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@tokoko @sudohainguyen @haoxu0
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.