GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Implement gRPC server to ingest streaming features by mehmettokgoz · Pull Request #3687 · feast-dev/feast

Conversation

@mehmettokgoz

@mehmettokgoz mehmettokgoz commented Jul 17, 2023

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Currently only way to ingest streaming features to Feast is using the Spark/Kafka processor. This processor has several limitations. It retrieves data in batches and forces to run all transformations in Python runtime.

The gRPC server that listens for the streaming features allows users to apply transformation on more powerful streaming engines and send it to Feast then. This feature introduces new streaming sources for Feast, in addition to Kafka and Kinesis.

Our main purpose for the gRPC ingestion service is to integrate Hazelcast streaming engine with Feast. Hazelcast provides various connectors, and supports to process those sources in real-time. This server makes possible to sink transformed data to Feast.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@feast-ci-bot

feast-ci-bot commented Jul 17, 2023

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mehmettokgoz
To complete the pull request process, please assign kevjumba after the PR has been reviewed.
You can assign the PR to them by writing /assign @kevjumba in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:
  • OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@feast-ci-bot feast-ci-bot added the size/L label Jul 17, 2023
@mehmettokgoz mehmettokgoz changed the title feat: gRPC server implementation for ingesting streaming features feat: Implement gRPC server to ingest streaming features Jul 17, 2023
adchia
adchia reviewed Jul 23, 2023
View reviewed changes
Comment thread sdk/python/feast/infra/contrib/grpc_server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread sdk/python/feast/infra/contrib/grpc_server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@mehmettokgoz
mehmettokgoz requested a review from adchia July 27, 2023 20:18
@mehmettokgoz

mehmettokgoz commented Jul 27, 2023

Copy link
Copy Markdown
Contributor Author

Thanks for review @adchia. Waiting for your answers.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

adchia
adchia reviewed Aug 3, 2023
View reviewed changes
Comment thread sdk/python/feast/infra/contrib/grpc_server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread sdk/python/feast/infra/contrib/grpc_server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread sdk/python/feast/infra/contrib/grpc_server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread sdk/python/feast/infra/contrib/grpc_server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread sdk/python/feast/infra/contrib/grpc_server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@mehmettokgoz
mehmettokgoz requested a review from adchia August 7, 2023 09:56
@mehmettokgoz

mehmettokgoz commented Aug 7, 2023

Copy link
Copy Markdown
Contributor Author

@adchia addressed review comments. ✅

Changes:

  • Do not force only one feature view. Proto object contains a feature_view_name field.
  • Push API is supported.
  • Health check service is added to server.
  • grpcio packages are upgraded to 1.56.2
  • CLI is refactored.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@crispin-ki

crispin-ki commented Aug 15, 2023

Copy link
Copy Markdown
Contributor

I saw a comment about tests failing in this PR in the community notes.

Failing python unit tests should be fixed by #3734. Once that PR is merged you can rebase this branch on top of master and the failing tests should pass.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mehmettokgoz

mehmettokgoz commented Aug 17, 2023

Copy link
Copy Markdown
Contributor Author

Thanks @crispin-ki

Do you have any idea why linter might be failing? The errors are not related to the changes in this PR and I see it's green for your PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@crispin-ki

crispin-ki commented Aug 17, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Thanks @crispin-ki

Do you have any idea why linter might be failing? The errors are not related to the changes in this PR and I see it's green for your PR.

Yeah it's a bit weird that these are failing now, seeing as you didn't touch them. I think you have a few options though:

  • logs for failing workflow (pasted at bottom of this comment too) are all E721s, so you could add E721 here so those errors are now ignored altogether
  • alternatively you could go to each line that is failing and add # noqa: E721 at the end of line similar to here but with E721 instead, so flake doesn't check that line for E721
  • alternatively you could go to each line that is failing and change them so flake is happy (in most cases change == to is)

In terms of which is preferred, I think all have pros and cons. I think just go with whichever you feel is most appropriate (or if unsure, might be one for maintainers to decide - they might in fact have a better solution )

Logs for the failing workflow:

cd /home/runner/work/feast/feast/sdk/python; python -m flake8 feast/ tests/
feast/diff/registry_diff.py:[13](https://github.com/feast-dev/feast/actions/runs/5887487203/job/15966956374?pr=3687#step:9:14)6:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
feast/feature_store.py:1109:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
feast/field.py:65:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
feast/infra/offline_stores/bigquery.py:728:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
feast/type_map.py:748:8: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/unit/local_feast_tests/test_local_feature_store.py:472:20: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/unit/test_feature_views.py:183:[16](https://github.com/feast-dev/feast/actions/runs/5887487203/job/15966956374?pr=3687#step:9:17): E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mehmettokgoz

mehmettokgoz commented Aug 17, 2023

Copy link
Copy Markdown
Contributor Author

@crispin-ki

I found that it is related to new version of flake8 (which upgrades pycodestyle). Now pycodetype check for EE721 (https://github.com/PyCQA/pycodestyle/blob/main/CHANGES.txt#L12). I set flake version to >=6.0.0,<6.1.0.

crispin-ki reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mehmettokgoz
mehmettokgoz force-pushed the grpc-server branch from f12dd5a to 43a2c91 Compare August 28, 2023 15:37
@adchia
adchia force-pushed the grpc-server branch from 43a2c91 to 1c074f0 Compare September 5, 2023 08:05
mehmettokgoz and others added 12 commits September 5, 2023 01:09
mehmettokgoz @adchia
Implemented gRPC server for ingesting streaming features.
b235d1a
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
[lint-python]
1b4bf8d
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
Fix unmatched gRPC field issue.
809b9eb
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
Mehmet Tokgöz @adchia
Make max_workers parameter configurable.
5ea58d1
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
[lint-python]
9766c53
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
Refactor gRPC server to have similar logic with HTTP server
b92b838
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
Refactor CLI.
caed020
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
Type parameter.
cd537d9
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
Upgrade 3.8 requirements.
d73c3d7
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
Configure health check service and update CI dependencies
a5000d8
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
mehmettokgoz @adchia
Downgrade flake8 version to prevent E721 checks.
be0010d
Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
@adchia
lint
5f73e77
Signed-off-by: Danny C <[email protected]>
@adchia
adchia force-pushed the grpc-server branch from bc36d7d to 5f73e77 Compare September 5, 2023 08:09
@adchia

adchia commented Sep 5, 2023

Copy link
Copy Markdown
Collaborator

/lgtm

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

adchia
adchia approved these changes Sep 5, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

View reviewed changes
@adchia

adchia commented Sep 5, 2023

Copy link
Copy Markdown
Collaborator

/lgtm

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

adchia
adchia approved these changes Sep 5, 2023
View reviewed changes
@adchia

adchia commented Sep 5, 2023

Copy link
Copy Markdown
Collaborator

/lgtm /approve

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@adchia adchia added approved lgtm labels Sep 5, 2023
adchia added 3 commits September 5, 2023 10:45
@adchia
fix grpcio pin
392768a
Signed-off-by: Danny C <[email protected]>
@adchia
fix lint
0c75491
Signed-off-by: Danny C <[email protected]>
@adchia
Fix linter
068de80
Signed-off-by: Danny C <[email protected]>
@adchia
adchia merged commit a3fcd1f into feast-dev:master Sep 7, 2023
adchia pushed a commit that referenced this pull request Sep 7, 2023
@feast-ci-bot
chore(release): release 0.34.0
a8aeee9
# [0.34.0](v0.33.0...v0.34.0) (2023-09-07)

### Bug Fixes

* Add NUMERIC to bq_to_feast type map ([#3719](#3719)) ([6474b4b](6474b4b))
* Fix python unit tests ([#3734](#3734)) ([e81684d](e81684d))
* Handle unknown postgres source types gracefully ([#3634](#3634)) ([d7041f4](d7041f4))
* Pin protobuf version to avoid seg fault on some machines ([028cc20](028cc20))
* Remove unwanted excessive splitting of gcs path, so expected gcs parquet paths are returned from BigQueryRetrievalJob.to_remote_storage() ([#3730](#3730)) ([f2c5988](f2c5988))
* Run store.plan() only when need it. ([#3708](#3708)) ([7bc7c47](7bc7c47))
* Saved datasets no longer break CLI registry-dump command ([#3717](#3717)) ([f28ccc2](f28ccc2))
* Update py3.8 ci requirements for cython 3.0 release ([#3735](#3735)) ([1695c13](1695c13))

### Features

* Enhance customization of Trino connections when using Trino-based Offline Stores ([#3699](#3699)) ([ed7535e](ed7535e))
* Implement gRPC server to ingest streaming features ([#3687](#3687)) ([a3fcd1f](a3fcd1f))
cburroughs added a commit to cburroughs/feast that referenced this pull request Jan 24, 2024
@cburroughs
fix: move gRPC dependencies to an extra
0ae2ffe
PR feast-dev#3687 added a spiffy feature to ingest streaming features, but this
came along with a large batch of depdencies.  Notable this induces a
core dependency on `protobuf>=4.21.6` while Feast itself is on
`protobuf<4.23.4,>3.20`.  This is a fiddly narrow range and excludes
all 3.x uses.

Signed-off-by: Chris Burroughs <[email protected]>
@cburroughs cburroughs mentioned this pull request Jan 24, 2024
fix: Move gRPC dependencies to an extra #3900
Merged
zseta pushed a commit to zseta/feast that referenced this pull request Feb 7, 2024
@mehmettokgoz @zseta
feat: Implement gRPC server to ingest streaming features (feast-dev#3687
30e96ab
)

* Implemented gRPC server for ingesting streaming features.

Signed-off-by: mehmettokgoz <[email protected]>
Signed-off-by: Danny C <[email protected]>
Signed-off-by: Attila Toth <[email protected]>
zseta pushed a commit to zseta/feast that referenced this pull request Feb 7, 2024
@feast-ci-bot @zseta
chore(release): release 0.34.0
08e11b9
# [0.34.0](feast-dev/feast@v0.33.0...v0.34.0) (2023-09-07)

### Bug Fixes

* Add NUMERIC to bq_to_feast type map ([feast-dev#3719](feast-dev#3719)) ([6474b4b](feast-dev@6474b4b))
* Fix python unit tests ([feast-dev#3734](feast-dev#3734)) ([e81684d](feast-dev@e81684d))
* Handle unknown postgres source types gracefully ([feast-dev#3634](feast-dev#3634)) ([d7041f4](feast-dev@d7041f4))
* Pin protobuf version to avoid seg fault on some machines ([028cc20](feast-dev@028cc20))
* Remove unwanted excessive splitting of gcs path, so expected gcs parquet paths are returned from BigQueryRetrievalJob.to_remote_storage() ([feast-dev#3730](feast-dev#3730)) ([f2c5988](feast-dev@f2c5988))
* Run store.plan() only when need it. ([feast-dev#3708](feast-dev#3708)) ([7bc7c47](feast-dev@7bc7c47))
* Saved datasets no longer break CLI registry-dump command ([feast-dev#3717](feast-dev#3717)) ([f28ccc2](feast-dev@f28ccc2))
* Update py3.8 ci requirements for cython 3.0 release ([feast-dev#3735](feast-dev#3735)) ([1695c13](feast-dev@1695c13))

### Features

* Enhance customization of Trino connections when using Trino-based Offline Stores ([feast-dev#3699](feast-dev#3699)) ([ed7535e](feast-dev@ed7535e))
* Implement gRPC server to ingest streaming features ([feast-dev#3687](feast-dev#3687)) ([a3fcd1f](feast-dev@a3fcd1f))

Signed-off-by: Attila Toth <[email protected]>
cburroughs added a commit to cburroughs/feast that referenced this pull request Feb 27, 2024
@cburroughs
fix: move gRPC dependencies to an extra
104fbfe
PR feast-dev#3687 added a spiffy feature to ingest streaming features, but this
came along with a large batch of depdencies.  Notable this induces a
core dependency on `protobuf>=4.21.6` while Feast itself is on
`protobuf<4.23.4,>3.20`.  This is a fiddly narrow range and excludes
all 3.x uses.

Signed-off-by: Chris Burroughs <[email protected]>
cburroughs added a commit to cburroughs/feast that referenced this pull request Mar 5, 2024
@cburroughs
fix: move gRPC dependencies to an extra
c7a302f
PR feast-dev#3687 added a spiffy feature to ingest streaming features, but this
came along with a large batch of depdencies.  Notable this induces a
core dependency on `protobuf>=4.21.6` while Feast itself is on
`protobuf<4.23.4,>3.20`.  This is a fiddly narrow range and excludes
all 3.x uses.

Signed-off-by: Chris Burroughs <[email protected]>
cburroughs added a commit to cburroughs/feast that referenced this pull request Mar 6, 2024
@cburroughs
fix: move gRPC dependencies to an extra
ef2d13c
PR feast-dev#3687 added a spiffy feature to ingest streaming features, but this
came along with a large batch of depdencies.  Notable this induces a
core dependency on `protobuf>=4.21.6` while Feast itself is on
`protobuf<4.23.4,>3.20`.  This is a fiddly narrow range and excludes
all 3.x uses.

Signed-off-by: Chris Burroughs <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adchia adchia adchia approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

approved lgtm size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@mehmettokgoz @feast-ci-bot @crispin-ki @adchia
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.