GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Cassandra online store, concurrent fetching for multiple entities by hemidactylus · Pull Request #3356 · feast-dev/feast

Conversation

@hemidactylus

@hemidactylus hemidactylus commented Nov 21, 2022

Copy link
Copy Markdown
Contributor

This changes the retrieval of features from the Cassandra online store by leveraging the
Cassandra driver's native concurrency capabilities.

When there are several entities to be retrieved, instead of a sequential read one-by-one, entity after entity,
the reads are executed concurrently, with the driver ensuring the results are kept in the correct order and the call
returns when all results are available.
This, as measured in realistic environments, implies a speedup of 2-3x for retrieval of 20 to 100 entities at once.

Using the Cassandra driver's execute_concurrent_with_args function requires a new parameter controlling the maximum amount of concurrency to use (somewhat bounded by the number of vCPUs at hand): for transparency, this is exposed in the feature store configuration yaml as a new parameter, which is documented and correctly handled by the guided procedure of feast init -t cassandra.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@feast-ci-bot feast-ci-bot added approved size/L labels Nov 21, 2022
@sl-at-ibm
concurrent fetching for multiple entities
e9c04f9 
minimal handling of exceptions in concurrent query execution
read_concurrency parameter in Cassandra online store config yaml

Signed-off-by: Stefano Lottini <[email protected]>
@hemidactylus
hemidactylus requested a review from achals November 22, 2022 10:39
@hemidactylus

hemidactylus commented Nov 29, 2022

Copy link
Copy Markdown
Contributor Author

/lgtm
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@feast-ci-bot

feast-ci-bot commented Nov 29, 2022

Copy link
Copy Markdown
Collaborator

@hemidactylus: you cannot LGTM your own PR.

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
adchia
adchia approved these changes Nov 29, 2022
View reviewed changes

@adchia adchia left a comment

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@feast-ci-bot feast-ci-bot assigned adchia Nov 29, 2022
@feast-ci-bot feast-ci-bot added the lgtm label Nov 29, 2022
@feast-ci-bot

feast-ci-bot commented Nov 29, 2022

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, hemidactylus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [adchia,hemidactylus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@feast-ci-bot
feast-ci-bot merged commit 00fa21f into feast-dev:master Nov 29, 2022
@hemidactylus
hemidactylus deleted the sl-cassandra-optimize-bulk-reads branch November 29, 2022 15:17
@hemidactylus hemidactylus mentioned this pull request Nov 30, 2022
feat: Cassandra online store, concurrency in bulk write operations #3367
Merged
kevjumba pushed a commit that referenced this pull request Dec 5, 2022
@feast-ci-bot
chore(release): release 0.27.0
1d3c111 
# [0.27.0](v0.26.0...v0.27.0) (2022-12-05)

### Bug Fixes

* Changing Snowflake template code to avoid query not implemented … ([#3319](#3319)) ([1590d6b](1590d6b))
* Dask zero division error if parquet dataset has only one partition ([#3236](#3236)) ([69e4a7d](69e4a7d))
* Enable Spark materialization on Yarn ([#3370](#3370)) ([0c20a4e](0c20a4e))
* Ensure that Snowflake accounts for number columns that overspecify precision ([#3306](#3306)) ([0ad0ace](0ad0ace))
* Fix memory leak from usage.py not properly cleaning up call stack ([#3371](#3371)) ([a0c6fde](a0c6fde))
* Fix workflow to contain env vars ([#3379](#3379)) ([548bed9](548bed9))
* Update bytewax materialization ([#3368](#3368)) ([4ebe00f](4ebe00f))
* Update the version counts ([#3378](#3378)) ([8112db5](8112db5))
* Updated AWS Athena template ([#3322](#3322)) ([5956981](5956981))
* Wrong UI data source type display ([#3276](#3276)) ([8f28062](8f28062))

### Features

* Cassandra online store, concurrency in bulk write operations ([#3367](#3367)) ([eaf354c](eaf354c))
* Cassandra online store, concurrent fetching for multiple entities ([#3356](#3356)) ([00fa21f](00fa21f))
* Get Snowflake Query Output As Pyspark Dataframe ([#2504](#2504)) ([#3358](#3358)) ([2f18957](2f18957))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adchia adchia adchia approved these changes
@achals achals Awaiting requested review from achals

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@adchia adchia

Labels

approved lgtm size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@hemidactylus @feast-ci-bot @adchia @sl-at-ibm
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.