GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Optimize SQL Registry proto() method by EXPEbdodla · Pull Request #4425 · feast-dev/feast

Conversation

@EXPEbdodla

@EXPEbdodla EXPEbdodla commented Aug 20, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Problem Summary: We are using SQL Registry with 150+ projects and ~600 feature views at the moment. They are going to grow in future.
1 . Starting up a Remote registry service is taking longer than expected due to proto() call during Registry initialization. This has to refresh all 150 projects metadata on start up.
2 . Refreshing the cache on a regular interval with cache_mode = thread taking longer even though most of the projects object (like entities, FVs, ODFVs) doesn't change so often.

Fixes:

  1. Running the proto() method using ThreadPoolExecutor to run in parallel with 5 default workers. This doesn't fully solve the problem. We may need to look into the lazy caching options which reduces the start up times and load only the projects for which it's serving the calls rather than loading all projects into the cache. Along with that we may need to control the size of cache_registry_proto object with some eviction policy for the cache.
  2. Leveraging the previously cached RegistryProto if the last_updated_timestamp of the project hasn't changed in the feast_metadata table. Reduces the number of database calls we make.
  3. Adding last_updated_timestamp column to ProjectMetadata table
  4. Added Indexes to the tables
  5. _maybe_init_project_metadata is called multiple times (Ex: In get, list and apply). Proto() method depends on list calls (~10 calls per project. With 150 projects, it just makes the function call 1500 times which is not needed)
  6. _get_all_projects() method is using individual tables to get the projects list instead of using feast_metadata table
  7. Added get_all_projects(), get_project_metadata() and delete_project() methods to SQL Registry.
  8. Updated test case for ODFV backward compatibility issue
  9. Added tests cases to ProjectMetadata object
  10. Added integration tests for the changes
  11. Switched sqlite fixture scope to "function". sqlite is not thread safe so added thread_pool_executor_worker_count configuration. If its 0, doesn't use ThreadPoolExecutor.

Questions:

  1. It make sense to refresh the proto() with all projects for remote registry service if SQL registry is the proxy. If users pass the project_name as part of Repo Config, proto() can only refresh the project passed. I'm not sure how to make the distinction when registry is a Proxy registry for Registry Service vs Regular Registry. Appreciate any thoughts on this.

Which issue(s) this PR fixes:

Misc

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo and j1wonpark reacted with rocket emoji
All reactions
  • 🚀 2 reactions
Bhargav Dodla added 2 commits August 19, 2024 18:34
Bhargav Dodla
fix: Optimize SQL Registry proto() method
214cc31
Signed-off-by: Bhargav Dodla <[email protected]>
Bhargav Dodla
fix: Fixed linting issue
3a0dee8
Signed-off-by: Bhargav Dodla <[email protected]>
@EXPEbdodla
EXPEbdodla force-pushed the registry__proto_optimization branch from 0d5b008 to 3a0dee8 Compare August 20, 2024 01:34
@franciscojavierarceo

franciscojavierarceo commented Aug 20, 2024

Copy link
Copy Markdown
Member

This is great! Would you mind adding maybe a test or two? Since we're changing the function names and quite a bit of the code it'd be helpful.

Separately, it would be super cool to do a blood post to hear about your experience scaling Feast, if you'd be open to it!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Bhargav Dodla
fix: Added tests to the optimization changes
8de470e
Signed-off-by: Bhargav Dodla <[email protected]>
@EXPEbdodla

EXPEbdodla commented Aug 20, 2024

Copy link
Copy Markdown
Contributor Author

This is great! Would you mind adding maybe a test or two? Since we're changing the function names and quite a bit of the code it'd be helpful.

Separately, it would be super cool to do a blood post to hear about your experience scaling Feast, if you'd be open to it!

@franciscojavierarceo Sorry. I would have marked this as a Draft PR. Added tests, integration tests for the changes.

I assume you are asking for blog post. It's in our plan to publish a blog. We will share the details when we have a Draft ready.

franciscojavierarceo reacted with hooray emoji franciscojavierarceo reacted with rocket emoji
All reactions
  • 🎉 1 reaction
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo
franciscojavierarceo approved these changes Aug 20, 2024
View reviewed changes

@franciscojavierarceo franciscojavierarceo left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this! /lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@franciscojavierarceo
franciscojavierarceo enabled auto-merge (squash) August 20, 2024 23:47
@tokoko

tokoko commented Aug 22, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

@EXPEbdodla I think this definitely needs to be broken up into several PRs.

  • Let's handle the new project methods in the registry first. I'm not really comfortable with adding new methods in a single implementation only. What's blocking us from changing BaseRegistry right from the start?
  • Now there's also authorization to think of as well. I'm particularly not sure about how delete_project can fit in there, right now we define permissions on FeastObject-level. Trying to fit an endpoint like delete_project might be a headache.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tokoko
tokoko disabled auto-merge August 22, 2024 11:54
@franciscojavierarceo

franciscojavierarceo commented Aug 22, 2024

Copy link
Copy Markdown
Member
  • Now there's also authorization to think of as well. I'm particularly not sure about how delete_project can fit in there, right now we define permissions on FeastObject-level. Trying to fit an endpoint like delete_project might be a headache.

An endpoint like delete_project is likely quite valuable but should be configurable by the users on who has the right access to execute it. I don't think we have to be prescriptive but I would think this is configurable by our new RBAC functionality, right?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@EXPEbdodla

EXPEbdodla commented Aug 22, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Let's handle the new project methods in the registry first. I'm not really comfortable with adding new methods in a single implementation only. What's blocking us from changing BaseRegistry right from the start?

@tokoko I haven't used the other registry's so I'm cautious in touching them at the moment. Do we need to do that before merging this PR?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@EXPEbdodla

EXPEbdodla commented Aug 24, 2024

Copy link
Copy Markdown
Contributor Author

@tokoko @franciscojavierarceo Created PR #4441 to add methods to base registry. I'll update this PR to cover the tests cases fully along with the changes for this.

tokoko reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jeremyary jeremyary added the ok-to-test label Aug 27, 2024
@EXPEbdodla EXPEbdodla mentioned this pull request Sep 3, 2024
feat: Added Project object to Feast Objects #4475
Merged
@EXPEbdodla

EXPEbdodla commented Sep 19, 2024

Copy link
Copy Markdown
Contributor Author

Closing this. This is split into multiple PRs already. Thanks everyone for inputs.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@EXPEbdodla EXPEbdodla closed this Sep 19, 2024
@EXPEbdodla EXPEbdodla mentioned this pull request Sep 19, 2024
SQL registry initialization fails #4543
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@franciscojavierarceo franciscojavierarceo franciscojavierarceo approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@EXPEbdodla @franciscojavierarceo @tokoko @jeremyary
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.