GeezerVison is showing github.com — fix: Optimize SQL Registry proto() method by EXPEbdodla · Pull Request #4425 · feast-dev/feast
Conversation
EXPEbdodla
commented
Aug 20, 2024
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
What this PR does / why we need it:
Problem Summary: We are using SQL Registry with 150+ projects and ~600 feature views at the moment. They are going to grow in future.
1 . Starting up a Remote registry service is taking longer than expected due to proto() call during Registry initialization. This has to refresh all 150 projects metadata on start up.
2 . Refreshing the cache on a regular interval with cache_mode = thread taking longer even though most of the projects object (like entities, FVs, ODFVs) doesn't change so often.
Fixes:
- Running the proto() method using ThreadPoolExecutor to run in parallel with 5 default workers. This doesn't fully solve the problem. We may need to look into the lazy caching options which reduces the start up times and load only the projects for which it's serving the calls rather than loading all projects into the cache. Along with that we may need to control the size of cache_registry_proto object with some eviction policy for the cache.
- Leveraging the previously cached RegistryProto if the last_updated_timestamp of the project hasn't changed in the feast_metadata table. Reduces the number of database calls we make.
- Adding last_updated_timestamp column to ProjectMetadata table
- Added Indexes to the tables
- _maybe_init_project_metadata is called multiple times (Ex: In get, list and apply). Proto() method depends on list calls (~10 calls per project. With 150 projects, it just makes the function call 1500 times which is not needed)
- _get_all_projects() method is using individual tables to get the projects list instead of using feast_metadata table
- Added get_all_projects(), get_project_metadata() and delete_project() methods to SQL Registry.
- Updated test case for ODFV backward compatibility issue
- Added tests cases to ProjectMetadata object
- Added integration tests for the changes
- Switched sqlite fixture scope to "function". sqlite is not thread safe so added
thread_pool_executor_worker_count configuration. If its 0, doesn't use ThreadPoolExecutor.
Questions:
- It make sense to refresh the proto() with all projects for remote registry service if SQL registry is the proxy. If users pass the project_name as part of Repo Config, proto() can only refresh the project passed. I'm not sure how to make the distinction when registry is a Proxy registry for Registry Service vs Regular Registry. Appreciate any thoughts on this.
Which issue(s) this PR fixes:
Misc
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
franciscojavierarceo and j1wonpark reacted with rocket emoji
|
This is great! Would you mind adding maybe a test or two? Since we're changing the function names and quite a bit of the code it'd be helpful.
Separately, it would be super cool to do a blood post to hear about your experience scaling Feast, if you'd be open to it!
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This is great! Would you mind adding maybe a test or two? Since we're changing the function names and quite a bit of the code it'd be helpful.
Separately, it would be super cool to do a blood post to hear about your experience scaling Feast, if you'd be open to it!
@franciscojavierarceo Sorry. I would have marked this as a Draft PR. Added tests, integration tests for the changes.
I assume you are asking for blog post. It's in our plan to publish a blog. We will share the details when we have a Draft ready.
|
franciscojavierarceo reacted with hooray emoji
franciscojavierarceo reacted with rocket emoji
All reactions
-
🎉
1 reaction
-
🚀
1 reaction
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this! /lgtm
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
tokoko
commented
Aug 22, 2024
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
@EXPEbdodla I think this definitely needs to be broken up into several PRs.
- Let's handle the new project methods in the registry first. I'm not really comfortable with adding new methods in a single implementation only. What's blocking us from changing
BaseRegistry right from the start?
- Now there's also authorization to think of as well. I'm particularly not sure about how
delete_project can fit in there, right now we define permissions on FeastObject-level. Trying to fit an endpoint like delete_project might be a headache.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
- Now there's also authorization to think of as well. I'm particularly not sure about how delete_project can fit in there, right now we define permissions on FeastObject-level. Trying to fit an endpoint like delete_project might be a headache.
An endpoint like delete_project is likely quite valuable but should be configurable by the users on who has the right access to execute it. I don't think we have to be prescriptive but I would think this is configurable by our new RBAC functionality, right?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
EXPEbdodla
commented
Aug 22, 2024
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Let's handle the new project methods in the registry first. I'm not really comfortable with adding new methods in a single implementation only. What's blocking us from changing BaseRegistry right from the start?
@tokoko I haven't used the other registry's so I'm cautious in touching them at the moment. Do we need to do that before merging this PR?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
tokoko reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Closing this. This is split into multiple PRs already. Thanks everyone for inputs.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|