Description

adchia

opened on Jan 31, 2022

Issue body actions

In some test benchmarks, using regular python dicts for inputs for executing the transformations is much faster (up to ~10x) than pandas for the online flow. This tends to be the more latency sensitive flow (offline flows seem to be ~40% slower if using vectorized operations).

Something that looks like:

@on_demand_feature_view(
    sources=[driver_hourly_stats_view, val_to_add_request],
    schema=[
        Field(name="conv_rate_plus_val1", dtype=Float64),
        Field(name="conv_rate_plus_val2", dtype=Float64),
    ],
    mode="python"
)
def transformed_conv_rate(driver_hourly_stats: Dict[str, Any], vals_to_add: Dict[str, Any]) -> Dict[str, Any]:
    features = {}
    features['conv_rate_plus_val1'] = (driver_hourly_stats['conv_rate'] + vals_to_add['val_to_add'])
    features['conv_rate_plus_val2'] = (driver_hourly_stats['conv_rate'] + vals_to_add['val_to_add_2'])
    return features

might be similar to what we want

Activity

1. adchia

   added

   kind/bug

   priority/p1

   area/serving

   Community Contribution NeededWe want community to contributeWe want community to contribute

   on Jan 31, 2022
2. github-actions

   assigned

   woop

   on Jan 31, 2022
3. judahrand commented on Feb 1, 2022

   

   judahrand

   on Feb 1, 2022

   Member

   More actions

   In your example what types would the driver_hourly_stats['conv_rate'] be? Would it be a list? If so this interface will require some additional transformation. Would it be better to pass Numpy Arrays? If so have you factored the instantiation of these into the ~10x speed up?
4. adchia commented on Feb 14, 2022

   

   adchia

   on Feb 14, 2022

   CollaboratorAuthor

   More actions

   Numpy arrays should also be a lot better yeah.

   It might make sense to support both, but really there's also the factor of what the user will have access to at serving time. Seems more likely to be a standard dict. numpy should def be faster, but I also worry since it's significantly more verbose.

   Could also see a world where we allow both pandas or dicts since pandas will be easier to write the transformations but less performant.

   In this specific situation, those conv_rate values with be individual doubles.
5. judahrand commented on Feb 14, 2022

   

   judahrand

   on Feb 14, 2022

   Member

   More actions

   I was actually thinking a dict of 1d Numpy arrays?
6. adchia

   changed the title [-]On demand feature views (ODFVs) should use python dicts instead of dataframes[/-] [+]On demand feature views (ODFVs) should use support python dicts[/+] on May 17, 2022
7. kevjumba

   added this to Feast Roadmapon Aug 3, 2022
8. kevjumba

   moved this to Todo in Feast Roadmapon Aug 3, 2022
9. stale commented on Sep 20, 2022

   

   staleBot

   on Sep 20, 2022

   More actions

   This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
10. stale

    added

    wontfixThis will not be worked onThis will not be worked on

    on Sep 20, 2022
11. stale

    closed this as completedon Sep 28, 2022
12. Repository owner moved this from Todo to Done in Feast Roadmapon Sep 28, 2022
13. judahrand

    reopened this on Sep 28, 2022
14. stale

    closed this as completedon Oct 14, 2022
15. judahrand

    reopened this on Oct 14, 2022
16. stale

    removed

    wontfixThis will not be worked onThis will not be worked on

    on Oct 14, 2022
17. adchia

    added

    keep-open

    on Oct 14, 2022
18. adchia

    unassigned

    woop

    on Oct 14, 2022
19. franciscojavierarceo commented on Sep 23, 2023

    

    franciscojavierarceo

    on Sep 23, 2023

    Member

    More actions

    tagging @maksstach who has implemented a version of this on our fork
20. franciscojavierarceo commented on Mar 30, 2024

    

    franciscojavierarceo

    on Mar 30, 2024

    Member

    More actions

    I did this one #4045 next I'll do writes.
21. franciscojavierarceo

    self-assigned this

    on Mar 30, 2024
22. franciscojavierarceo

    closed this as completedon Apr 2, 2024
23. franciscojavierarceo commented on Nov 21, 2024

    

    franciscojavierarceo

    on Nov 21, 2024

    Member

    More actions

    I added writes here: #4376

Issue actions