GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Handle BigQuery partitions when event_timestamp is not the partition column, deal with event_timestamp columns of DATE type · Issue #2530 · feast-dev/feast

Handle BigQuery partitions when event_timestamp is not the partition column, deal with event_timestamp columns of DATE type #2530

New issue
Copy link
New issue
Copy link
Closed
#6362
Closed
Handle BigQuery partitions when event_timestamp is not the partition column, deal with event_timestamp columns of DATE type#2530
#6362
Copy link
Labels
Community Contribution NeededWe want community to contributeWe want community to contributegood first issueGood for newcomersGood for newcomerskeep-openkind/featureNew feature or requestNew feature or requestpriority/p1

Description

@rory-nia
rory-nia
opened on Apr 13, 2022
Issue body actions

Is your feature request related to a problem? Please describe.
At a high level, I'd like to be able to install Feast, describe my sources and FVs, and immediately start working to get historical data and materialize features. However, by not being partition-aware and by not accounting for non-timestamp date columns, a large number of our tables are not usable without modification.

This feature request is specific to the GCP provider and BigQuery offline store. It has two parts: 1) relating to respecting partition when constructing date ranges in SQL, 2) relating to source tables with "timestamp" column not of TIMESTAMP type.

  1. In any table were the partition column is not the same column as the event_timestamp, Feast will ignore partitions and do a full table scan on each query. For large tables, this causes BQ to fail due to "insufficient resources" - not a slot/quote issue, but having to do with needing to marshal too much transient data. A common example with BQ is when doing DAY partitioning using ingestion time, the magic column _PARTITIONTIME is the partition column. Each record will contain its own actual event_timestamp column, but this column is not the partition column.

  2. For tables with an "event_timestamp" that is not a TIMESTAMP type, the SQL can break when there's a type mismatch, e.g. when the column is a DATE type. A common use case is a DAY partitioned summary table that performs aggregations over the day or possibly just takes the final value for each entity/date. These tables have a DATE column on each record. In this case, the DATE column is usually the partition column, but it is not the correct format for the date conditions in the Feast template SQL. I would prefer not to have to create a new column or a view on each table just to make Feast work.

Describe the solution you'd like
I'd like to be able to specify information about the partition column such that Feast could formulate SQL optimized to my table definition taking advantage of partitioning. This information would probably need to include the partition column name, type, and partition type at a minimum. I believe optimized SQL would not be much different from the existing queries but would require, in cases where the partition column is not the same as the event_timestamp column, to add an additional range condition using the partition column.

I'd also like to be able to indicate that my "event_timestamp" column is not a TIMESTAMP type - and have Feast adjust the SQL accordingly. Specifically, I'd like to specify in the Feast BigQuerySource definition that the "event_timestamp" column is the partition column and is of DATE type and have Feast make the corresponding adjustments to the templatized SQL.

Describe alternatives you've considered
To deal with (1), the only alternatives I can think of are a) creating a view that changes the partition column to the event_timestamp column for every table, b) creating a duplicate of each table using the event_timestamp column.

To deal with (2), similarly, either a) create a view with a new timestamp typed column to use as the event_timestamp, b) add a new timestamp column to the existing tables.

All these solutions require a good amount of work and/or creating duplicate tables/views. I would really like Feast to just do the right thing after I create a Source.

Additional context
Happy to provide any further details if the above is not clear.

Activity

  1. rory-nia
    added
    kind/featureNew feature or requestNew feature or request
    on Apr 13, 2022
  2. github-actions
    assigned
    woop
    on Apr 13, 2022
  3. adchia
    added
    priority/p1
    on Apr 13, 2022
  4. adchia
    added this to Feast Roadmapon Apr 21, 2022
  5. adchia
    added
    Community Contribution NeededWe want community to contributeWe want community to contribute
    on Apr 21, 2022
  6. adchia
    moved this to Todo in Feast Roadmapon May 24, 2022
  7. adchia
    unassigned
    woop
    on May 24, 2022
  8. adchia
    added
    good first issueGood for newcomersGood for newcomers
    on May 24, 2022
  9. stale commented on Sep 22, 2022

    @stale
    staleBot
    on Sep 22, 2022
    More actions

    This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

  10. stale
    added
    wontfixThis will not be worked onThis will not be worked on
    on Sep 22, 2022
  11. stale
    closed this as completedon Oct 1, 2022
  12. Repository owner moved this from Todo to Done in Feast Roadmapon Oct 1, 2022
  13. adchia
    reopened this on Oct 1, 2022
  14. stale
    removed
    wontfixThis will not be worked onThis will not be worked on
    on Oct 1, 2022
  15. adchia
    added
    wontfixThis will not be worked onThis will not be worked on
    keep-open
    and removed
    wontfixThis will not be worked onThis will not be worked on
    on Oct 1, 2022
  16. Anarion-zuo commented on Mar 4, 2026

    @Anarion-zuo
    Anarion-zuo
    on Mar 4, 2026
    Contributor
    More actions

    Added support on filtering on date_partition_column in some cases. #6076
    The PR should resolve this issue partially.

  17. Anarion-zuo commented on Mar 12, 2026

    @Anarion-zuo
    Anarion-zuo
    on Mar 12, 2026
    Contributor
    More actions

    Hi. Should someone take a look at this? #6076
    @adchia

  18. Jwrede
    added a commit that references this issue on May 14, 2026

    feat(bigquery): Support DATE-type event timestamp columns

    This issue will close once commit dafdd5e is merged into the 'master' branch.
    dafdd5e
  19. ntkathole
    added a commit that references this issue on May 14, 2026

    feat(bigquery): Support DATE-type event timestamp columns

    This issue will close once commit 00f852d is merged into the 'master' branch.
    00f852d
  20. ntkathole
    closed this as completedin #6362on May 14, 2026
  21. ntkathole
    added a commit that references this issue on May 14, 2026

    feat(bigquery): Support DATE-type event timestamp columns (feast-dev#…

    This issue will close once commit 753dee5 is merged into the 'master' branch.
    753dee5
  22. rpathade
    added 2 commits that reference this issue on May 21, 2026

    feat(bigquery): Support DATE-type event timestamp columns (feast-dev#…

    This issue will close once commit 403b527 is merged into the 'master' branch.
    403b527

    feat(bigquery): Support DATE-type event timestamp columns (feast-dev#…

    This issue will close once commit 2014acd is merged into the 'master' branch.
    2014acd
  23. ntkathole
    added a commit that references this issue on May 23, 2026

    feat: Add enabled/disabled toggle for feature views (#6401)

    This issue will close once commit 5f1fa0d is merged into the 'master' branch.
    5f1fa0d
  24. franciscojavierarceo
    added a commit that references this issue on Jun 13, 2026

    chore(release): release 0.64.0

    9bb73cc
  25. Srihari1192
    added a commit that references this issue on Jun 22, 2026

    chore(release): release 0.64.0

    22c3721
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Community Contribution NeededWe want community to contributeWe want community to contributegood first issueGood for newcomersGood for newcomerskeep-openkind/featureNew feature or requestNew feature or requestpriority/p1

    Type

    No type

    Projects

    • Feast Roadmap
      Status
      Done
      Show more project fields

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.