GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Add `date_partition_column` to `SparkSource` · Issue #4835 · feast-dev/feast

Add date_partition_column to SparkSource #4835

New issue
Copy link
New issue
Copy link
Closed
Closed
Add date_partition_column to SparkSource#4835
Copy link
Assignees
niklasvm
Labels
kind/featureNew feature or requestNew feature or request

Description

@niklasvm
niklasvm
opened on Dec 11, 2024
Issue body actions

Is your feature request related to a problem? Please describe.
The current spark implementation scans over all parquet files. This process can be made faster and more efficient by specifying a date_partition_column. During execution, this column would be used to filter the data at a file level. Only files who's date is within the range would be scanned.

Describe the solution you'd like
Add date_partition_column to SparkSource. A similar implementation exists for the AthenaSource

Describe alternatives you've considered
None

I have implemented this locally and it works. I'm happy to open a PR

Reacted by Francisco Javier Arceo
Reacted by Francisco Javier Arceo and Brett de Villiers

Activity

  1. niklasvm
    added
    kind/featureNew feature or requestNew feature or request
    on Dec 11, 2024
  2. niklasvm
    self-assigned this
    on Dec 12, 2024
  3. niklasvm
    closed this as completedon Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @niklasvm
    niklasvm

Labels

kind/featureNew feature or requestNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.