GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Support GCS filesystem for bytewax engine by sudohainguyen · Pull Request #3774 · feast-dev/feast

Conversation

@sudohainguyen

@sudohainguyen sudohainguyen commented Sep 22, 2023

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it: support gcs file for bytewax materialization job to process

Which issue(s) this PR fixes:

Fixes #

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@sudohainguyen
sudohainguyen requested a review from whoahbot as a code owner September 22, 2023 09:20
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch 2 times, most recently from 66a2237 to a514dc1 Compare September 22, 2023 10:12
whoahbot
whoahbot approved these changes Sep 22, 2023
View reviewed changes
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch 4 times, most recently from 808df18 to a92e56d Compare September 28, 2023 04:05
@sudohainguyen
sudohainguyen requested a review from whoahbot September 28, 2023 04:32
whoahbot
whoahbot approved these changes Sep 28, 2023
View reviewed changes
@sudohainguyen

sudohainguyen commented Sep 30, 2023

Copy link
Copy Markdown
Collaborator Author

hi @whoahbot , did I miss any actions here?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@whoahbot whoahbot added lgtm ok-to-test labels Oct 2, 2023
@whoahbot

whoahbot commented Oct 3, 2023

Copy link
Copy Markdown
Collaborator

hi @whoahbot , did I miss any actions here?

I don't think so. I added the required labels to this PR, but the integration tests continue to fail.

@adchia I think the integration test failures are unrelated. Is there something else that we need to do?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
mjurkus
mjurkus reviewed Oct 10, 2023
View reviewed changes
Comment thread sdk/python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_dataflow.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch 2 times, most recently from 050f623 to b5d1e45 Compare October 11, 2023 06:42
@achals achals added the approved label Oct 12, 2023
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch from 0a786a3 to 2203e3b Compare October 20, 2023 11:25
@sudohainguyen sudohainguyen self-assigned this Oct 20, 2023
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch from bb0e9bb to 10647fb Compare October 20, 2023 12:50
@sudohainguyen
fix: Support param timeout when persisting
0744f12 
Signed-off-by: Hai Nguyen <[email protected]>
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch from 10647fb to 5a1b5c2 Compare October 20, 2023 13:14
@sudohainguyen sudohainguyen closed this Oct 20, 2023
@sudohainguyen sudohainguyen reopened this Oct 20, 2023
@sudohainguyen sudohainguyen closed this Oct 20, 2023
@sudohainguyen sudohainguyen reopened this Oct 20, 2023
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch 3 times, most recently from b6b72d9 to ce60877 Compare October 20, 2023 15:32
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch from ce60877 to 8cfd5b9 Compare October 20, 2023 15:48
@sudohainguyen
fix: fix java integration test
81b4268 
Signed-off-by: Hai Nguyen <[email protected]>
@sudohainguyen
sudohainguyen force-pushed the fix/bytewax_gcs branch from 8cfd5b9 to 81b4268 Compare October 20, 2023 15:54
@achals
achals merged commit fb6b807 into feast-dev:master Oct 20, 2023
@sudohainguyen
sudohainguyen deleted the fix/bytewax_gcs branch October 21, 2023 02:12
@GytisBraz GytisBraz mentioned this pull request Oct 30, 2023
add bytewax GCS support vinted/feast#10
Merged
woop pushed a commit that referenced this pull request Jan 13, 2024
@feast-ci-bot
chore(release): release 0.35.0
9237361 
# [0.35.0](v0.34.0...v0.35.0) (2024-01-13)

### Bug Fixes

* Add async refresh to prevent synchronous refresh in main thread ([#3812](#3812)) ([9583ed6](9583ed6))
* Adopt connection pooling for HBase ([#3793](#3793)) ([b3852bf](b3852bf))
* Bytewax engine create configmap from object ([#3821](#3821)) ([25e9775](25e9775))
* Fix warnings from deprecated paths and update default log level ([#3757](#3757)) ([68a8737](68a8737))
* improve parsing bytewax job status ([5983f40](5983f40))
* make bytewax settings unexposed ([ae1bb8b](ae1bb8b))
* Make generated temp table name escaped ([#3797](#3797)) ([175d796](175d796))
* Pin numpy version to avoid spammy deprecation messages ([774ed33](774ed33))
* Redundant feature materialization and premature incremental materialization timestamp updates ([#3789](#3789)) ([417b16b](417b16b)), closes [#6](#6) [#7](#7)
* Resolve hbase hotspot issue when materializing ([#3790](#3790)) ([7376db8](7376db8))
* Set keepalives_idle None by default ([#3756](#3756)) ([8717e9b](8717e9b))
* Set upper bound for bigquery client due to its breaking changes ([2151c39](2151c39))
* UI project cannot handle fallback routes ([#3766](#3766)) ([96ece0f](96ece0f))
* update dependencies versions due to conflicts ([5dc0b24](5dc0b24))
* Update jackson and remove unnecessary logging ([#3809](#3809)) ([018d0ea](018d0ea))
* upgrade the pyarrow to latest v14.0.1 for CVE-2023-47248. ([052182b](052182b))

### Features

* Add get online feature rpc to gprc server ([#3815](#3815)) ([01db8cc](01db8cc))
* Add materialize and materialize-incremental rest endpoints ([#3761](#3761)) ([fa600fe](fa600fe)), closes [#3760](#3760)
* add redis sentinel support ([3387a15](3387a15))
* add redis sentinel support ([4337c89](4337c89))
* add redis sentinel support format lint ([aad8718](aad8718))
* Add support for `table_create_disposition` in bigquery job for offline store ([#3762](#3762)) ([6a728fe](6a728fe))
* Add support for in_cluster config and additional labels for bytewax materialization ([#3754](#3754)) ([2192e65](2192e65))
* Apply cache to load proto registry for performance ([#3702](#3702)) ([709c709](709c709))
* Make bytewax job write as mini-batches ([#3777](#3777)) ([9b0e5ce](9b0e5ce))
* Optimize bytewax pod resource with zero-copy ([9cf9d96](9cf9d96))
* Support GCS filesystem for bytewax engine ([#3774](#3774)) ([fb6b807](fb6b807))
tokoko pushed a commit to tokoko/feast that referenced this pull request Feb 6, 2024
@feast-ci-bot @tokoko
chore(release): release 0.35.0
85b5b55 
# [0.35.0](feast-dev/feast@v0.34.0...v0.35.0) (2024-01-13)

### Bug Fixes

* Add async refresh to prevent synchronous refresh in main thread ([feast-dev#3812](feast-dev#3812)) ([9583ed6](feast-dev@9583ed6))
* Adopt connection pooling for HBase ([feast-dev#3793](feast-dev#3793)) ([b3852bf](feast-dev@b3852bf))
* Bytewax engine create configmap from object ([feast-dev#3821](feast-dev#3821)) ([25e9775](feast-dev@25e9775))
* Fix warnings from deprecated paths and update default log level ([feast-dev#3757](feast-dev#3757)) ([68a8737](feast-dev@68a8737))
* improve parsing bytewax job status ([5983f40](feast-dev@5983f40))
* make bytewax settings unexposed ([ae1bb8b](feast-dev@ae1bb8b))
* Make generated temp table name escaped ([feast-dev#3797](feast-dev#3797)) ([175d796](feast-dev@175d796))
* Pin numpy version to avoid spammy deprecation messages ([774ed33](feast-dev@774ed33))
* Redundant feature materialization and premature incremental materialization timestamp updates ([feast-dev#3789](feast-dev#3789)) ([417b16b](feast-dev@417b16b)), closes [feast-dev#6](feast-dev#6) [feast-dev#7](feast-dev#7)
* Resolve hbase hotspot issue when materializing ([feast-dev#3790](feast-dev#3790)) ([7376db8](feast-dev@7376db8))
* Set keepalives_idle None by default ([feast-dev#3756](feast-dev#3756)) ([8717e9b](feast-dev@8717e9b))
* Set upper bound for bigquery client due to its breaking changes ([2151c39](feast-dev@2151c39))
* UI project cannot handle fallback routes ([feast-dev#3766](feast-dev#3766)) ([96ece0f](feast-dev@96ece0f))
* update dependencies versions due to conflicts ([5dc0b24](feast-dev@5dc0b24))
* Update jackson and remove unnecessary logging ([feast-dev#3809](feast-dev#3809)) ([018d0ea](feast-dev@018d0ea))
* upgrade the pyarrow to latest v14.0.1 for CVE-2023-47248. ([052182b](feast-dev@052182b))

### Features

* Add get online feature rpc to gprc server ([feast-dev#3815](feast-dev#3815)) ([01db8cc](feast-dev@01db8cc))
* Add materialize and materialize-incremental rest endpoints ([feast-dev#3761](feast-dev#3761)) ([fa600fe](feast-dev@fa600fe)), closes [feast-dev#3760](feast-dev#3760)
* add redis sentinel support ([3387a15](feast-dev@3387a15))
* add redis sentinel support ([4337c89](feast-dev@4337c89))
* add redis sentinel support format lint ([aad8718](feast-dev@aad8718))
* Add support for `table_create_disposition` in bigquery job for offline store ([feast-dev#3762](feast-dev#3762)) ([6a728fe](feast-dev@6a728fe))
* Add support for in_cluster config and additional labels for bytewax materialization ([feast-dev#3754](feast-dev#3754)) ([2192e65](feast-dev@2192e65))
* Apply cache to load proto registry for performance ([feast-dev#3702](feast-dev#3702)) ([709c709](feast-dev@709c709))
* Make bytewax job write as mini-batches ([feast-dev#3777](feast-dev#3777)) ([9b0e5ce](feast-dev@9b0e5ce))
* Optimize bytewax pod resource with zero-copy ([9cf9d96](feast-dev@9cf9d96))
* Support GCS filesystem for bytewax engine ([feast-dev#3774](feast-dev#3774)) ([fb6b807](feast-dev@fb6b807))

Signed-off-by: tokoko <[email protected]>
zseta pushed a commit to zseta/feast that referenced this pull request Feb 7, 2024
@sudohainguyen @zseta
feat: Support GCS filesystem for bytewax engine (feast-dev#3774)
c1513c1 
* fix: Support param timeout when persisting

Signed-off-by: Hai Nguyen <[email protected]>

* fix: fix java integration test

Signed-off-by: Hai Nguyen <[email protected]>

---------

Signed-off-by: Hai Nguyen <[email protected]>
Signed-off-by: Attila Toth <[email protected]>
zseta pushed a commit to zseta/feast that referenced this pull request Feb 7, 2024
@feast-ci-bot @zseta
chore(release): release 0.35.0
ef5ff9e 
# [0.35.0](feast-dev/feast@v0.34.0...v0.35.0) (2024-01-13)

### Bug Fixes

* Add async refresh to prevent synchronous refresh in main thread ([feast-dev#3812](feast-dev#3812)) ([9583ed6](feast-dev@9583ed6))
* Adopt connection pooling for HBase ([feast-dev#3793](feast-dev#3793)) ([b3852bf](feast-dev@b3852bf))
* Bytewax engine create configmap from object ([feast-dev#3821](feast-dev#3821)) ([25e9775](feast-dev@25e9775))
* Fix warnings from deprecated paths and update default log level ([feast-dev#3757](feast-dev#3757)) ([68a8737](feast-dev@68a8737))
* improve parsing bytewax job status ([5983f40](feast-dev@5983f40))
* make bytewax settings unexposed ([ae1bb8b](feast-dev@ae1bb8b))
* Make generated temp table name escaped ([feast-dev#3797](feast-dev#3797)) ([175d796](feast-dev@175d796))
* Pin numpy version to avoid spammy deprecation messages ([774ed33](feast-dev@774ed33))
* Redundant feature materialization and premature incremental materialization timestamp updates ([feast-dev#3789](feast-dev#3789)) ([417b16b](feast-dev@417b16b)), closes [feast-dev#6](feast-dev#6) [feast-dev#7](feast-dev#7)
* Resolve hbase hotspot issue when materializing ([feast-dev#3790](feast-dev#3790)) ([7376db8](feast-dev@7376db8))
* Set keepalives_idle None by default ([feast-dev#3756](feast-dev#3756)) ([8717e9b](feast-dev@8717e9b))
* Set upper bound for bigquery client due to its breaking changes ([2151c39](feast-dev@2151c39))
* UI project cannot handle fallback routes ([feast-dev#3766](feast-dev#3766)) ([96ece0f](feast-dev@96ece0f))
* update dependencies versions due to conflicts ([5dc0b24](feast-dev@5dc0b24))
* Update jackson and remove unnecessary logging ([feast-dev#3809](feast-dev#3809)) ([018d0ea](feast-dev@018d0ea))
* upgrade the pyarrow to latest v14.0.1 for CVE-2023-47248. ([052182b](feast-dev@052182b))

### Features

* Add get online feature rpc to gprc server ([feast-dev#3815](feast-dev#3815)) ([01db8cc](feast-dev@01db8cc))
* Add materialize and materialize-incremental rest endpoints ([feast-dev#3761](feast-dev#3761)) ([fa600fe](feast-dev@fa600fe)), closes [feast-dev#3760](feast-dev#3760)
* add redis sentinel support ([3387a15](feast-dev@3387a15))
* add redis sentinel support ([4337c89](feast-dev@4337c89))
* add redis sentinel support format lint ([aad8718](feast-dev@aad8718))
* Add support for `table_create_disposition` in bigquery job for offline store ([feast-dev#3762](feast-dev#3762)) ([6a728fe](feast-dev@6a728fe))
* Add support for in_cluster config and additional labels for bytewax materialization ([feast-dev#3754](feast-dev#3754)) ([2192e65](feast-dev@2192e65))
* Apply cache to load proto registry for performance ([feast-dev#3702](feast-dev#3702)) ([709c709](feast-dev@709c709))
* Make bytewax job write as mini-batches ([feast-dev#3777](feast-dev#3777)) ([9b0e5ce](feast-dev@9b0e5ce))
* Optimize bytewax pod resource with zero-copy ([9cf9d96](feast-dev@9cf9d96))
* Support GCS filesystem for bytewax engine ([feast-dev#3774](feast-dev#3774)) ([fb6b807](feast-dev@fb6b807))

Signed-off-by: Attila Toth <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@whoahbot whoahbot whoahbot approved these changes
+1 more reviewer
@mjurkus mjurkus mjurkus left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@sudohainguyen sudohainguyen

Labels

approved lgtm ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@sudohainguyen @whoahbot @mjurkus @achals
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.