GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||
GeezerVison is showing github.com — feat: Adjust ray offline store to support abfs(s) ADLS Azure Storage by jbauer12 · Pull Request #5911 · feast-dev/feast Conversation
|
| destination_path = storage.file_options.uri | ||
| if not destination_path.startswith(("s3://", "gs://", "hdfs://")): | ||
| if not destination_path.startswith( | ||
| ("s3://", "gs://", "hdfs://", "abfs://", "abfss://") |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
# Remote storage URI schemes supported by the Ray offline store
# S3: Amazon S3
# GCS: Google Cloud Storage
# HDFS: Hadoop Distributed File System
# Azure: Azure Storage Gen2
REMOTE_STORAGE_SCHEMES = ("s3://", "gs://", "hdfs://", "abfs://", "abfss://")
Can we define a constant for supported remote storage URI schemes at top and use it later at all three locations?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Also, let's update doc https://github.com/feast-dev/feast/blob/master/docs/reference/offline-stores/ray.md?plain=1#L30 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
a77ca9e to
58d8df3
Compare
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Devin Review found 1 new potential issue.
sdk/python/feast/infra/compute_engines/ray/job.py:208)The PR adds Azure storage support (abfs://, abfss://) to the Ray offline store by introducing a REMOTE_STORAGE_SCHEMES constant, but fails to update the RayDAGRetrievalJob.persist() method in job.py which still uses a hardcoded tuple ("s3://", "gs://", "hdfs://").
When users try to persist datasets to Azure storage using the Ray compute engine (via RayDAGRetrievalJob), the code will incorrectly:
os.path.exists(destination_path))os.makedirs(os.path.dirname(destination_path), exist_ok=True))This happens because Azure paths like abfss://[email protected]/path are not recognized as remote storage.
job.py:208 uses ("s3://", "gs://", "hdfs://") which doesn't include Azure schemesREMOTE_STORAGE_SCHEMES from ray.py which includes ("s3://", "gs://", "hdfs://", "abfs://", "abfss://")ray.py:70 defines:
REMOTE_STORAGE_SCHEMES = ("s3://", "gs://", "hdfs://", "abfs://", "abfss://")
But job.py:208 still uses:
if not destination_path.startswith(("s3://", "gs://", "hdfs://")):
Recommendation: Import and use REMOTE_STORAGE_SCHEMES from feast.infra.offline_stores.contrib.ray_offline_store.ray or define a shared constant in a common module that both files can import.
View issue and 6 additional flags in Devin Review.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
@jbauer12 I think this comment make sense, we need to update job.py as well |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Devin Review found 2 new potential issues.
sdk/python/feast/infra/compute_engines/ray/job.py:208)The PR adds Azure storage support (abfs://, abfss://) to the Ray offline store by introducing a REMOTE_STORAGE_SCHEMES constant, but fails to update the RayDAGRetrievalJob.persist() method in job.py which still uses a hardcoded tuple ("s3://", "gs://", "hdfs://").
When users try to persist datasets to Azure storage using the Ray compute engine (via RayDAGRetrievalJob), the code will incorrectly:
os.path.exists(destination_path))os.makedirs(os.path.dirname(destination_path), exist_ok=True))This happens because Azure paths like abfss://[email protected]/path are not recognized as remote storage.
job.py:208 uses ("s3://", "gs://", "hdfs://") which doesn't include Azure schemesREMOTE_STORAGE_SCHEMES from ray.py which includes ("s3://", "gs://", "hdfs://", "abfs://", "abfss://")ray.py:70 defines:
REMOTE_STORAGE_SCHEMES = ("s3://", "gs://", "hdfs://", "abfs://", "abfss://")
But job.py:208 still uses:
if not destination_path.startswith(("s3://", "gs://", "hdfs://")):
Recommendation: Import and use REMOTE_STORAGE_SCHEMES from feast.infra.offline_stores.contrib.ray_offline_store.ray or define a shared constant in a common module that both files can import.
View issues and 8 additional flags in Devin Review.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
bb4d75a to
698bd1a
Compare
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
177d70d to
764be5a
Compare
Signed-off-by: Jonas Bauer <[email protected]>
Signed-off-by: Jonas Bauer <[email protected]>
Signed-off-by: Jonas Bauer <[email protected]>
Signed-off-by: Jonas Bauer <[email protected]>
Signed-off-by: Jonas Bauer <[email protected]>
Signed-off-by: Jonas Bauer <[email protected]>
Signed-off-by: Jonas Bauer <[email protected]>
Signed-off-by: Jonas Bauer <[email protected]>
764be5a to
74967ba
Compare
There was an error while loading. Please reload this page.
|
Thank you @jbauer12 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…east-dev#5911) * feat: adjust ray offline store to support abfs(s) ADLS Azure Storage Signed-off-by: Jonas Bauer <[email protected]> * Delete test_registry file. Signed-off-by: Jonas Bauer <[email protected]> * Feedback from PR Review. Signed-off-by: Jonas Bauer <[email protected]> * Added additional comment about env var. Signed-off-by: Jonas Bauer <[email protected]> * Fixed potential issue from PR. Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> * Apply formatting Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> --------- Signed-off-by: Jonas Bauer <[email protected]> Co-authored-by: Jonas Bauer <[email protected]> Signed-off-by: yassinnouh21 <[email protected]>
…east-dev#5911) * feat: adjust ray offline store to support abfs(s) ADLS Azure Storage Signed-off-by: Jonas Bauer <[email protected]> * Delete test_registry file. Signed-off-by: Jonas Bauer <[email protected]> * Feedback from PR Review. Signed-off-by: Jonas Bauer <[email protected]> * Added additional comment about env var. Signed-off-by: Jonas Bauer <[email protected]> * Fixed potential issue from PR. Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> * Apply formatting Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> --------- Signed-off-by: Jonas Bauer <[email protected]> Co-authored-by: Jonas Bauer <[email protected]> Signed-off-by: yassinnouh21 <[email protected]>
# [0.60.0](v0.59.0...v0.60.0) (2026-02-17) ### Bug Fixes * Added a flag to correctly download the go binaries ([0f77135](0f77135)) * Adds mapping of date Trino's type into string Feast's type ([531e839](531e839)) * **ci:** Use uv run for pytest in master_only benchmark step ([#5957](#5957)) ([5096010](5096010)) * Disable materialized odfvs for historical retrieval ([#5880](#5880)) ([739d28a](739d28a)) * Fix linting and formatting issues ([#5907](#5907)) ([42ca14a](42ca14a)) * Make timestamp field handling compatible with Athena V3 ([#5936](#5936)) ([e2bad34](e2bad34)) * Support pgvector under non-default schema ([#5970](#5970)) ([c636cd4](c636cd4)) * unit tests not running on main branch ([#5909](#5909)) ([62fe664](62fe664)) * Update java dep which blocking release ([#5903](#5903)) ([a5b8186](a5b8186)) * Update the dockerfile with golang 1.24.12. ([#5918](#5918)) ([be1b522](be1b522)) * Use context.Background() in client constructors ([#5897](#5897)) ([984f93a](984f93a)) ### Features * Add blog post for PyTorch ecosystem announcement ([#5906](#5906)) ([d2eb629](d2eb629)) * Add blog post on Feast dbt integration ([#5915](#5915)) ([b3c8138](b3c8138)) * Add DynamoDB in-place list update support for array-based features ([#5916](#5916)) ([aa5973f](aa5973f)) * Add HTTP connection pooling for remote online store client ([#5895](#5895)) ([e022bf8](e022bf8)) * Add integration tests for dbt import ([#5899](#5899)) ([a444692](a444692)) * Add lazy initialization and feature service caching ([#5924](#5924)) ([b37b7d0](b37b7d0)) * Add multiple entity support to dbt integration ([#5901](#5901)) ([05a4fb5](05a4fb5)), closes [#5872](#5872) * Add PostgreSQL online store support for Go feature server ([#5963](#5963)) ([b8c6f3d](b8c6f3d)) * Add publish docker image of Go feature server. ([#5923](#5923)) ([759d8c6](759d8c6)) * Add Set as feature type ([#5888](#5888)) ([52458fc](52458fc)) * Added online server worker config support in operator ([#5926](#5926)) ([193c72a](193c72a)) * Added support for OpenLineage integration ([#5884](#5884)) ([df70d8d](df70d8d)) * Adjust ray offline store to support abfs(s) ADLS Azure Storage ([#5911](#5911)) ([d6c0b2d](d6c0b2d)) * Batch_engine config injection in feature_store.yaml through operator ([#5938](#5938)) ([455d56c](455d56c)) * Consolidate Python packaging - remove setup.py/setup.cfg, standardize on pyproject.toml and uv ([16696b8](16696b8)) * **go:** Add MySQL registry store support for Go feature server ([#5933](#5933)) ([19f9bb8](19f9bb8)) * Improve local dev experience with file-aware hooks and auto parallelization ([#5956](#5956)) ([839b79e](839b79e)) * Modernize precommit hooks and optimize test performance ([#5929](#5929)) ([ea7d4fa](ea7d4fa)) * Optimize container infrastructure for production ([#5881](#5881)) ([5ebdac8](5ebdac8)) * Optimize DynamoDB online store for improved latency ([#5889](#5889)) ([fcc8274](fcc8274))
# [0.60.0](feast-dev/feast@v0.59.0...v0.60.0) (2026-02-17) ### Bug Fixes * Added a flag to correctly download the go binaries ([0f77135](feast-dev@0f77135)) * Adds mapping of date Trino's type into string Feast's type ([531e839](feast-dev@531e839)) * **ci:** Use uv run for pytest in master_only benchmark step ([feast-dev#5957](feast-dev#5957)) ([5096010](feast-dev@5096010)) * Disable materialized odfvs for historical retrieval ([feast-dev#5880](feast-dev#5880)) ([739d28a](feast-dev@739d28a)) * Fix linting and formatting issues ([feast-dev#5907](feast-dev#5907)) ([42ca14a](feast-dev@42ca14a)) * Make timestamp field handling compatible with Athena V3 ([feast-dev#5936](feast-dev#5936)) ([e2bad34](feast-dev@e2bad34)) * Support pgvector under non-default schema ([feast-dev#5970](feast-dev#5970)) ([c636cd4](feast-dev@c636cd4)) * unit tests not running on main branch ([feast-dev#5909](feast-dev#5909)) ([62fe664](feast-dev@62fe664)) * Update java dep which blocking release ([feast-dev#5903](feast-dev#5903)) ([a5b8186](feast-dev@a5b8186)) * Update the dockerfile with golang 1.24.12. ([feast-dev#5918](feast-dev#5918)) ([be1b522](feast-dev@be1b522)) * Use context.Background() in client constructors ([feast-dev#5897](feast-dev#5897)) ([984f93a](feast-dev@984f93a)) ### Features * Add blog post for PyTorch ecosystem announcement ([feast-dev#5906](feast-dev#5906)) ([d2eb629](feast-dev@d2eb629)) * Add blog post on Feast dbt integration ([feast-dev#5915](feast-dev#5915)) ([b3c8138](feast-dev@b3c8138)) * Add DynamoDB in-place list update support for array-based features ([feast-dev#5916](feast-dev#5916)) ([aa5973f](feast-dev@aa5973f)) * Add HTTP connection pooling for remote online store client ([feast-dev#5895](feast-dev#5895)) ([e022bf8](feast-dev@e022bf8)) * Add integration tests for dbt import ([feast-dev#5899](feast-dev#5899)) ([a444692](feast-dev@a444692)) * Add lazy initialization and feature service caching ([feast-dev#5924](feast-dev#5924)) ([b37b7d0](feast-dev@b37b7d0)) * Add multiple entity support to dbt integration ([feast-dev#5901](feast-dev#5901)) ([05a4fb5](feast-dev@05a4fb5)), closes [feast-dev#5872](feast-dev#5872) * Add PostgreSQL online store support for Go feature server ([feast-dev#5963](feast-dev#5963)) ([b8c6f3d](feast-dev@b8c6f3d)) * Add publish docker image of Go feature server. ([feast-dev#5923](feast-dev#5923)) ([759d8c6](feast-dev@759d8c6)) * Add Set as feature type ([feast-dev#5888](feast-dev#5888)) ([52458fc](feast-dev@52458fc)) * Added online server worker config support in operator ([feast-dev#5926](feast-dev#5926)) ([193c72a](feast-dev@193c72a)) * Added support for OpenLineage integration ([feast-dev#5884](feast-dev#5884)) ([df70d8d](feast-dev@df70d8d)) * Adjust ray offline store to support abfs(s) ADLS Azure Storage ([feast-dev#5911](feast-dev#5911)) ([d6c0b2d](feast-dev@d6c0b2d)) * Batch_engine config injection in feature_store.yaml through operator ([feast-dev#5938](feast-dev#5938)) ([455d56c](feast-dev@455d56c)) * Consolidate Python packaging - remove setup.py/setup.cfg, standardize on pyproject.toml and uv ([16696b8](feast-dev@16696b8)) * **go:** Add MySQL registry store support for Go feature server ([feast-dev#5933](feast-dev#5933)) ([19f9bb8](feast-dev@19f9bb8)) * Improve local dev experience with file-aware hooks and auto parallelization ([feast-dev#5956](feast-dev#5956)) ([839b79e](feast-dev@839b79e)) * Modernize precommit hooks and optimize test performance ([feast-dev#5929](feast-dev#5929)) ([ea7d4fa](feast-dev@ea7d4fa)) * Optimize container infrastructure for production ([feast-dev#5881](feast-dev#5881)) ([5ebdac8](feast-dev@5ebdac8)) * Optimize DynamoDB online store for improved latency ([feast-dev#5889](feast-dev#5889)) ([fcc8274](feast-dev@fcc8274)) Signed-off-by: soojin <[email protected]>
…east-dev#5911) * feat: adjust ray offline store to support abfs(s) ADLS Azure Storage Signed-off-by: Jonas Bauer <[email protected]> * Delete test_registry file. Signed-off-by: Jonas Bauer <[email protected]> * Feedback from PR Review. Signed-off-by: Jonas Bauer <[email protected]> * Added additional comment about env var. Signed-off-by: Jonas Bauer <[email protected]> * Fixed potential issue from PR. Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> * Apply formatting Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> --------- Signed-off-by: Jonas Bauer <[email protected]> Co-authored-by: Jonas Bauer <[email protected]>
…east-dev#5911) * feat: adjust ray offline store to support abfs(s) ADLS Azure Storage Signed-off-by: Jonas Bauer <[email protected]> * Delete test_registry file. Signed-off-by: Jonas Bauer <[email protected]> * Feedback from PR Review. Signed-off-by: Jonas Bauer <[email protected]> * Added additional comment about env var. Signed-off-by: Jonas Bauer <[email protected]> * Fixed potential issue from PR. Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> * Apply formatting Signed-off-by: Jonas Bauer <[email protected]> * Fixed found issue by Devin from PR. Signed-off-by: Jonas Bauer <[email protected]> --------- Signed-off-by: Jonas Bauer <[email protected]> Co-authored-by: Jonas Bauer <[email protected]> Signed-off-by: Shizoqua <[email protected]>
# [0.60.0](feast-dev/feast@v0.59.0...v0.60.0) (2026-02-17) ### Bug Fixes * Added a flag to correctly download the go binaries ([0f77135](feast-dev@0f77135)) * Adds mapping of date Trino's type into string Feast's type ([531e839](feast-dev@531e839)) * **ci:** Use uv run for pytest in master_only benchmark step ([feast-dev#5957](feast-dev#5957)) ([5096010](feast-dev@5096010)) * Disable materialized odfvs for historical retrieval ([feast-dev#5880](feast-dev#5880)) ([739d28a](feast-dev@739d28a)) * Fix linting and formatting issues ([feast-dev#5907](feast-dev#5907)) ([42ca14a](feast-dev@42ca14a)) * Make timestamp field handling compatible with Athena V3 ([feast-dev#5936](feast-dev#5936)) ([e2bad34](feast-dev@e2bad34)) * Support pgvector under non-default schema ([feast-dev#5970](feast-dev#5970)) ([c636cd4](feast-dev@c636cd4)) * unit tests not running on main branch ([feast-dev#5909](feast-dev#5909)) ([62fe664](feast-dev@62fe664)) * Update java dep which blocking release ([feast-dev#5903](feast-dev#5903)) ([a5b8186](feast-dev@a5b8186)) * Update the dockerfile with golang 1.24.12. ([feast-dev#5918](feast-dev#5918)) ([be1b522](feast-dev@be1b522)) * Use context.Background() in client constructors ([feast-dev#5897](feast-dev#5897)) ([984f93a](feast-dev@984f93a)) ### Features * Add blog post for PyTorch ecosystem announcement ([feast-dev#5906](feast-dev#5906)) ([d2eb629](feast-dev@d2eb629)) * Add blog post on Feast dbt integration ([feast-dev#5915](feast-dev#5915)) ([b3c8138](feast-dev@b3c8138)) * Add DynamoDB in-place list update support for array-based features ([feast-dev#5916](feast-dev#5916)) ([aa5973f](feast-dev@aa5973f)) * Add HTTP connection pooling for remote online store client ([feast-dev#5895](feast-dev#5895)) ([e022bf8](feast-dev@e022bf8)) * Add integration tests for dbt import ([feast-dev#5899](feast-dev#5899)) ([a444692](feast-dev@a444692)) * Add lazy initialization and feature service caching ([feast-dev#5924](feast-dev#5924)) ([b37b7d0](feast-dev@b37b7d0)) * Add multiple entity support to dbt integration ([feast-dev#5901](feast-dev#5901)) ([05a4fb5](feast-dev@05a4fb5)), closes [feast-dev#5872](feast-dev#5872) * Add PostgreSQL online store support for Go feature server ([feast-dev#5963](feast-dev#5963)) ([b8c6f3d](feast-dev@b8c6f3d)) * Add publish docker image of Go feature server. ([feast-dev#5923](feast-dev#5923)) ([759d8c6](feast-dev@759d8c6)) * Add Set as feature type ([feast-dev#5888](feast-dev#5888)) ([52458fc](feast-dev@52458fc)) * Added online server worker config support in operator ([feast-dev#5926](feast-dev#5926)) ([193c72a](feast-dev@193c72a)) * Added support for OpenLineage integration ([feast-dev#5884](feast-dev#5884)) ([df70d8d](feast-dev@df70d8d)) * Adjust ray offline store to support abfs(s) ADLS Azure Storage ([feast-dev#5911](feast-dev#5911)) ([d6c0b2d](feast-dev@d6c0b2d)) * Batch_engine config injection in feature_store.yaml through operator ([feast-dev#5938](feast-dev#5938)) ([455d56c](feast-dev@455d56c)) * Consolidate Python packaging - remove setup.py/setup.cfg, standardize on pyproject.toml and uv ([16696b8](feast-dev@16696b8)) * **go:** Add MySQL registry store support for Go feature server ([feast-dev#5933](feast-dev#5933)) ([19f9bb8](feast-dev@19f9bb8)) * Improve local dev experience with file-aware hooks and auto parallelization ([feast-dev#5956](feast-dev#5956)) ([839b79e](feast-dev@839b79e)) * Modernize precommit hooks and optimize test performance ([feast-dev#5929](feast-dev#5929)) ([ea7d4fa](feast-dev@ea7d4fa)) * Optimize container infrastructure for production ([feast-dev#5881](feast-dev#5881)) ([5ebdac8](feast-dev@5ebdac8)) * Optimize DynamoDB online store for improved latency ([feast-dev#5889](feast-dev#5889)) ([fcc8274](feast-dev@fcc8274)) Signed-off-by: Shizoqua <[email protected]>
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.