GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Unable to apply parquet dataset from s3 · Issue #3216 · feast-dev/feast

Unable to apply parquet dataset from s3 #3216

New issue
Copy link
New issue
Copy link
Closed
#3217
Closed
Unable to apply parquet dataset from s3#3216
#3217
Copy link
Labels
kind/bugpriority/p2

Description

@mzwiessele
mzwiessele
opened on Sep 15, 2022
Issue body actions

Expected Behavior

We point our file source to a parquet dataset:

file_source = FileSource(
    name="dummy_file_source",
    path="s3://data/dummy/"),
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
    file_format=ParquetFormat(),
)

I'm expecting to be able to use the parquet dataset format the same way I'd use a single file.

Current Behavior

Feast errors at the apply stage:

File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/infra/offline_stores/file_source.py", line 164, in get_table_column_names_and_types
      filesystem.open_input_file(path), filesystem=filesystem
    File "pyarrow/_fs.pyx", line 588, in pyarrow._fs.FileSystem.open_input_file
    File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
    File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
  OSError: Path does not exist 'data/dummy'
Full Traceback 
Traceback (most recent call last):
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/cli.py", line 519, in apply_total_command
    apply_total(repo_config, repo, skip_source_validation)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/usage.py", line 283, in wrapper
    return func(*args, **kwargs)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/repo_operations.py", line 335, in apply_total
    apply_total_with_repo_instance(
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/repo_operations.py", line 296, in apply_total_with_repo_instance
    registry_diff, infra_diff, new_infra = store.plan(repo)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/usage.py", line 294, in wrapper
    raise exc.with_traceback(traceback)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/usage.py", line 283, in wrapper
    return func(*args, **kwargs)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/feature_store.py", line 723, in plan
    self._make_inferences(
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/feature_store.py", line 601, in _make_inferences
    update_feature_views_with_inferred_features_and_entities(
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/inference.py", line 179, in update_feature_views_with_inferred_features_and_entities
    _infer_features_and_entities(
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/inference.py", line 217, in _infer_features_and_entities
    table_column_names_and_types = fv.batch_source.get_table_column_names_and_types(
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
  File "/Users/mzwiessele/feast_s3_dataset_error/.venv/lib/python3.8/site-packages/feast/infra/offline_stores/file_source.py", line 164, in get_table_column_names_and_types
    filesystem.open_input_file(path), filesystem=filesystem
  File "pyarrow/_fs.pyx", line 588, in pyarrow._fs.FileSystem.open_input_file
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
OSError: Path does not exist 'data/dummy'

Steps to reproduce

  1. Create default example using feast init
  2. Upload example driver_stats.parquet data to s3 dataset path: s3://data/dummy/driver_stats.parquet
  3. Change the data source to point to the s3 dataset: 
    @@ -1,28 +1,28 @@
driver_stats_source = FileSource(
    name="driver_hourly_stats_source",
-   path="data/driver_stats.parquet",
+   path="s3://data/dummy/",
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
)
  4. Run feast apply

Specifications

  • Version: 0.24.0
  • Platform: MacOS
  • Subsystem: Python 3.8

Possible Solution

This PR fixes this issue: #3217

I have found this line to cause the error:

feast/sdk/python/feast/infra/offline_stores/file_source.py

Line 164 in 7bc1dff

filesystem.open_input_file(path), filesystem=filesystem

Activity

  1. mzwiessele
    added
    kind/bug
    priority/p2
    on Sep 15, 2022

  2. mzwiessele commented on Sep 16, 2022

    @mzwiessele
    mzwiessele
    on Sep 16, 2022
    ContributorAuthor
    More actions

    @adchia any chance you could take a look into this? It really should only take a couple of minutes to verify.

  3. feast-ci-bot
    closed this as completedin #3217on Sep 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugpriority/p2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions
      You are visitor 01402520 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.