GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Fix NumPy - PyArrow array type mapping in Trino offline store by ebolblga · Pull Request #5393 · feast-dev/feast

fix: Fix NumPy - PyArrow array type mapping in Trino offline store #5393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
franciscojavierarceo merged 3 commits into feast-dev:master from ebolblga:fix-trino-arrays
Jun 4, 2025

Uh oh!

There was an error while loading. Please reload this page.
Merged

fix: Fix NumPy - PyArrow array type mapping in Trino offline store #5393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6610a75
fix: remove forced schema when converting pandas to pyarrow table
ebolblga May 27, 2025
dd118d6
fix: add arrays to trino supported types in docs
ebolblga May 27, 2025
cd1f210
chore: linting
ebolblga May 27, 2025
File filter
Filter by extension
Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Diff view
Diff view

There are no files selected for viewing

2 changes: 1 addition & 1 deletion 2 docs/reference/data-sources/overview.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@ Below is a matrix indicating which data sources support which types.
| `float64` | yes | yes | yes | yes | yes | yes | yes | yes |
| `bool` | yes | yes | yes | yes | yes | yes | yes | yes |
| `timestamp` | yes | yes | yes | yes | yes | yes | yes | yes |
| array types | yes | yes | yes | no | yes | yes | no | no |
| array types | yes | yes | yes | no | yes | yes | yes | no |
4 changes: 2 additions & 2 deletions 4 docs/reference/data-sources/trino.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ These can be specified either by a table reference or a SQL query.
## Disclaimer

The Trino data source does not achieve full test coverage.
Please do not assume complete stability.
Please do not assume complete stability.

## Examples

Expand All @@ -30,5 +30,5 @@ The full set of configuration options is available [here](https://rtd.feast.dev/

## Supported Types

Trino data sources support all eight primitive types, but currently do not support array types.
Trino data sources support all eight primitive types and their corresponding array types.
For a comparison against other batch data sources, please see [here](overview.md#functionality-matrix).
4 changes: 1 addition & 3 deletions 4 sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/trino.py
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,7 @@ def _to_df_internal(self, timeout: Optional[int] = None) -> pd.DataFrame:

def _to_arrow_internal(self, timeout: Optional[int] = None) -> pyarrow.Table:
"""Return payrrow dataset as synchronously including on demand transforms"""
return pyarrow.Table.from_pandas(
self._to_df_internal(timeout=timeout), schema=self.pyarrow_schema
)
return pyarrow.Table.from_pandas(self._to_df_internal(timeout=timeout))
Comment thread

@franciscojavierarceo franciscojavierarceo May 27, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove the explicit schema declaration?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@ebolblga ebolblga May 28, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's say I have Pandas DataFrame like this:

feature_x feature_y
[1] None
[3, 4] [54, 38]
[75, 1, 12] [40, 0]

If I output column dtypes then feature_x would still be object (array(int)) and feature_y would be converted to float by underlying NumPy because of the Null.

By default, pandas uses NumPy data types, which do not support missing values in integer arrays. If you create a Series or DataFrame column with integers and include a null value (e.g., None or np.nan), pandas will upcast the column to a floating-point type (float64) to accommodate the missing value.

If I later pass this dataframe WITH forced schema, PyArrow will see that I want to cast float to array(int) and throw an error. If I don't pass the schema though, it will infer type itself and work as expected.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

def to_sql(self) -> str:
"""Returns the SQL query that will be executed in Trino to build the historical feature table"""
Expand Down
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.