GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Handle numpy.ndarray from Arrow/Athena in Array(String) materialisation by Abhishek8108 · Pull Request #6388 · feast-dev/feast

Conversation

@Abhishek8108

@Abhishek8108 Abhishek8108 commented May 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #6325

Materialising feature views with Array(String) columns via the Athena offline store fails with TypeError or ValueError. Arrow/Athena deserialises these columns as numpy.ndarray with object dtype instead of plain Python lists, and two code paths in type_map.py do not handle this.

Root causes

1. _validate_collection_item_types — TypeError on None elements inside ndarrays

Nullable Arrow columns embed None elements inside the ndarray. The validator iterates over the ndarray directly and checks type(item) in valid_types. For STRING_LIST, valid_types = [np.str_, str], and type(None) is not in that set, so it raises TypeError: bad argument type for built-in operation.

2. _convert_list_values_to_proto generic path — TypeError from protobuf

The generic list conversion calls StringList(val=value) where value is a numpy.ndarray. Protobuf rejects non-list inputs with TypeError: bad argument type for built-in operation.

Fix

  • In _validate_collection_item_types: coerce ndarray to a plain Python list via .tolist() before element-level checks, and skip None elements (nullable elements are valid in Arrow columns).
  • In the generic conversion path of _convert_list_values_to_proto: apply the same .tolist() coercion so protobuf always receives a plain Python list. None elements are stripped (protobuf fixed-type lists cannot hold nulls).

The fix covers all three failure modes from #6325:

  • np.array(['a', 'b', 'c']) → ['a', 'b', 'c'] ✓
  • np.array(['a', None, 'c']) → ['a', 'c'] (None stripped) ✓
  • np.array([]) → [] ✓

Changes

  • sdk/python/feast/type_map.py — two targeted edits, no API surface change
  • sdk/python/tests/unit/test_type_map.py — four new unit tests in TestAthenaArrayStringConversion

Test plan

  • TestAthenaArrayStringConversion::test_string_list_from_ndarray — plain ndarray converts correctly
  • TestAthenaArrayStringConversion::test_string_list_from_ndarray_with_none_elements — None elements stripped, no TypeError
  • TestAthenaArrayStringConversion::test_string_list_from_empty_ndarray — empty ndarray yields empty list, no ValueError
  • TestAthenaArrayStringConversion::test_string_list_mixed_null_and_ndarray_rows — mix of null rows and ndarray rows
  • ruff format --check and ruff check — clean on both modified files
  • Existing test_type_map.py tests unaffected

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Abhishek8108
fix: Handle numpy.ndarray from Arrow/Athena in Array(String) material…
1250453
…isation

Athena (and other Arrow-backed offline stores) deserialises Array(String)
feature columns as numpy.ndarray with object dtype rather than plain Python
lists.  Two code paths in type_map.py did not handle this:

1. _validate_collection_item_types iterated over the ndarray directly.
   Nullable Arrow columns can embed None elements, and type(None) is not in
   the valid_types set for STRING_LIST ([np.str_, str]), causing TypeError.

2. The generic list conversion path in _convert_list_values_to_proto passed
   the raw ndarray to StringList(val=...).  Protobuf rejects non-list inputs
   with TypeError: bad argument type for built-in operation.

Fix:
- Coerce ndarray to a plain Python list via .tolist() before type validation,
  and skip None elements (nullable elements cannot be held in protobuf
  fixed-type lists and are stripped).
- In the generic conversion path, apply the same coercion so protobuf always
  receives a plain list.

Adds four unit tests covering: plain ndarray, ndarray with None elements,
empty ndarray, and mixed None/ndarray rows.

Fixes feast-dev#6325

Signed-off-by: Abhishek8108 <[email protected]>
@Abhishek8108
Abhishek8108 requested a review from a team as a code owner May 9, 2026 09:08
@Abhishek8108 Abhishek8108 mentioned this pull request May 9, 2026
Proposal: Version-qualified feature refs in get_historical_features (offline store versioning) #6389
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Bug: TypeError / ValueError when materializing Array(String) feature views with Athena offline store

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@Abhishek8108
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.