GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — fix: Handle array of strings columns in Athena materialization by alan-gauthier-jt · Pull Request #6324 · feast-dev/feast

Conversation

alan-gauthier-jt commented Apr 24, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

What this PR does / why we need it

Fixes two related bugs that cause TypeError and ValueError when materializing
feature views with array-typed columns (e.g. Array(String), Array(Int64)) using
the Athena offline store.

Arrow/Athena deserializes array columns as numpy.ndarray (object dtype) instead of
plain Python lists. This breaks two code paths in type_map.py:

_convert_scalar_values_to_proto: pd.isnull(ndarray) returns an array of bools,
and not <array> raises ValueError: The truth value of an empty array is ambiguous.
→ Already guarded by _is_array_like in newer Feast versions; no change needed here.
_convert_list_values_to_proto (generic list path): proto_type(val=ndarray) passes
the raw numpy array to the protobuf constructor, which only accepts Python lists →
TypeError: bad argument type for built-in operation. Additionally, Arrow nullable
columns can yield None elements inside the ndarray, which protobuf repeated fields
also reject.
_validate_collection_item_types: None elements inside an ndarray failed the
type(item) in valid_types check before reaching the sanitization step.

Changes

`feast/type_map.py`

Add module-level _LIST_NONE_DEFAULTS dict mapping each list ValueType to a
type-appropriate zero/empty default value used to replace None elements:
- STRING_LIST, UUID_LIST, TIME_UUID_LIST, DECIMAL_LIST → ""
- BYTES_LIST → b""
- INT32_LIST, INT64_LIST → 0
- FLOAT_LIST, DOUBLE_LIST → 0.0
- BOOL_LIST → False
- UNIX_TIMESTAMP_LIST → NULL_TIMESTAMP_INT_VALUE
Add module-level _sanitize_list_value(value, feast_value_type) helper that:
- Calls .tolist() on any numpy.ndarray to produce a plain Python list
  (empty ndarray → None, treated as a missing row)
- Replaces None elements with the type-appropriate default from _LIST_NONE_DEFAULTS
- Is a no-op for plain Python lists without None and for scalar values
Apply sanitization upfront in _convert_list_values_to_proto: both values and
sample are normalised via _sanitize_list_value before any type-checking or proto
conversion, removing the need for per-path ndarray handling.
Remove the old _to_proto_safe_list / _DROP_NONE / _LIST_TYPE_NONE_REPLACEMENT
module-level helpers, which have been superseded by the above.
Skip None elements in _validate_collection_item_types — None entries are
valid in nullable Arrow columns and are sanitized upstream; raising a TypeError on
them before that point was incorrect.

Testing

Added TestArrowArrayStringListMaterialization in
sdk/python/tests/unit/test_type_map.py covering:

Test	Scenario
`test_sanitize_list_value_ndarray`	ndarray → plain list
`test_sanitize_list_value_empty_ndarray`	empty ndarray → `None` (missing row)
`test_sanitize_list_value_ndarray_with_none`	`None` elements in STRING_LIST replaced with `""`
`test_sanitize_list_value_plain_list`	plain list passthrough
`test_sanitize_list_value_plain_list_with_none`	`None` in plain STRING_LIST list replaced with `""`
`test_sanitize_list_value_numeric_none_replaced`	`None` in numeric/bool lists replaced with zero default
`test_sanitize_list_value_bytes_none_replaced`	`None` in BYTES_LIST replaced with `b""`
`test_sanitize_list_value_scalar_passthrough`	non-list, non-ndarray values unchanged
`test_string_list_from_ndarray`	full round-trip via `python_values_to_proto_values`
`test_string_list_from_empty_ndarray`	empty ndarray no longer raises `ValueError`
`test_string_list_from_ndarray_with_none_elements`	`None` in ndarray no longer raises `TypeError`
`test_string_list_null_row_produces_empty_proto`	`None` rows produce empty `ProtoValue`
`test_mixed_batch_simulating_athena_chunk`	full simulation of a failing Athena materialization batch

pytest sdk/python/tests/unit/test_type_map.py::TestArrowArrayStringListMaterialization -v

Which issues this PR fixes

Fixes #6325

Does this PR introduce a user-facing change?

Yes — materialization of array-typed feature columns from Athena no longer fails with
TypeError or ValueError when a batch contains empty arrays, None rows, or None
elements inside arrays. None elements inside an array are now stored as the
type-appropriate zero/empty value (e.g. "" for strings, 0 for integers).

Previously:
  TypeError: bad argument type for built-in operation
  ValueError: The truth value of an empty array is ambiguous

After this fix:
  Materialization completes successfully.
  None elements inside arrays are replaced with type-appropriate defaults.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

alan-gauthier-jt requested a review from a team as a code owner

April 24, 2026 08:17

alan-gauthier-jt mentioned this pull request

Apr 24, 2026

Bug: TypeError / ValueError when materializing Array(String) feature views with Athena offline store #6325

Closed

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

alan-gauthier-jt force-pushed the fix-empty-string-array branch from 86036b7 to c58ca8a Compare

April 24, 2026 08:29

alan-gauthier-jt changed the title ~~fix: handle numpy.ndarray Array(String) columns in Athena materialization~~ fix: handle numpyndarray Array(String) columns in Athena materialization

Apr 24, 2026

alan-gauthier-jt force-pushed the fix-empty-string-array branch from c58ca8a to cf2f0c4 Compare

April 24, 2026 08:53

alan-gauthier-jt changed the title ~~fix: handle numpyndarray Array(String) columns in Athena materialization~~ fix: Handle array of strings columns in Athena materialization

Apr 24, 2026

ntkathole mentioned this pull request

Apr 25, 2026

fix: Handle numpy ndarray in Array(String) materialization #6327

Closed

5 of 7 tasks

ntkathole reviewed

Apr 25, 2026

View reviewed changes

Comment thread

sdk/python/feast/type_map.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

ntkathole reviewed

Apr 25, 2026

View reviewed changes

Comment thread

sdk/python/feast/type_map.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

alan-gauthier-jt requested a review from ntkathole

May 7, 2026 09:57

alan-gauthier-jt force-pushed the fix-empty-string-array branch from d21d32c to ac81649 Compare

May 13, 2026 07:46

franciscojavierarceo approved these changes

May 13, 2026

View reviewed changes

alan-gauthier-jt force-pushed the fix-empty-string-array branch from ac81649 to 66173d7 Compare

May 19, 2026 07:03

ntkathole added the ok-to-test label

May 19, 2026

alan-gauthier-jt force-pushed the fix-empty-string-array branch 2 times, most recently from 828cd23 to 4030d4b Compare

June 2, 2026 11:52

ntkathole approved these changes

Jun 3, 2026

View reviewed changes

alan-gauthier-jt added 3 commits

June 3, 2026 14:31


          fix: handle string arrays in materialization

2a27d40

Signed-off-by: Alan Gauthier <[email protected]>


          pr feedback: none default values per type

Signed-off-by: Alan Gauthier <[email protected]>


          fix integration tests

066c9a2

Signed-off-by: Alan Gauthier <[email protected]>

ntkathole force-pushed the fix-empty-string-array branch from 4030d4b to 066c9a2 Compare

June 3, 2026 09:01

ntkathole merged commit 4ed0278 into feast-dev:master

Jun 3, 2026

20 of 27 checks passed

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo pushed a commit that referenced this pull request

Jun 13, 2026


          chore(release): release 0.64.0

9bb73cc

# [0.64.0](v0.63.0...v0.64.0) (2026-06-13)

### Bug Fixes

* Add async_supported property to RedisOnlineStore ([9b088fe](9b088fe))
* Add missing feast init templates to operator CRD and enhance persistence documentation ([1941d4d](1941d4d))
* Allow to publish from reference branch ([5458ec8](5458ec8))
* API calls list ([4203eb7](4203eb7))
* **bigquery:** Enable list inference for parquet loads in offline_write_batch ([9243497](9243497)), closes [#5845](#5845)
* Bump grpcio dependencies ([07b4782](07b4782))
* **compute-engine/local:** Honor field_mapping on join keys in dedup + join nodes ([#6395](#6395)) ([bd01824](bd01824))
* **dynamodb:** Avoid tag race condition by using diff-based tag updates ([#6479](#6479)) ([bad2b7d](bad2b7d)), closes [#6418](#6418)
* **dynamodb:** Fix mypy type for _build_projection_expression return ([217b4da](217b4da))
* Fix intermittent async test failures for DynamoDB and Redis ([63c5eb1](63c5eb1))
* Fix mongodb blog title ([57d28d4](57d28d4))
* Fix shared SQL registry crash - avoid unnecessary UDF deserialization in proto cache building ([ac588d7](ac588d7))
* Fix SparkRetrievalJob.persist() failing for SparkSource ([209d7cd](209d7cd))
* Fixed formatting and image for mongo blog ([#6377](#6377)) ([f8389fb](f8389fb))
* Fixes for ray source ([7f592a4](7f592a4))
* **go:** skip registry refresh when cache_ttl_seconds <= 0 ([97ed40c](97ed40c))
* Handle array of strings columns in Athena materialization ([#6324](#6324)) ([4ed0278](4ed0278))
* make milvus VARCHAR max_length configurable, remove hardcoded 512 limit ([3b98c22](3b98c22))
* **operator:** Set appProtocol: grpc on registry gRPC Service ([#6367](#6367)) ([c9ae2b4](c9ae2b4))
* PyJWT 2.10+ added validation that rejects empty HMAC keys ([e756ffe](e756ffe))
* RemoteOnlineStore sends all features in a single HTTP request ([8f187dd](8f187dd))
* Remove registry proto dump to enforce RBAC and add permission checks to Commit/Refresh RPCs ([328431f](328431f))
* Remove selector migration job - no longer needed ([51c325e](51c325e))
* replace broken .claude skill symlink with correct relative path ([4541690](4541690))
* Replace selector label strip patch with migration Job for upgrade-safe selector uniqueness ([00dea50](00dea50))
* Scope feature view name conflict check to current project in file-based registry ([#6369](#6369)) ([a4fde83](a4fde83)), closes [#6209](#6209)
* **snowflake:** Stop double-quoting connection identifiers ([#6462](#6462)) ([e914d59](e914d59))
* **spark:** S3/GCS PyArrow filesystem resolution for staging paths ([#6442](#6442)) ([ae50414](ae50414))
* **trino:** Clean up temporary entity tables after retrieval ([#6381](#6381)) ([d86b13d](d86b13d)), closes [#6306](#6306)
* Update go-feature-server base image to Go 1.25 and fix operator Dockerfile COPY permissions ([86ef0bc](86ef0bc))

### Features

* [Backend] Data Quality Monitoring with native compute, multi-backend support, REST API, CLI ([#6202](#6202)) ([5458c37](5458c37))
* Add apache flink compute engine ([#6476](#6476)) ([9636d6a](9636d6a))
* Add demo noteboooks for users ([e362173](e362173))
* Add enabled/disabled toggle for feature views ([#6401](#6401)) ([5f1fa0d](5f1fa0d)), closes [#6395](#6395)
* Add Label View to init template ([ec272d5](ec272d5))
* Add mTLS support to remote registry gRPC client ([#6474](#6474)) ([c9602d8](c9602d8))
* Add Prometheus gauges for FeatureStore installation telemetry ([#6354](#6354)) ([1b681b7](1b681b7))
* Adds registry REST API endpoints for managing entities, data sources, and feature views ([#6413](#6413)) ([f77bd1d](f77bd1d))
* Allow CRUD on entities, data sources, and feature views from UI ([#6412](#6412)) ([2321c07](2321c07))
* Allow default openlineage configuration ([#6467](#6467)) ([276b6df](276b6df))
* **bigquery:** Support DATE-type event timestamp columns ([#6362](#6362)) ([753dee5](753dee5)), closes [#2530](#2530)
* **cli:** Add `feast projects delete` command (closes [#5095](#5095)) ([#6318](#6318)) ([1a4b96c](1a4b96c))
* Data Quality Monitoring added in feast UI ([#6422](#6422)) ([fa271be](fa271be))
* **dynamodb:** Use ProjectionExpression when requested_features is set ([0adc906](0adc906)), closes [#6058](#6058)
* Enhance DataSource and FeatureView modals with error handling and submission states ([96d7169](96d7169))
* Expose registry endpoints on feature server for MCP access ([f77981c](f77981c))
* Feast First-Class LabelView Implementation ([#6292](#6292)) ([c0e7e5d](c0e7e5d))
* Feast-MLflow Integration ([#6235](#6235)) ([7279c75](7279c75))
* Operational metrics for offline store and SOX metrics for both ([#6340](#6340)) ([65b1b80](65b1b80))
* Pre-compute feature service ([8011550](8011550))
* REST API-backed UI for RBAC compatibility and per-page lazy loading ([#6414](#6414)) ([6ae80af](6ae80af))
* Support non-string map key types ([#6382](#6382)) ([#6383](#6383)) ([728aa2e](728aa2e))
* Update FeatureStore CRD with DRA Fields ([01241e4](01241e4))

### Performance Improvements

* Cache feature view resolution in get_online_features to reduce per-request overhead ([55c2f18](55c2f18))
* Optimize feature serving latency with batched async Redis, cached checks fix ([103809a](103809a))
* Replace MessageToDict with optimized custom dict builder ([#6015](#6015)) ([9902064](9902064))

Srihari1192 pushed a commit to Srihari1192/feast that referenced this pull request

Jun 22, 2026


          chore(release): release 0.64.0

22c3721

# [0.64.0](feast-dev/feast@v0.63.0...v0.64.0) (2026-06-13)

### Bug Fixes

* Add async_supported property to RedisOnlineStore ([9b088fe](feast-dev@9b088fe))
* Add missing feast init templates to operator CRD and enhance persistence documentation ([1941d4d](feast-dev@1941d4d))
* Allow to publish from reference branch ([5458ec8](feast-dev@5458ec8))
* API calls list ([4203eb7](feast-dev@4203eb7))
* **bigquery:** Enable list inference for parquet loads in offline_write_batch ([9243497](feast-dev@9243497)), closes [feast-dev#5845](feast-dev#5845)
* Bump grpcio dependencies ([07b4782](feast-dev@07b4782))
* **compute-engine/local:** Honor field_mapping on join keys in dedup + join nodes ([feast-dev#6395](feast-dev#6395)) ([bd01824](feast-dev@bd01824))
* **dynamodb:** Avoid tag race condition by using diff-based tag updates ([feast-dev#6479](feast-dev#6479)) ([bad2b7d](feast-dev@bad2b7d)), closes [feast-dev#6418](feast-dev#6418)
* **dynamodb:** Fix mypy type for _build_projection_expression return ([217b4da](feast-dev@217b4da))
* Fix intermittent async test failures for DynamoDB and Redis ([63c5eb1](feast-dev@63c5eb1))
* Fix mongodb blog title ([57d28d4](feast-dev@57d28d4))
* Fix shared SQL registry crash - avoid unnecessary UDF deserialization in proto cache building ([ac588d7](feast-dev@ac588d7))
* Fix SparkRetrievalJob.persist() failing for SparkSource ([209d7cd](feast-dev@209d7cd))
* Fixed formatting and image for mongo blog ([feast-dev#6377](feast-dev#6377)) ([f8389fb](feast-dev@f8389fb))
* Fixes for ray source ([7f592a4](feast-dev@7f592a4))
* **go:** skip registry refresh when cache_ttl_seconds <= 0 ([97ed40c](feast-dev@97ed40c))
* Handle array of strings columns in Athena materialization ([feast-dev#6324](feast-dev#6324)) ([4ed0278](feast-dev@4ed0278))
* make milvus VARCHAR max_length configurable, remove hardcoded 512 limit ([3b98c22](feast-dev@3b98c22))
* **operator:** Set appProtocol: grpc on registry gRPC Service ([feast-dev#6367](feast-dev#6367)) ([c9ae2b4](feast-dev@c9ae2b4))
* PyJWT 2.10+ added validation that rejects empty HMAC keys ([e756ffe](feast-dev@e756ffe))
* RemoteOnlineStore sends all features in a single HTTP request ([8f187dd](feast-dev@8f187dd))
* Remove registry proto dump to enforce RBAC and add permission checks to Commit/Refresh RPCs ([328431f](feast-dev@328431f))
* Remove selector migration job - no longer needed ([51c325e](feast-dev@51c325e))
* replace broken .claude skill symlink with correct relative path ([4541690](feast-dev@4541690))
* Replace selector label strip patch with migration Job for upgrade-safe selector uniqueness ([00dea50](feast-dev@00dea50))
* Scope feature view name conflict check to current project in file-based registry ([feast-dev#6369](feast-dev#6369)) ([a4fde83](feast-dev@a4fde83)), closes [feast-dev#6209](feast-dev#6209)
* **snowflake:** Stop double-quoting connection identifiers ([feast-dev#6462](feast-dev#6462)) ([e914d59](feast-dev@e914d59))
* **spark:** S3/GCS PyArrow filesystem resolution for staging paths ([feast-dev#6442](feast-dev#6442)) ([ae50414](feast-dev@ae50414))
* **trino:** Clean up temporary entity tables after retrieval ([feast-dev#6381](feast-dev#6381)) ([d86b13d](feast-dev@d86b13d)), closes [feast-dev#6306](feast-dev#6306)
* Update go-feature-server base image to Go 1.25 and fix operator Dockerfile COPY permissions ([86ef0bc](feast-dev@86ef0bc))

### Features

* [Backend] Data Quality Monitoring with native compute, multi-backend support, REST API, CLI ([feast-dev#6202](feast-dev#6202)) ([5458c37](feast-dev@5458c37))
* Add apache flink compute engine ([feast-dev#6476](feast-dev#6476)) ([9636d6a](feast-dev@9636d6a))
* Add demo noteboooks for users ([e362173](feast-dev@e362173))
* Add enabled/disabled toggle for feature views ([feast-dev#6401](feast-dev#6401)) ([5f1fa0d](feast-dev@5f1fa0d)), closes [feast-dev#6395](feast-dev#6395)
* Add Label View to init template ([ec272d5](feast-dev@ec272d5))
* Add mTLS support to remote registry gRPC client ([feast-dev#6474](feast-dev#6474)) ([c9602d8](feast-dev@c9602d8))
* Add Prometheus gauges for FeatureStore installation telemetry ([feast-dev#6354](feast-dev#6354)) ([1b681b7](feast-dev@1b681b7))
* Adds registry REST API endpoints for managing entities, data sources, and feature views ([feast-dev#6413](feast-dev#6413)) ([f77bd1d](feast-dev@f77bd1d))
* Allow CRUD on entities, data sources, and feature views from UI ([feast-dev#6412](feast-dev#6412)) ([2321c07](feast-dev@2321c07))
* Allow default openlineage configuration ([feast-dev#6467](feast-dev#6467)) ([276b6df](feast-dev@276b6df))
* **bigquery:** Support DATE-type event timestamp columns ([feast-dev#6362](feast-dev#6362)) ([753dee5](feast-dev@753dee5)), closes [feast-dev#2530](feast-dev#2530)
* **cli:** Add `feast projects delete` command (closes [feast-dev#5095](feast-dev#5095)) ([feast-dev#6318](feast-dev#6318)) ([1a4b96c](feast-dev@1a4b96c))
* Data Quality Monitoring added in feast UI ([feast-dev#6422](feast-dev#6422)) ([fa271be](feast-dev@fa271be))
* **dynamodb:** Use ProjectionExpression when requested_features is set ([0adc906](feast-dev@0adc906)), closes [feast-dev#6058](feast-dev#6058)
* Enhance DataSource and FeatureView modals with error handling and submission states ([96d7169](feast-dev@96d7169))
* Expose registry endpoints on feature server for MCP access ([f77981c](feast-dev@f77981c))
* Feast First-Class LabelView Implementation ([feast-dev#6292](feast-dev#6292)) ([c0e7e5d](feast-dev@c0e7e5d))
* Feast-MLflow Integration ([feast-dev#6235](feast-dev#6235)) ([7279c75](feast-dev@7279c75))
* Operational metrics for offline store and SOX metrics for both ([feast-dev#6340](feast-dev#6340)) ([65b1b80](feast-dev@65b1b80))
* Pre-compute feature service ([8011550](feast-dev@8011550))
* REST API-backed UI for RBAC compatibility and per-page lazy loading ([feast-dev#6414](feast-dev#6414)) ([6ae80af](feast-dev@6ae80af))
* Support non-string map key types ([feast-dev#6382](feast-dev#6382)) ([feast-dev#6383](feast-dev#6383)) ([728aa2e](feast-dev@728aa2e))
* Update FeatureStore CRD with DRA Fields ([01241e4](feast-dev@01241e4))

### Performance Improvements

* Cache feature view resolution in get_online_features to reduce per-request overhead ([55c2f18](feast-dev@55c2f18))
* Optimize feature serving latency with batched async Redis, cached checks fix ([103809a](feast-dev@103809a))
* Replace MessageToDict with optimized custom dict builder ([feast-dev#6015](feast-dev#6015)) ([9902064](feast-dev@9902064))

Reviewers

devin-ai-integration[bot]

devin-ai-integration[bot] left review comments

franciscojavierarceo

franciscojavierarceo approved these changes

ntkathole

ntkathole approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Bug: TypeError / ValueError when materializing Array(String) feature views with Athena offline store

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

alan-gauthier-jt commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it

Changes

feast/type_map.py

Testing

Which issues this PR fixes

Does this PR introduce a user-facing change?

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alan-gauthier-jt commented Apr 24, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

`feast/type_map.py`