Description

danbaron63

opened on Sep 2, 2025

Issue body actions

Expected Behavior

Spark engine when enabled should materialize features to the online store.

Current Behavior

Materialization fails with the following error:

pyspark.errors.exceptions.base.PySparkTypeError: [CANNOT_INFER_SCHEMA_FOR_TYPE] Can not infer schema for type: `ChunkedArray`.

Steps to reproduce

See this repo that should reproduce the error locally for you. Ensure your JAVA_HOME is pointing to a java 17 installation and run make run to run the workflow. Optionally inspect the Makefile and run the commands yourself.

The repo has the default feast init repo created and will run a feast plan, feast apply and finally a materialize.py script which should trigger a Spark materialization job.

Specifications

• Version: 0.53.0
• Platform: Python 3.12, PySpark 3.5.5, Java 17
• Subsystem: MacOS 15.6.1

Possible Solution

I believe this happens when an arrow dataframe is converted to a Spark dataframe. Why this happens I am unsure, it may be my configuration or it may be a bug. If someone could advise on a solution that would be great.

Thanks!

Reacted by Dan Baron

Activity

1. danbaron63

   added

   kind/bug

   priority/p2

   on Sep 2, 2025
2. danbaron63 commented on Sep 3, 2025

   

   danbaron63

   on Sep 3, 2025

   ContributorAuthor

   More actions

   Note: I've tested this with pyspark==4.0.0 and it works, I believe this is due to changes in the way Spark now natively uses arrow. Unfortunately we're still blocked though as Spark 4 isn't widely available on most data compute platforms (e.g. AWS EMR) yet.

   I believe converting the pyarrow table to a pandas DataFrame as an intermediary should resolve this which shouldn't be too memory intensive if spark.sql.execution.arrow.enabled is set to true

   Reacted by Jacob Weinhold
3. haoxu0 commented on Sep 3, 2025

   

   haoxu0

   on Sep 3, 2025

   Collaborator

   More actions

   That make sense. Feel free to submit a pr
4. jfw-ppi commented on Dec 29, 2025

   

   jfw-ppi

   on Dec 29, 2025

   Collaborator

   More actions

   I will look into this as it blocks my solution for #5671
5. jfw-ppi

   added 2 commits that reference this issue on Jan 1, 2026

   fix: build Spark DataFrame from Arrow with schema and empty handling(f…

   9de90e5

   fix: build Spark DataFrame from Arrow with schema and empty handling(f…

   c0201ad
6. jfw-ppi commented on Jan 1, 2026

   

   jfw-ppi

   on Jan 1, 2026

   Collaborator

   More actions

   The reproduction repository passes now, using your suggested fix @danbaron63
7. jfw-ppi

   added 3 commits that reference this issue on Jan 1, 2026

   fix: build Spark DataFrame from Arrow with schema and empty handling(f…

   cf9b66e

   fix: build Spark DataFrame from Arrow with schema and empty handling(f…

   f507b7c

   fix: build Spark DataFrame from Arrow with schema and empty handling(f…

   b77a740
8. ntkathole

   added 3 commits that reference this issue on Jan 2, 2026

   fix: build Spark DataFrame from Arrow with schema and empty handling(f…

   611ead0

   fix: build Spark DataFrame from Arrow with schema and empty handling(f…

   237a65d

   fix: Spark Materialization Engine Cannot Infer Schema (#5806)

   58d0325
9. franciscojavierarceo

   added a commit that references this issue on Jan 16, 2026

   chore(release): release 0.59.0

   6214d05
10. kyledepasquale

    added a commit that references this issue on Jan 21, 2026

    chore(release): release 0.59.0

    b52f528
11. YassinNouh21

    added 2 commits that reference this issue on Feb 7, 2026

    chore(release): release 0.59.0

    61dff40

    chore(release): release 0.59.0

    ac546ff
12. jyejare

    added 2 commits that reference this issue on Mar 9, 2026

    fix: Spark Materialization Engine Cannot Infer Schema (feast-dev#5806)

    6ab1b78

    chore(release): release 0.59.0

    dedd393
13. ntkathole

    closed this as completedon Apr 26, 2026

Issue actions