GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — BigQuery offline store loses array data when pushing features with list types · Issue #5845 · feast-dev/feast

BigQuery offline store loses array data when pushing features with list types #5845

New issue
Copy link
New issue
Copy link
Closed
#6365
Closed
BigQuery offline store loses array data when pushing features with list types#5845
#6365
Copy link
Labels
kind/bugpriority/p2

Description

@max36067
max36067
opened on Jan 13, 2026
Issue body actions

Summary

When using store.push() with PushMode.OFFLINE or PushMode.ONLINE_AND_OFFLINE, array/list type columns (e.g., STRING_LIST) are written as empty arrays [] to BigQuery, even though the data is correct in the DataFrame and PyArrow table.

Root Cause

The BigQuery LoadJobConfig in offline_write_batch() is missing parquet_options.enable_list_inference = True. Without this option, BigQuery's parquet loader doesn't correctly interpret PyArrow's list format.

Related issue: googleapis/google-cloud-python#15705 (comment)

Steps to Reproduce

  from feast import FeatureStore
  from feast.data_source import PushMode
  import pandas as pd
  from datetime import datetime, timezone

  # Assuming feature view with STRING_LIST field is configured
  data = {
      "entity_id": "test_123",
      "tags": ["category_a", "category_b"],  # STRING_LIST type
      "event_time": datetime.now(timezone.utc),
  }

  df = pd.DataFrame([data])
  store = FeatureStore(repo_path=".")
  store.push("my_push_source", df, to=PushMode.ONLINE_AND_OFFLINE)

  # Result in BigQuery: tags = [] (empty array)
  # Expected: tags = ["category_a", "category_b"]

Expected Behavior

Array data should be correctly written to BigQuery with values preserved.

Actual Behavior

Array columns are written as empty arrays [] in BigQuery, while the online store receives correct data.

Proposed Fix

In feast/infra/offline_stores/bigquery.py, update offline_write_batch() (~line 428):

  @staticmethod
  def offline_write_batch(
      config: RepoConfig,
      feature_view: FeatureView,
      table: pyarrow.Table,
      progress: Optional[Callable[[int], Any]],
  ):
      # ... existing code ...

      parquet_options = bigquery.ParquetOptions()
      parquet_options.enable_list_inference = True

      job_config = bigquery.LoadJobConfig(
          source_format=bigquery.SourceFormat.PARQUET,
          schema=arrow_schema_to_bq_schema(pa_schema),
          create_disposition=config.offline_store.table_create_disposition,
          write_disposition="WRITE_APPEND",
          parquet_options=parquet_options,  # Add this line
      )
      # ... rest of code ...

Environment

  • Feast version: 0.58.0
  • Python version: 3.12
  • BigQuery client version: (latest)

Additional Context

  • Online store (PostgreSQL) receives array data correctly
  • The PyArrow table contains correct array data before parquet write
  • Parquet file contains correct data when read locally
  • Only BigQuery load loses the array content
  • Using load_table_from_json instead of parquet works correctly
  • Adding enable_list_inference=True to ParquetOptions fixes the issue

Activity

  1. max36067
    added
    kind/bug
    priority/p2
    on Jan 13, 2026
  2. ntkathole
    added 3 commits that reference this issue on May 14, 2026

    fix(bigquery): Enable list inference for parquet loads in offline_wri…

    This issue will close once commit 899ca66 is merged into the 'master' branch.
    899ca66

    fix(bigquery): Enable list inference for parquet loads in offline_wri…

    This issue will close once commit fe19c06 is merged into the 'master' branch.
    fe19c06

    fix(bigquery): Enable list inference for parquet loads in offline_wri…

    This issue will close once commit 9243497 is merged into the 'master' branch.
    9243497
  3. ntkathole
    closed this as completedin #6365on May 14, 2026
  4. rpathade
    added 2 commits that reference this issue on May 21, 2026

    fix(bigquery): Enable list inference for parquet loads in offline_wri…

    This issue will close once commit c1dcfcf is merged into the 'master' branch.
    c1dcfcf

    fix(bigquery): Enable list inference for parquet loads in offline_wri…

    This issue will close once commit de050ab is merged into the 'master' branch.
    de050ab
  5. ntkathole
    added a commit that references this issue on May 23, 2026

    feat: Add enabled/disabled toggle for feature views (#6401)

    This issue will close once commit 5f1fa0d is merged into the 'master' branch.
    5f1fa0d
  6. franciscojavierarceo
    added a commit that references this issue on Jun 13, 2026

    chore(release): release 0.64.0

    9bb73cc
  7. Srihari1192
    added a commit that references this issue on Jun 22, 2026

    chore(release): release 0.64.0

    22c3721
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugpriority/p2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.