GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — NaN in FLOAT/DOUBLE columns is returned as None (same as NULL) when pandas is enabled · Issue #978 · databricks/databricks-sql-python

NaN in FLOAT/DOUBLE columns is returned as None (same as NULL) when pandas is enabled #978

New issue
Copy link
New issue
Copy link
Open
#979
Open
NaN in FLOAT/DOUBLE columns is returned as None (same as NULL) when pandas is enabled#978
#979
Copy link
Labels
bugSomething isn't workingSomething isn't workingengineer-botMaintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).Maintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).

Description

@maharanay22
maharanay22
opened on Oct 8, 2026
Issue body actions

With pandas enabled (the default), a floating-point NaN in a FLOAT or DOUBLE column is returned as None, so it can't be told apart from SQL NULL. With _disable_pandas=True the same data comes back correctly as float('nan').

Example: a DOUBLE column containing [NaN, NULL, inf, 1.5]

  • default (pandas enabled): [None, None, inf, 1.5]
  • _disable_pandas=True: [nan, None, inf, 1.5]

Reproduced on pandas 2.3.3 and 3.0.6 (pyarrow 25, connector main at 61b9a7f).

Cause: ResultSet._convert_arrow_table (src/databricks/sql/result_set.py) maps float32/float64 to pandas' nullable Float32Dtype/Float64Dtype. That conversion turns IEEE NaN into pd.NA, and df.to_numpy(na_value=None, dtype="object") then turns it into None.

This changes user-visible results, so I'm opening this issue alongside the fix: I have a fix with regression tests ready and am opening the PR now. It restores NaN from the Arrow column after the pandas conversion, keeps NULL as None, and matches the _disable_pandas=True output.

Activity

  1. vuanhphung
    added
    bugSomething isn't workingSomething isn't working
    engineer-botMaintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).Maintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).
    on Oct 29, 2026
  2. peco-engineer-bot commented on Oct 29, 2026

    @peco-engineer-bot
    peco-engineer-botBot
    on Oct 29, 2026 – with Peco Engineer Bot
    Contributor
    More actions

    🤖 engineer-bot opened a fix PR: #1005

    Review before merge.

  3. maharanay22 commented on Oct 30, 2026

    @maharanay22
    maharanay22
    on Oct 30, 2026
    Author
    More actions

    Thanks! There's already a fix for this in #979, open since Oct 8, rebased on 4.8.0 with all unit tests passing. Happy to fold anything useful from #1005 into it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingSomething isn't workingengineer-botMaintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).Maintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.