GeezerVison is showing github.com — Convert nested result columns without numpy float conversion by aminghadersohi · Pull Request #960 · databricks/databricks-sql-python
Conversation
What type of PR is this?
Description
With pandas enabled (the default), ResultSet._convert_arrow_table converted every column through to_pandas() and to_numpy(). For nested columns an ARRAY<BIGINT> containing a NULL became a float64 array. Observed on a SQL warehouse with 4.4.0 and 4.6.0: array(9007199254740993, -9223372036854775808, NULL) read back as [9007199254740992.0, -9.223372036854776e+18, nan].
Change:
- Nested (ARRAY/MAP/STRUCT) columns are converted from Arrow directly and no longer pass through pandas; only the scalar columns go through the existing pandas path.
- Return types are unchanged and match the
_use_arrow_native_complex_types documentation: ARRAY is a numpy.ndarray, MAP a list of (key, value) tuples, STRUCT a dict, at every nesting level.
- ARRAY elements keep their native dtype for float arrays and integer/boolean arrays without NULLs (e.g.
int64). An integer array with a NULL is now an object ndarray of exact int/None instead of float64 with NaN. The same applies to integer values nested inside MAP and STRUCT values.
- Changelog entry under
# Unreleased.
How is this tested?
- Unit tests
- E2E Tests
- Manually
- N/A
tests/unit/test_pandas_compatibility.py: test_list_of_bigint_with_null_is_exact (fails on main), plus tests for native dtypes of NULL-free arrays, nested ARRAY/MAP/STRUCT values, and a table of only nested columns split across chunks. The original fix was verified live against a SQL warehouse (Core select of ARRAY/DECIMAL(38,18)/BOOLEAN with NULLs, exact values); the follow-up commit restoring numpy.ndarray for arrays has not yet been run against a warehouse or through the unit suite locally, so CI is the check for it.
Related Tickets & Documents
None.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
With pandas enabled, ARRAY/MAP/STRUCT values went through numpy: an
ARRAY<BIGINT> containing a NULL came back as float64, so values beyond
2**53 lost precision and NULL became NaN. Take nested columns directly
from Arrow (to_pylist), as the disable_pandas path already does.
Signed-off-by: Amin Ghadersohi <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🟡 Changes recommended
Unresolved API compatibility and nested-value conversion performance concerns remain.
Review effort: Lite
Findings: 2 
Open (2)
What changed in this PR
Updates Arrow result conversion to preserve exact nested values without NumPy float conversion.
Changes:
- Converts nested columns with
to_pylist().
- Adds regression coverage for nullable large
BIGINT arrays.
- Leaves scalar pandas conversion unchanged.
| File |
Description |
src/databricks/sql/result_set.py |
Preserves nested Arrow values during conversion. |
tests/unit/test_pandas_compatibility.py |
Tests exact nullable nested integer preservation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Nested columns are converted from Arrow directly, so they are no longer
converted through pandas and then replaced. ARRAY values stay
numpy.ndarray, the documented type: native dtype when the elements are
float, or integer/boolean without NULLs; object dtype with exact int and
None when integer elements contain NULL (previously float64 with NaN).
MAP stays a list of tuples and STRUCT a dict, with the same conversion
applied to their nested values.
Signed-off-by: Amin Ghadersohi <[email protected]>
This branch has not been deployed
No deployments
Reviewers
At least 1 approving review is required to merge this pull request.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|