GeezerVison

What type of PR is this?

• Other (performance)

Description

fetchall() and large fetchmany() calls convert the Arrow result with one to_pandas() call and one DataFrame.to_numpy(dtype=object) over the whole table, so a DataFrame, a full object array and the final Row list are all alive at the peak. On a 5M-row mixed-type result (INT/BIGINT with NULLs, DOUBLE, STRING, DECIMAL, DATE, TIMESTAMP, BOOLEAN, ARRAY) under Python 3.11, conversion peaked at about +5.3 GB RSS, while the returned rows alone hold about +3.9 GB.

This converts 100,000 rows per to_pandas() call. That is the default arraysize, so fetchone() and default-size fetchmany() take the same single-call path as before. Some columns are still converted once over the whole table, exactly as before, so every returned value and type is unchanged:

• Nested columns. pyarrow picks the element dtype of a list or map from the whole column (one NULL element turns every ARRAY<BIGINT> value into a float64 array) and of a struct per Arrow chunk.
• Decimal, duration and interval columns. Their Python values (Decimal on Python 3.13+, Timedelta, DateOffset) are tracked by the cyclic GC. Created slice by slice between batches of rows, they made the GC run 13 full collections instead of 5 for this result on Python 3.13/3.14, and the conversion was 7–15% slower than main there. Created first, as on main, they don't.

Null placeholder columns keep each partial DataFrame multi-block like the whole table's, because pandas < 2.2 returns durations from a single-block DataFrame as ints.

Rows are now also built with tuple.__new__(Row, values) and the usual __fields__ assignment, instead of three Python calls per row (Row.__call__ → _create_row → Row.__new__). The rows are the same objects; tests compare them against Row(*names)(*values). With more values than column names, it falls back to ResultRow(*values), which raises the same error.

Related open PRs: #982 and #1007 call _convert_arrow_table unchanged and merge cleanly. #979 (NaN vs NULL) and #960 (nested conversion) edit the same function body, so whichever lands second needs a small rebase: #979's NaN restore moves into to_object_array, and #960's nested conversion replaces the whole-column call. This PR doesn't change either behavior. #1043 is an optional follow-up stacked on this branch that pauses the GC while rows are built.

How is this tested?

• Unit tests
• E2E Tests
• Manually

The unit tests cover:

• sliced conversion matching whole-table conversion value by value and type by type (nested, struct, decimal, duration, interval, tz timestamps, duplicate column names);
• bounded slices, and which columns are converted whole;
• converted rows behaving like Row(*names)(*values): type, __fields__, equality, hash, repr, asDict(), attribute/index/key access, pickled bytes, copy, and the too-many-values error.

They pass on Python 3.9–3.14 with pandas 1.5/2.2/3.1 and pyarrow 14–26. A main-vs-branch script also compared every value and type on about 3,600 table layouts and slice sizes in each of those environments, and on live Thrift inline, CloudFetch and kernel results. All were identical.

The table below is for the 5M-row result above: _convert_arrow_table alone, a fresh process per run, main and this PR interleaved, median of 5. p95 is within 6% of the median. "+ later GC" adds a phase after the conversion that keeps allocating, so GC work deferred past the conversion is counted too.

Before the last two commits, live fetchall() over Thrift CloudFetch for this query on Python 3.11 had a 5.99 → 4.45 GB peak RSS. _disable_pandas=True is unaffected.

This pull request and its description were written by Isaac.