GeezerVison

Optional, stacked on #1039. The base is perf/fetchall-rows, and this diff only adds a GC pause while rows are built. #1039 is safe and faster than main on its own, so this can be merged after it or not at all. I'll retarget to main after #1039 merges.

What type of PR is this?

• Other (performance)

Description

After #1039, about a third of fetchall()'s conversion time goes to Python's cyclic garbage collector. Each row adds two GC-tracked objects (the Row and its __dict__), and the full collections triggered along the way rescan every row built so far. For 5M mixed-type rows under Python 3.11, #1039 runs 13,046 young, 1,185 middle and 17 full collections. A non-GC alternative isn't available without changing Row's shape, because the __dict__ holding __fields__ is the second tracked object, and removing it is an API decision.

This change pauses the cyclic GC while each batch of up to 100,000 rows is built. GC state is process-wide, so the pause has these guards:

• Only conversions of at least 10,000 rows pause, so fetchone()-sized conversions never touch the GC.
• The pause is shared by threads through a lock-protected counter. The first batch disables the GC only if it is enabled. The last batch to finish re-enables it and runs gc.collect(0), so other threads' cyclic garbage is still collected between batches even when conversions overlap.
• Batches stop joining a pause once it is 0.5 s old. The pause ends when the batches already in it finish.
• A GC that is already disabled when a batch starts stays disabled. If other code disables the GC while a pause is running, the end of the pause re-enables it, because the gc module gives no way to tell who disabled it.
• The pause is exception-safe, including an interrupt that arrives while a batch is joining it. A fork child resets it, since pauses held by the parent's other threads never end there.

How is this tested?

• Unit tests
• E2E Tests
• Manually

The unit tests cover per-batch pausing, small conversions, a caller-disabled GC, an exception mid-batch, overlapping pauses, the 0.5 s limit, the fork reset (including a real os.fork()), and four threads converting at once. They pass on Python 3.9–3.14 with pandas 1.5/2.2/3.1.

The table below is for the 5M-row result from #1039: _convert_arrow_table alone, a fresh process per run, main, #1039 and this PR interleaved, median of 5. "+ later GC" adds a phase after the conversion that keeps allocating. That phase counts the one full collection over the rows that now happens afterwards, where #1039 instead runs 13–17 of them during the conversion. Peak RSS is the same as #1039's.

To check concurrency, four threads converted 1M-row results at once for 90 s while a fifth thread created about 10 MB/s of cyclic garbage:

The GC was enabled at the end of every run. The longest window exceeds 0.5 s because a thread holding the pause can wait for the GIL while other threads run to_pandas(). By comparison, disabling the GC for a whole conversion gave windows of 22–26 s and 23–27 MB of uncollected garbage.

This pull request and its description were written by Isaac.