GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Fix chop_threshold crashes in the pandas-free complex repr by eriknw · Pull Request #620 · python-graphblas/python-graphblas

Conversation

@eriknw

@eriknw eriknw commented Aug 5, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Stacked on #605. An overnight differential fuzz of #605's pandas-free renderer against the real pandas renderer (34,618 repr/HTML comparisons across 17,309 case pairs) found a crash family in the display.chop_threshold path for complex dtypes, reachable in the COO (long) display form:

  1. Tip-only crash, pandas renders fine: abs() of a python complex raises OverflowError for finite components near the float max, where numpy's abs (which pandas uses) returns inf and never chops. Fixed by treating an overflowing magnitude as never chopped; the result is byte-identical to the pandas renderer (verified, repr and HTML hashes equal against 6f1eb02).
import numpy as np, pandas as pd, graphblas as gb
big = float(np.finfo(np.float64).max)
v = gb.Vector.from_coo([10_000, 500_000], [complex(big, big), 1e-7], size=10**6, dtype="FC64")
with pd.option_context("display.chop_threshold", 1e-8):
    repr(v)   # OverflowError before this fix; byte-identical to pandas after
  1. Both-sides crash: a complex value that actually chops was replaced with float 0.0, putting a j-less string into the complex column, which _trim_zeros_complex cannot parse (IndexError). pandas 3.0.3 crashes the same way on the same input in its own chop path, so byte-identity would mean reproducing a pandas crash. This fix chops to the value's own type of zero, so the cell renders 0.000000e+00+0.000000e+00j and the object reprs where pandas cannot. A deliberate, documented divergence, reachable only with chop_threshold set on a complex dtype.

Gates: full pinned suite 1114 passed / 145 skipped; pre-commit all hooks pass. The broader fuzz findings (all in non-crash territory) are written up in the overnight stack report for the maintainer meeting.

Two review follow-ups add tests:

  • Test that the complex repr's overflow warning cannot escape on any numpy: the
    np.errstate(over="ignore") fix matters only on numpy 1.24 on arm64, so no test could fail
    without it. The test swaps in an abs that overflows, so removing the fix fails on every numpy.
  • Test the complex repr's chop_threshold on FC32 too: FC32 values reach the formatter as Python
    complexes, so their magnitudes never overflow. A one-line comment notes the only divergence from
    pandas this causes, which needs a chop_threshold above the float32 maximum.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@eriknw eriknw mentioned this pull request Aug 5, 2026
Keep the CI matplotlib pick compatible with the numpy pick #621
Merged
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 7ee751f to 44033e7 Compare August 5, 2026 17:44
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 44033e7 to 2fa3cb8 Compare August 5, 2026 18:03
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 2fa3cb8 to 376ee94 Compare August 5, 2026 18:05
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 376ee94 to e0eb986 Compare August 6, 2026 07:59
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from e0eb986 to 9d99244 Compare August 6, 2026 15:39
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 2 times, most recently from 25d557b to dbff5f6 Compare August 6, 2026 20:36
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from dbff5f6 to b022fa4 Compare August 6, 2026 20:41
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from b022fa4 to b66bc7c Compare August 7, 2026 02:49
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from b66bc7c to d09186f Compare August 7, 2026 05:09
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from d09186f to f9295a2 Compare August 26, 2026 17:30
@eriknw eriknw mentioned this pull request Aug 26, 2026
Stop gb.ss import from turning the SuiteSparse JIT on #630
Merged
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from f9295a2 to 9a142d0 Compare September 18, 2026 03:09
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 9a142d0 to 582a1d6 Compare September 18, 2026 03:46
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 582a1d6 to 5910201 Compare September 18, 2026 17:34
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 2 times, most recently from 363fb20 to 9fbecb9 Compare September 23, 2026 16:22
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 4 times, most recently from 7ea40c7 to 7929d00 Compare October 3, 2026 04:57
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 7929d00 to 9541cd1 Compare October 4, 2026 06:21
@eriknw
eriknw removed this pull request from stack #634 October 8, 2026 21:02
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 9541cd1 to ac94068 Compare October 8, 2026 21:40
@eriknw
eriknw added this pull request to stack #635 October 8, 2026 21:46
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from ac94068 to f0c372e Compare October 9, 2026 03:06
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch 2 times, most recently from 440df4a to 887eca6 Compare October 14, 2026 18:45
@eriknw
eriknw removed this pull request from stack #635 October 19, 2026 07:45
@eriknw
eriknw changed the base branch from 38-monoid-guard-udt-fallback-builtins to main October 19, 2026 07:46
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 887eca6 to 1178c46 Compare October 19, 2026 07:47
@eriknw
eriknw changed the base branch from main to 25-pandas-free-repr October 19, 2026 07:49
@eriknw
eriknw added this pull request to stack #650 October 19, 2026 07:50
@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from 1178c46 to a4bdb8d Compare October 19, 2026 07:55
@codecov

codecov Bot commented Oct 19, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.87179% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (25-pandas-free-repr@cb47ed0). Learn more about missing BASE report.

Additional details and impacted files
@@                  Coverage Diff                   @@
##             25-pandas-free-repr     #620   +/-   ##
======================================================
  Coverage                       ?   87.00%           
======================================================
  Files                          ?      108           
  Lines                          ?    34583           
  Branches                       ?     3981           
======================================================
  Hits                           ?    30088           
  Misses                         ?     4196           
  Partials                       ?      299           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eriknw
eriknw force-pushed the 39-repr-chop-complex-overflow branch from a4bdb8d to 53ad8e3 Compare October 20, 2026 22:28
eriknw added 5 commits November 12, 2026 11:25
@eriknw
Fix chop_threshold crashes in the pandas-free complex repr
15a35ed
The COO form formats a complex column through _format_float_column, where
display.chop_threshold compares each value's magnitude. Two crashes hid
there, found by an overnight differential fuzz against the pandas
renderer:

- abs() of a python complex raises OverflowError for finite components
  near the float max, where numpy's abs returns inf. pandas therefore
  never chops such a value and renders the object fine; the pandas-free
  path crashed. Treat an overflowing magnitude as never chopped.
  Byte-identity for this case is restored, verified against the pandas
  renderer at 6f1eb02 (repr and HTML hashes equal).

- A value that actually chops was replaced with float 0.0, which puts a
  j-less string into the complex column, and _trim_zeros_complex cannot
  parse that (IndexError). pandas 3.0.3 has the same bug in its own chop
  path (pandas.io.formats.format._trim_zeros_complex raises the same
  IndexError on the same input), so byte-identity here means identity
  with a crash. Chop to the value's own type of zero instead: the
  chopped cell renders 0.000000e+00+0.000000e+00j and the object reprs
  where pandas cannot.

The second point is a deliberate, documented divergence from pandas,
reachable only with display.chop_threshold set on a complex dtype.
@eriknw
Skip the complex chop_threshold test when FC64 is unavailable
0c3dd62
On Windows with python-suitesparse-graphblas older than 7.4.3.1 the CFFI
bindings are built without complex support (MSVC lacks C99 _Complex), so
dtypes._supports_complex is False, FC64 is never registered, and
Vector.from_coo(..., dtype="FC64") raises ValueError before the test
gets to the formatting code under test. Guard with the same
skipif("not dtypes._supports_complex") pattern test_scalar.py uses;
the module already imports dtypes, which the string condition needs.

Verified the test still runs and passes on a complex-capable host, and
that the skipif string evaluates to True in the module namespace when
_supports_complex is flipped to False.
@eriknw
Compute the complex repr's magnitudes without numpy's overflow warning
9eb500e
test_chop_threshold_complex failed on the macos-latest job that drew numpy
1.24.4:

    RuntimeWarning: overflow encountered in absolute

The pandas-free float formatter takes np.abs of the column to choose between
fixed and scientific notation. For a complex value with components near the
float max the magnitude is inf, which those checks already handle, but numpy
1.24 on arm64 also raises the FP overflow flag computing it (1.26 and 2.4 do
not), so the warning escaped from repr and the test suite's filterwarnings
setting turned it into an error. Reproduced locally with numpy 1.24.4,
pandas 2.2.3 and psg 8.2.0.1, then green with the magnitudes computed under
np.errstate(over="ignore").

pandas' FloatArrayFormatter makes the same np.abs call unguarded (2.2 and
3.0), so this departs from it only in not warning.
@eriknw
Test the complex repr's chop_threshold on FC32 too
d411415
test_chop_threshold_complex covered FC64 only. Run it on FC32 as well:
its largest value renders, and a value below the threshold chops to
complex zero, as for FC64. Replacing the chop's own type of zero with a
float zero makes the FC32 case fail with the same IndexError.

FC32 reaches the threshold by a different route. The COO frame hands the
formatter python complexes, so abs() is computed in double and is finite
where pandas computes an inf magnitude in float32. The two differ only for
a threshold above the float32 max, so a comment notes it rather than code.
@eriknw
Test that the complex repr's overflow warning cannot escape on any numpy
e11c543
Computing the magnitudes under np.errstate(over="ignore") fixed a
RuntimeWarning that only numpy 1.24 on arm64 raises, so on any other numpy
no test failed with that fix removed. Stand in for that numpy with an abs
whose own arithmetic overflows (the squares of components near the float
max), and check that repr and the HTML repr raise no RuntimeWarning for an
FC64 value there. Without the errstate the test fails with "overflow
encountered in multiply".

FC32 values reach the formatter widened to double, so their magnitudes
never overflow and the test covers FC64 only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@eriknw
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.