GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |
GeezerVison is showing github.com — Render rich reprs without pandas, byte-identical to pandas output by eriknw · Pull Request #605 · python-graphblas/python-graphblas Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@ ## 33-udf-ret-dtype #605 +/- ## ===================================================== + Coverage 73.86% 98.44% +24.58% ===================================================== Files 108 108 Lines 33800 34582 +782 Branches 3919 3984 +65 ===================================================== + Hits 24967 34045 +9078 + Misses 8369 376 -7993 + Partials 464 161 -303 🚀 New features to boost your workflow:
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
The rich repr and _repr_html_ previously built a pandas DataFrame and let pandas render it. Rendering is now done in-house: a lightweight numpy-only frame, a ported text formatter (cell formatting with precision and scientific switchover, column widths, max_rows and max_columns truncation, terminal fit, multiline wrap), and an html builder reproducing the notebook table. With pandas absent, users now get the same rich repr they previously needed pandas for; with pandas present, output is byte-identical, and pandas display options that could affect the old output are still honored (max_rows, min_rows, max_columns, width, expand_frame_repr, precision, max_colwidth, chop_threshold, colheader_justify, float_format, html.border, html.use_mathjax). Known non-reproduced corners, all obscure: CJK east-asian width handling; pprint_nest_depth and max_seq_items on huge UDT cells; notebook_repr_html=False, where pandas' _repr_html_ returns None and the old code embedded the literal string "None" in its place, so the new table is a fix rather than a regression; and max_colwidth of 3 or less, where pandas narrows the column labels too and leaves them ragged while the hand renderer keeps them padded to a uniform width. That last case needs a deliberate max_colwidth <= 3 (the default is 50) together with column labels of differing digit counts, and it affects the text repr only, since the html path strips cell padding. The no-pandas tests now assert the pandas-absent output equals the pandas-present output instead of pinning a header-only fallback. Verified against the previous pandas render with a differential that swaps which formatting module graphblas.core resolves to and compares repr() and _repr_html_() end to end: 12376 checks over shapes, density, mask and transpose views, and every dtype including two UDTs; 82320 checks over float and complex cell formatting (precision 0 through 17, the scientific switchover boundaries, chop_threshold, float_format); 1024 checks over expression and infix-expression reprs in both autocompute modes; and 49140 checks over pandas display options crossed with six terminal sizes. The only divergences are the two described above. Rendering is also faster, since pandas' formatter machinery is no longer built per call: repr of a 4x4 Matrix ~850us -> ~90us, of a 10-element Vector ~1.5ms -> ~124us, of a 1e6-element Vector ~9.6ms -> ~1.1ms, and _repr_html_ of a 4x4 Matrix ~373us -> ~53us. These are best-of-5 with both paths interleaved in one process; the ratios (7x to 12x) held across repeated runs on a shared machine, the absolute figures less so.
The rich repr renders without pandas as of the previous commit, so 30
of the 38 skipif("not pd") gates in test_formatting.py are obsolete;
in a pandas-free environment those tests now run instead of skipping
(12 -> 42 passing). The 8 tests exercising pd.option_context stay
gated, as do the four autocompute tests with their own runtime skip.
Several un-gated tests assert the full pandas-style HTML verbatim and
pass without pandas, confirming the hand renderer is byte-identical.
Verified with pandas installed (unchanged: the removed gates never
fired there) and with pandas blocked at sys.meta_path, where
test_formatting.py goes from 12 passed / 42 skipped to 42 passed / 12
skipped, with no failures.
Since the pandas-free repr, test_autocompute, test_autocompute_html, test_index_expr_autocompute, and test_scalar_infix_expr_repr_auto only assert repr/_repr_html_ output that renders byte-identically without pandas, so the runtime "if not pd: skip" guards are obsolete. Removed them. The eight large-repr tests that call pd.option_context stay gated (they genuinely need pandas to exercise user display options). Verified with pandas installed (unchanged: the removed guards never fired there) and with pandas blocked at sys.meta_path, where test_formatting.py goes from 42 passed / 12 skipped to 46 passed / 8 skipped; the four moved from skip to pass, with no failures.
A sparse object whose values all fit in display.max_rows is shown in COO form with a typed value column, and only that path reaches the float and complex formatting ported from pandas: one number of decimals per column, trimmed trailing zeros, NaN and inf, the switch to scientific notation, chop_threshold, and a float_format callable. test_formatting.py pinned only int-valued COO reprs, so none of it ran. Pin text and HTML reprs of float, float32, complex and complex64 values (including each mix of NaN real and imaginary parts), the precision and chop_threshold options, a float_format callable in COO form and in the dense grid, a bool column, and a float column in a COO form cut short by "...". Every expected string is what main's pandas renderer gives for the same input. display_options sets an option in the renderer's own defaults, which it reads when pandas is absent, and also through pandas when pandas is installed, so these tests run with the same expected strings in both environments.
The hand renderer honors pandas display options that no test set: max_colwidth cutting a cell short, colheader_justify "left" and "center", html.use_mathjax off, a frame wider than display.width wrapping into blocks, expand_frame_repr off, max_rows=0 sizing the rows to the terminal's height, and max_rows or max_columns leaving room for a single row or column, where the first one is kept instead of a head and a tail. Pin each, with expected strings from main's pandas renderer.
The HTML repr renders a table even for an object with no rows or no columns, where the text repr stops at the header, so its columns can be empty. A record UDT with one field shows as a 1-tuple with its trailing comma. Pin both, with expected strings from main's pandas renderer. Two branches that no input reaches are excluded from coverage: the object-column formatter's case for a None cell, kept from pandas (its cells are numpy values or the dense grid's, which blanks None), and _trim_zeros_complex's guard for no values (it runs only for a complex column, and a column with no values becomes a float array).
"Render rich reprs without pandas, byte-identical to pandas output" lists two places where the hand renderer departs from the pandas-based one on purpose. With display.max_colwidth of 3 or less, pandas narrows the column labels too and leaves labels of different digit counts ragged, while the hand renderer keeps them at one padded width. With display.notebook_repr_html off, pandas' DataFrame._repr_html_ returns None, which the old code embedded as the text "None" in place of the table; the table is now shown either way. Pin both.
Behavior fix. To read only the rows and columns it shows, the repr copied
them into an empty object by assigning a value found with
reduce(monoid.any), then the object's own values over it. A UDT has no
monoid without Numba, so the repr of any UDT Vector or Matrix with more
rows or columns than fit raised there:
udt = dtypes.register_anonymous(np.dtype([("x", np.int16)]), "rec")
v = Vector(udt, size=300)
v[:] = (1,)
repr(v)
# before, without Numba: KeyError: 'any does not work with (rec, rec) types'
# after: the first and last columns, as with Numba
On main too. The rows and columns are now extracted and assigned in
place, which needs no operator for any dtype, and with Numba no monoid is
compiled for a UDT just to show it. The output is unchanged: the new test
expects what main's pandas renderer shows with Numba, and every repr
pinned in test_formatting.py is unchanged.
Two inputs that main's pandas renderer raised on are shown by the hand
renderer, so here it is an improvement, not a byte-identical copy:
- display.max_rows=0 in a terminal with room for one row: the frame is
truncated to no rows, and pandas raised TypeError ("'int' object is not
iterable") formatting it. Now the header and a ".." row are shown.
- a record UDT in COO form: pandas raised TypeError (isnan on the
structured column). Now each entry is a tuple, as in the dense grid.
This is the only COO value column that is neither numeric nor a bool.
Pin both.
Behavior fix. repr and _repr_html_ of a sparse Vector or Matrix of an
array UDT raised whenever the values were shown in COO form:
udt = dtypes.register_anonymous(np.dtype((np.int8, (3,))), "int8x3")
v = Vector(udt, size=1000)
v[100] = (1, 2, 3)
repr(v)
# main: ValueError: Per-column arrays must each be 1-dimensional
# (from pandas.DataFrame)
# before: TypeError: unsupported format string passed to list.__format__
The COO value column got the int formatter from the 2-D values' element
type. Its entries are now lists, formatted as objects, as the dense grid
already shows them; this is what pandas renders for the same frame with
one list per entry, in text and HTML:
index val
0 100 [1, 2, 3]
1 500 [-1, 0, 1]
Behavior fix. The HTML repr of a Matrix infix expression, such as A | B
or A @ B, put the left operand's dtype under "right_dtype" and the right
operand's under "left_dtype". Its text repr, and the Vector and Scalar
HTML reprs, label them correctly. With A INT8 and B FP64:
(A | B)._repr_html_()
# before: right_dtype INT8, left_dtype FP64
# after: left_dtype INT8, right_dtype FP64
On main too, since "Add reprs for infix expressions!" (502ed2a, first
released in 1.3.4). Five expected strings pinned the swapped labels,
with both operands of one dtype so the swap never showed in the values:
A & B.T, A | A, A @ B and A.T @ B.T in test_infix_expr_repr_html, and
A & A in test_autocompute_html. They now expect the labels in order, and
a new case, A | F with INT64 and FP64 operands, checks each dtype sits
under its own side's label.
Behavior fix. A Matrix or Vector shown as a grid reads only the rows and
columns the renderer will show: the first and last half of what fits,
with blank filler between them that the renderer cuts. _get_chunk sized
those halves from min_rows alone, and from max_columns for columns, which
broke whenever that disagreed with what the renderer shows (min(min_rows,
max_rows), or max_rows when min_rows is 0, as pandas does), or when the
half was 0. On main too, with pandas:
display.max_columns=1, or min_rows=1 or 0
# TypeError: Invalid dtype for index: float64 (nothing was read)
display.max_rows=4 (below the default min_rows=10), 40x3 Matrix
# rows 35 and 36 shown in place of rows 0 and 1 (and row 0's
# value under the label 35 when row 35 is empty)
display.max_rows=2, 3x3 Matrix
# IndexOutOfBound (the halves overlapped)
display.max_rows=0 (fit the terminal), 40x3 Matrix, 8-line terminal
# rows 35 to 39 alone, under no "..", and row 0's value at 35
# when row 35 is empty
The halves now come from the count the renderer shows, the first
max_rows (or max_columns) items are read when that count is 1, and with
max_rows=0 the count is the terminal's rows, which the renderer uses
then. A Matrix is shown in COO form instead when the entries that form
would show outnumber twice the values read for the grid; that count of
entries now stops at the rows the renderer shows too, so reading fewer
rows does not tip a dense Matrix into COO form. Every case above now
shows what pandas shows for the whole frame, as does a grid of max_rows,
min_rows, max_columns and terminal heights over several shapes, wherever
the repr keeps the grid form and pandas does not raise itself.
display.max_rows, min_rows or max_columns set to None still raises, as
on main.
Behavior fix. ss.itervalues and ss.iteritems looked up
lib.GxB_Iterator_get_<dtype name>, which exists only for built-in types,
so for any UDT they raised AttributeError, and so did ss.head(n) with n
below nvals, which iterates. The repr reads the first values with head,
so a sparse UDT object with more values than display.max_rows could not
be shown:
udt = dtypes.register_anonymous(np.dtype([("x", np.int16)]), "rec")
v = Vector(udt, size=1000)
v[100:800:10] = (1,)
repr(v)
# before: AttributeError: cffi library 'suitesparse_graphblas._graphblas'
# has no function, constant or global variable named
# 'GxB_Iterator_get_rec'
# after: the first display.min_rows entries in COO form, then "..."
On main too, since the iterators were added (#201, 2022.4.1). A UDT's
value is now read with GxB_Iterator_get_UDT (in every
python-suitesparse-graphblas from 7.4.0.0 on) into a buffer and converted
as Scalar.value converts it: a record or an array. head fills its result
from a list, since numpy reads a tuple of records as one record, and an
array UDT's dtype would broadcast each array value to its own shape.
Behavior fix, in the hand renderer only. pandas formats integer row and
column labels with a sign column, makes them one width, cuts any longer
than display.max_colwidth (when above 3) to max_colwidth - 3 characters
plus "...", and then trims the leading spaces they all share. The hand
renderer padded them without the cut, so labels wider than
max_colwidth showed whole, in the text and in every HTML label cell:
A = Matrix(int, 10**6, 10**6)
A[:, :] << 1
with pd.option_context("display.max_colwidth", 6, "display.max_rows", 4,
"display.min_rows", 4, "display.max_columns", 4):
repr(A)
# main (pandas):
# 0 1 ... 99... 99...
# 0 1 1 ... 1 1
# ...
# 99... 1 1 ... 1 1
# before:
# 0 1 ... 999998 999999
# 0 1 1 ... 1 1
# ...
# 999999 1 1 ... 1 1
_format_labels now formats integer labels the way pandas does, and the
text's row labels go through it as the HTML's already did.
The text's row-label column is also padded to one width, so the "..."
separator of a Matrix with truncated rows keeps all three dots when
max_colwidth is 3 or less, as in pandas.
The same labels caused the difference at a max_colwidth of 3 or less that
"Render rich reprs without pandas, byte-identical to pandas output"
documents and "Pin two reprs that deliberately differ from pandas'
output" pins: pandas narrows them to max_colwidth there without cutting.
That difference is gone too, so the output now matches pandas at every
max_colwidth, and that pin now expects pandas' output.
display.notebook_repr_html is the one documented difference left.
The "..." label of truncated rows keeps all three dots only because _get_index_strcol pads every row label to the widest one; no test failed with that padding removed.
test_repr_max_rows_below_min_rows built its dense Matrix from np.arange, whose integers are int32 by default on Windows with numpy before 2, so its repr said INT32 there and the test failed (windows-latest, numpy 1.25). The renderer is right; give the array an int64 dtype.
…repr
Behavior change. pandas allows None for display.max_rows,
display.min_rows and display.max_columns ("no limit"), and repr and
_repr_html_ of a Matrix or Vector raised TypeError for each of them, in
reading the values to show or in choosing the COO form:
A = Matrix.from_coo([0, 1], [0, 1], [1, 2], nrows=100, ncols=3)
with pd.option_context("display.max_rows", None):
repr(A) # was TypeError: '<=' not supported between instances
# of 'int' and 'NoneType'; now shows all 100 rows
A repr cannot read every row of a Matrix with 2**60 rows, so None now
shows a frame of up to 151 rows or 150 columns whole, and the first and
last 75 rows or columns of a larger one. 150 is what a max_columns of 0
already reads for the renderer to fit to the terminal; it bounds a grid
at 150 by 150 values, which renders in about 20 ms. In pandas' terms:
- max_rows=None is max_rows=151 with min_rows=150, whatever min_rows is
(pandas ignores min_rows when max_rows is None). The extra row lets
the COO form show 150 entries and its "..." row: with a max_rows of
150 the renderer would cut that 151-row frame again, as it does
whenever min_rows is at least max_rows.
- min_rows=None is max_rows, as pandas documents.
- max_columns=None is max_columns=150, and a wide frame wraps at
display.width, as with any max_columns above 0.
The renderer and the code that reads the values take these limits from
one place, so the rows and columns read are the ones shown.
…mplex types
psg 7.4.0 to 7.4.3.0 on Windows is built without complex types, so
dtypes._supports_complex is False and a complex literal registers as an
anonymous UDT named dtype('complex128'). Its values render the same, but
the header's dtype cell says dtype('complex128'), not FC64, so four repr
tests from this PR failed there: test_sparse_complex_repr,
test_sparse_repr_chop_threshold, test_repr_float_format and
test_repr_max_rows_zero (i605w).
test_sparse_complex_repr now skips without complex types, as the other
complex tests do; the chop-threshold and float-format tests stop before
their complex cases; and the max_rows=0 test, which is about rows, not
complex, uses FP64 values.
The chop-threshold and float-format tests ended early on a build without complex types, and those return lines went uncovered on every other build. As separate tests with the skip of test_sparse_complex_repr, every line runs wherever complex types exist, and nothing else changes.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.