GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||
GeezerVison is showing github.com — Fix Scalar infix expressions with a literal operand by eriknw · Pull Request #654 · python-graphblas/python-graphblas Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@ ## main #654 +/- ## ========================================== - Coverage 98.79% 98.71% -0.08% ========================================== Files 105 105 Lines 30670 30890 +220 Branches 3607 3632 +25 ========================================== + Hits 30299 30493 +194 - Misses 240 263 +23 - Partials 131 134 +3 🚀 New features to boost your workflow:
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
ea6fb88 to
2ff3579
Compare
ce63433 to
70b2aee
Compare
70b2aee to
3c96265
Compare
3c96265 to
0dd4f52
Compare
0dd4f52 to
b433c10
Compare
b433c10 to
5ebe867
Compare
be9d4ec to
fc11612
Compare
…ns (#591) * Reject record UDT pairs whose leaf counts differ `_check_udt_pair` rejects two record operands that disagree on shape, but it compared only top-level field names. Two records can share those and still nest differently: `[("a", f8), ("b", f8)]` against `[("a", [("n1", f8), ("n2", f8)]), ("b", f8)]` both report fields `["a", "b"]` while contributing two leaves and three. The codegen walks one operand's leaf paths and applies them to both, so applying the scalar path `["a"]` to the nested side asks Numba to add a float to a record. Measured before this change, that pair gets through the checks and comes back from Numba's typing pass as: UdfParseError: binary.plus does not work with (_NestFlat, _NestDeep): No implementation of function Function(<built-in function add>) found for signature: >>> add(float64, Record(nst_n1[type=float64;offset=0], nst_n2[type=float64;offset=8];16;False)) That reports a compile failure for what is the same shape disagreement the three sibling checks in the same function already report as a KeyError. Comparing leaf counts makes it the fourth of those checks. The exception class changes, and that is visible in both directions: UdfParseError derives from GraphblasException, not from KeyError, so code catching KeyError to mean "no such op" now catches this pair, and code catching UdfParseError no longer does. `OpBase.__contains__` catches both, so `udt in binary.plus` is unaffected either way. * Ignore the UDT repr size warning in the nesting mismatch test * Compare record leaf paths and field shapes, not leaf counts Equal leaf counts do not make two records pair up. The codegen reads both operands at the left one's leaf paths, so records nested under different fields, or with differently named sub-records, still reached Numba's typing: a UdfParseError for plus, a raw TypingError for eq. The check now requires the same leaf paths in the same order, as it already does for top-level names. That also rejects reordered sub-record fields. They work here, but they give wrong answers once #593 pairs leaf dtypes by position and #595 branches on them. Array fields whose shapes do not broadcast were worse: (3,) with (4,) raised inside the cfunc, unseen, and left garbage in the result. Those are now KeyErrors, as is a scalar field on the left of an array one, which never compiled. The arithmetic ops also need the broadcast to fit the left field. A (1,) field left of a (3,) one was caught only by the UDF return check, worded as a UDF the user did not write. eq and ne reduce each field to one bool and keep accepting that pair. Fields that broadcast, such as (3,) with () or (1, 3), still combine; across 64 field-shape pairs under plus, min and eq, everything that worked before still does. The user guide now states the pairing rules. * Type UDT ops like built-in ops, cast UDT stores, broadcast array UDTs Lifted ops (plus, minus, times, truediv, floordiv, min, max, eq, ne, ainv, abs) type each UDT element or record leaf as the built-in op types that dtype, so results never narrow. A mixed pair gives an operand's type if it holds the result, else the promoted layout. Records pair on field names, order, nesting and field shapes; array UDTs broadcast as numpy arrays do. A Python number beside a UDT is weak per leaf (NEP 50), the same on numpy 1 and 2. A literal that must become a UDT element (tuple, array, user-op literal, ewise_union default) is converted only if no value changes. apply and ewise ops with a Monoid on a mixed pair use its BinaryOp. Storing into an object of another UDT casts each leaf as GraphBLAS casts built-ins; incompatible layouts and UDT/built-in pairs raise DomainMismatch. Assigning or setting an element from another UDT used to read raw bytes, past the end of a smaller element. ewise_add converts operands to the result type, which GraphBLAS cannot cast. Array codegen uses loops, in Numba and in C: typing an op on FP64[64,64] took ~112 s and now takes ~0.02 s. Also fixes several errors, eq on 2-D array fields, array-field truediv on small ints, a UDF registered under a lifted op's name, and packed vs aligned records. Docs and tests updated. * Cast UDT stores without temporaries, type UDT literals, fold in #592-#595 Store casts: an existing object casts in one pass, and element-wise lifted ops (apply, ewise_mult, ewise_union) write the target type directly. Any other store into another UDT type raises DomainMismatch and suggests obj << expr.new(). ewise_add on mixed-type Vectors/Matrices raises; infix + takes the union with each operand's zero (-0.0 in float fields). Literals beside a UDT: Python numbers and tuples/lists/dicts of them are weak per field (NEP 50), numpy values and Scalars are strong. A value that must become a UDT element (user op literal, select/IndexUnaryOp thunk, ewise_union default) must fit by type; one that fits only by value (0 for a bool field, 2.0 for an int field) still converts, with a DeprecationWarning. Lossy values raise. From #592-#595: min/max (fmin semantics, fixed -0.0 tie), floordiv, integer and complex division, and complex multiplication give the same bits on the Numba and C JIT paths. Array fields in records compute element by element. Also: extracting an element into another UDT type casts it instead of copying raw bytes (a segfault on main); Monoid literals are typed as their BinaryOp's; outer works on mixed types; the user guide's UDT rules are two tables. * Type Python numbers beside built-in dtypes as numpy 2 does (NEP 50) A Python number beside a Vector, Matrix or Scalar of a built-in dtype was typed through numpy, with a 0-d array of value 0 standing in for it (dtypes.unify). numpy 1 judged that array by its value, so the collection's dtype won and a large literal wrapped (int8_vec * 1000 multiplied by -24, int8_vec < 300 compared with 44); numpy 2 (NEP 50) made 0-d arrays strong, so every int was INT64 and every float FP64 (int8_vec + 1 was INT64, uint64_vec + 1 FP64, losing precision). The same script gave other dtypes, and on numpy 1 wrong answers, depending on the installed numpy. A Python bool, int, float or complex operand is now weak, by the table UDT leaves already use: it takes the other operand's dtype when that dtype's kind holds it, else INT64, FP64 or FC64, on numpy 1 and 2 alike. So int8_vec + 1 is INT8 and wraps as numpy does, int8_vec * 0.5 is FP64, fp32_vec * 0.5 is FP32, int8_vec + 300 raises OverflowError, and fp32_vec + 1e300 (or + 2**200) is inf with a RuntimeWarning, as in numpy 2. A strong operand keeps the old widening: int8_vec + np.int64(1). This covers apply (infix ops, binary.plus(v, 1), Monoids), Scalar ewise ops, ewise_union defaults, value thunks of select and IndexUnaryOps, and UDFs. A comparison keeps no result type, so an int out of range compares exactly (int8_vec < 300 is all True), and so does an op whose result is another type anyway: int8_vec / 300 is FP64, as numpy divides integers in float64, for UDT fields too. A to_dense fill value that does not fit widens, an int or a float (fp32_vec.to_dense(fill_value=1e300) is float64), rather than raising or becoming inf. numpy scalars, Scalars and typed ops stay strong, and a thunk that is not compared with the values, such as a row index, keeps its own type. unify no longer stands a scalar in as a value-0 array. eq and ne beside a UDT now type a Python number or sequence the same way, leaf by leaf, with the same exact fallback: fp32_udt == 0.1 compares in float32, as fp32_vec == 0.1 and numpy do. * Compare select value thunks exactly; isequal compares arrays as np.array_equal SuiteSparse:GraphBLAS's select with a value op (valueeq, valuelt, ...) typed for another type than the input's reads the thunk's bytes as the input's type (8.0.2 through 10.6.0; apply with the same op is right; fix proposed upstream in DrTimothyAldenDavis/GraphBLAS#463). On an INT8 vector, select(valueeq, 1.0) kept nothing and select(valuelt, 300) compared with 44, on main too. select now compares in the input's own type with a thunk that keeps the same entries (x < 2.5 is x < 3 for integers). isequal on array UDTs compares as np.array_equal: elements must have the same shape, apart from leading axes of length 1, so an FP64[3] of [1, 1, 1] is no longer isequal to an FP64[1] of [1]. == still broadcasts, and a Scalar's ==, which returns a bool, now computes binary.eq, as == on a Vector does (it called isequal). A literal beside a UDT Scalar is compared as == types it, so [0, 0, 0] == 0.5 is False (0.5 was truncated to 0), and Scalar(FP32 0.1) == 0.1 compares in FP32, as numpy does. A float too large for FP32 in a comparison (fp32_vec == 1e300) compares with infinity, as numpy 2 does, and now warns at the caller's line, as arithmetic does. A Python int from 2**63 to 2**64 - 1 now goes into a UINT64 Scalar, Vector or Matrix element (Scalar.from_value(2**63, "UINT64") and uint64_vec[0] = 2**63 raised OverflowError, on main too). * Compare a Python int beyond a 64-bit element's range exactly Commit 2 typed an int beyond an integer element's range by a type that holds it, so a comparison no longer raised, but beside INT64 and UINT64 that comparison runs in FP64, which rounds both: int64_vec < 2**63 was False for 2**63 - 1, uint64_vec == 2**64 was True for 2**64 - 1, and Scalar(2**63 - 1) == 2**63 was True, as for UDT int64 and uint64 leaves (main raised OverflowError). Such an int now compares as infinity of its sign, which every integer element compares with as with the int, so infix comparisons, apply, ewise_union defaults and Scalar == are exact, as in numpy 2, and beside a UDT a literal with one equals no element. An int too large for a float compares too (int64_vec < 10**400 and select(valueeq, 10**400) raised OverflowError). A float too large for a float32 UDT leaf (fp32_udt + 1e300, or a store) now warns at the caller's line, as a built-in FP32 does, instead of from inside python-graphblas, where the default filter showed it once per process. Drops a test helper nothing used. * Give each UDT layout its own JIT C name; Scalar == without Numba; CI fixes SuiteSparse's JIT knows a type by its C name alone, and a UDT's C name was the name it was registered under, so two layouts registered under one name shared JIT kernels: making an iso Vector non-iso ran a kernel compiled for the other layout, which corrupted the heap or gave wrong values (on main too; the new broadcasting test registers four layouts as _BcastX, and crashed 9 Linux CI jobs). A name already given to another layout now gets a fresh _gbudt_NNN, and gb.ss.register_new refuses a name another type has. Scalar == on a UDT runs binary.eq, which needs Numba; without Numba it raised KeyError, where it compared in Python before. It now compares the values in numpy, leaf by leaf in the type binary.eq compares in, and raises binary.eq's KeyError for UDTs that do not pair. Tests: guard the cases that need Numba, skip the 32 KB UDT on Windows (MSVC builds of SuiteSparse refuse a UDT over 1024 bytes), give the JIT name tests array lengths no other test registers, and expect numpy's own int dtype (int32 on Windows with numpy 1). * Spell out complex * and / for UDTs; Numba 0.68 follows Python 3.14 From Python 3.14, CPython recovers infinities that the textbook complex formulas leave as nan (C99 Annex G), and Numba 0.68 follows it there: 1j * (inf+infj) is -inf+infj and (inf+nanj) / (1+1j) is inf-infj, where numpy and the C JIT kernel give nan+nanj. The UDT cfunc used Numba's operators, so it disagreed with the C JIT kernel on CI's macOS job with Python 3.14 and Numba 0.68. The cfunc now spells out the formulas the C kernel uses (the textbook product, and Smith's division as CPython divided before 3.14), so the two agree on any Python and Numba. The user guide no longer calls a complex quotient numpy's: numpy multiplies by the divisor's reciprocal, so the last bit can differ, and numpy's overflows where the quotient is finite. * Move UDT and UDF logic out of the general modules Code moves, plus import changes and a few small helpers, so scalar, vector, matrix and base read mostly as the built-in path. - core/udt.py (new): rules for UDT elements: layout, op typing, store casts, the ewise_add rule, equality without Numba, and element values as bytes. The type rules moved here from udt_utils, which now only generates code and is renamed udt_codegen. - core/literals.py (new): NEP 50 typing of literal operands, for built-in and UDT types (moved from scalar.py and udt_utils). - operator/udf.py (new): the Numba-only UDF-on-UDT block from operator/base.py, plus _bool_to_int8. - scalar._union_default replaces the ewise_union default block, which was copied six times across Scalar, Vector and Matrix. - New helpers pulled out of scalar.py's inline code: _scalar_ewise_add_converted, _udt_eq_without_numba, _element_bytes and _element_value. core/udt.py imports operator, base and scalar names inside functions, after the built-in early returns, so built-in calls pay nothing extra. * Pass CODECOV_TOKEN to the coverage upload Codecov takes tokenless uploads only from forks. Every upload from this PR's CI runs was refused with "Token required because branch is protected", so the PR had no coverage report, and the PRs stacked on it had no base to compare against. The repository now has the secret. * Cover how core/literals.py types literal operands Tests for the literal-typing paths CI never ran: - Beside built-in dtypes: a comparison op given as a string with an int out of range (ewise_union with "<"), an op string that names no BinaryOp (the op reports it), an int beyond INT64 beside a bool (OverflowError, as numpy's bools & 2**70), a user's BinaryOp with an int out of range, and a complex literal on a backend without complex types (vanilla only). - Beside UDTs: an infinite or NaN Decimal, literals not laid out like an element (ragged, too long or too short for a field, not broadcastable) or not numbers, which numpy then rejects, a number beside a bytes field, a nested record that is not the last field, and the conversions that lose a number (an imaginary part, a float32 overflow, a fraction in a record's array field) or keep a bytes field as it is. - Scalar.isequal with a ragged literal on an array UDT (ValueError). * Cover select's rewrite of value thunks of another type Tests for the branches of operator/select.py's _value_select_in_own_type that CI never ran: a complex thunk beside integers and floats (only valueeq and valuene, which have an answer), a NaN thunk of another float type, an infinite thunk beside integers, and an empty thunk Scalar, which GraphBLAS rejects. * Cover UDT store casts, ewise_add casts and fused stores Tests for the paths in core/udt.py, Matrix and udt_codegen that CI never ran: - store casts between records whose field shapes differ or whose bytes fields differ in width (DomainMismatch saying why), a Scalar expression assigned to many elements of another UDT type, one element extracted from a Matrix into a Scalar of another type, and Matrix.power with a semiring whose result is another type; - without Numba no UDT cast compiles, so every such store raises (monkeypatched, so it runs everywhere); - Matrix.ewise_add with operands of another type than the result's (DomainMismatch, and DimensionMismatch first for other shapes), and Scalar.ewise_add with a Monoid and with a non-lifted op; - a fused store with a built-in Scalar operand, and fused stores that cannot be made (first, a user's op); - (x | y) + z of non-BOOL vectors, which reports the infix error. * Cover UDF return-type errors and the Numba error summary Tests for operator/udf.py: a UDF on a UDT that returns None, or an array with no array UDT input, or a tuple beside a UDT that is neither a record nor an array, raises UdfParseError naming the type; built-in ops on such a UDT raise their KeyError. The one-line summary of a Numba error falls back to the first non-empty line, then to a fixed sentence. * Cover the JIT layout checks for records a C struct cannot match A record with explicit offsets can have a C struct's size with a field elsewhere, as can such a record nested in one whose own offsets match, and a nested record with an array field has no C typedef. Each skips the JIT and still computes with the cfunc. * Remove branches no caller can reach in the UDT code Behavior-neutral; each guard tests a state its callers never produce: - literals._check_literal_fits: every caller passes a UDT, never None. - select._value_select_in_own_type: an int too large for any float never reaches the float branch (beside a float type it is typed as that type, or float() raises first, as in numpy). - udt_codegen._c_aligned_version is only called with records. - udt_codegen._c_expr_unary: both callers pass the field's dtype. - udt_codegen._make_jit_c_comparison_definition: _udt_c_typedef has already returned None for a UDT that is neither array nor record. - udt_codegen.set_jit_c_on_op and set_jit_c_comparison_on_op: callers check _has_jit_set first, since the setter they pass does not exist without it. * Exclude Numba-compiled bodies and unreachable guards from coverage - numba: _complex_times and _complex_truediv run only inside compiled code, as do two UDFs in test_op.py. - safety: guards no public call reaches: a record leaf lookup_dtype refuses (a void field fails to register first), a Numba error in the UDF probe (the wrapper has the same operand types), a tuple whose length differs from the record _resolve_udt_return_type matched, a C expression for an op or dtype no lifted op gives, and the frame walk in _warn_from_caller running off the stack. - only on failure: the udt_op_path fixture's demotion check, and the re-raise for a large UDT off Windows. * Drop test conditions that cannot change and a needless Numba skip - test_dtype: the JIT name test is skipped unless _has_jit_set, which needs the suitesparse backend, so it always has gb.ss. - test_ssjit: the two complex JIT tests are skipped without Numba, so their own "numba is None" skips never ran. - test_op: test_compile_codegen_helper compiles plain Python, so its Numba skip was needless; it now runs without Numba too, which covers _compile_codegen's no-Numba namespace in CI's no-Numba jobs. * Fix select on complex values with a thunk of another type Behavior fix: wrong results before, on main too (not a #591 regression). SuiteSparse:GraphBLAS reads the thunk of a value select as the input's type when the op is typed for another type, and the workaround from 69173110 converted the thunk only for bool, integer and float inputs. On an FC32 vector of [1, 2+1j, 3, nan], select(valueeq, np.complex128(1)) kept nothing (now [0]), valueeq with np.float64(3) or np.int64(3) kept nothing (now [2]), and valuene with np.complex128(2+1j) kept everything (now [0, 2, 3]); apply with the same ops was right. (Main also got a Python 3 wrong there; #591's typing of a weak int as FC32 already fixed that.) A complex input now gets the thunk in its own type when that converts exactly; otherwise (or with a NaN part) the thunk equals no element, so valueeq keeps nothing and valuene keeps everything. "Exactly" is as for float inputs: the thunk in the input's type must equal the thunk, part by part, in Python's int and float comparison, so np.int64(2**53 + 1) equals no complex64 element, as it equals no float32 one, where apply compares in complex128 and matches 2**53. The ordered value ops still raise KeyError on complex values, which have no order. A real input with a complex thunk keeps its rule: an imaginary part of 0 converts, any other equals nothing. * Compare UDTs that are neither records nor arrays in eq and ne Behavior fix. A UDT of bytes, dates or the like holds one value per element. binary.eq and binary.ne on it raised TypeError ('NoneType' object is not subscriptable), and so did Vector and Matrix isequal and ==: on main too. A Scalar's == raised it as well, where main compared the values (a #591 regression, from 69173110's Scalar == through binary.eq), as did == with a literal (b"ab"), which the literal checks read as an array UDT. On the vanilla backend a complex literal is such a UDT, so int8_vec == 1j raised the same TypeError. eq and ne now compare the two values as numpy does, as a record field of that type compares (bytes by value, NaT equal to nothing), and only two elements of the one UDT compare: any other pair raises KeyError, from _check_udt_pair, with or without Numba. So int8_vec == 1j on vanilla is a KeyError, and so is a Scalar's == with another type (another bytes UDT, or a number), where main compared the values (True or False). A literal beside such a UDT is checked as one value. A Scalar's == on such a UDT compares in numpy, as without Numba, so it needs no Numba type for the element: a float16 UDT, which Numba cannot compile, compares as on main. Vector and Matrix == on float16 still need Numba and raise its NotImplementedError, as main raised there too. * Type each product of Matrix.power for its UDT operands Behavior fix of a #591 regression: main, which typed UDT ops otherwise, computed int8 results here. power types the semiring for A, and GraphBLAS casts a built-in operand of another type to it, but not a UDT. With a semiring whose result is another type, such as plus_truediv on an int8 array UDT (float64 results), power(2) worked, while from n = 3 on the products of float64 squares with A or with each other raised a bare GrB_DOMAIN_MISMATCH from GrB_mxm. Each product of UDT operands that are not of the types the semiring was typed for now takes the semiring typed for its own operands, as any mxm of them does. power(n) into a float64 array object now equals the power of a float64 copy of A, as for built-in INT8, for every n. Unchanged: A.power(n, plus_truediv).new() makes an object of A's int8 type, and an mxm cannot cast a UDT as it stores, so it raises DomainMismatch by #591's store rule, where main computed int8 results (built-in INT8 truncates as it stores). * Select with a float64 thunk too large for FP32 without a warning Behavior fix, small: f32_vec.select(valueeq, np.float64(1e300)) gave the right result but numpy warned "overflow encountered in cast" from inside python-graphblas (an error under -W error), as the select workaround from 69173110 converted the thunk to float32. A #591 regression: main had no such conversion (it let SuiteSparse read the thunk's bytes as float32, which was wrong in another way). numpy's own f32_array == np.float64(1e300) does not warn. The conversion now ignores the overflow: the thunk is infinity in float32, which compares with every element as 1e300 does, so the selected elements are unchanged. * Refuse values that a bytes, date or duration leaf of a UDT would not keep Behavior fix. A literal that must become an element of a UDT and does not fit it by type converts, with a DeprecationWarning, only when its values convert exactly (#591's MAINT rule). The check skipped every leaf that holds no number, so whatever was given for a bytes, str, date or duration leaf counted as exact. Under a user's op, (1.5, 1) for a record of a bytes field and an int8 field became (b"1.5", 1), and 1/3 became b"0.33", with a warning that said "Its values convert exactly"; np.timedelta64(5, "ms") for a timedelta of seconds became 0 s, and a structured literal's six bytes became four, under the same warning. - main: converted silently (1.5 became b"1.5"). - 0e1ad03c: a record field converted with that warning; a bare UDT of bytes or durations raised "does not fit" (the check failed on its layout). - the commit "Compare UDTs that are neither records nor arrays in eq and ne" let a bare UDT convert with the warning too. - now: such a leaf keeps a value only if it converts back to it, so a Python number, a numpy bool, a string cut short or a duration in a coarser unit raises ValueError "does not fit". bytes or str that fit still convert, as does np.timedelta64(5000, "ms") as 5 s (with the warning, since ms to s is not a safe cast). Only a literal whose type does not fit reaches this check. A numpy number that numpy casts safely to the leaf's type fits by type: np.uint8(5) beside a 3-byte bytes UDT is b"5" without a warning, by numpy's rule. bytes or str alone beside a bare bytes UDT has no type, so numpy still cuts b"abcdef" to b"abc" there. A record literal holding a numpy number, text or a date gets no type at all and so no check; the next commit types it. * Type every leaf of a record literal, numpy numbers, text and dates too Behavior fix, in #591's literal typing (older than its last commits; main had it too). A tuple or dict literal for a record UDT got no type at all when a leaf held anything but Python numbers: a numpy number, text or a date. Then nothing checked it, and numpy converted it into the record, losing value where a Python number alone raises. In an int8 record, under a user's op, (np.float64(1.5), 1) became (1, 1) while (1.5, 1) raised "does not fit"; under a lifted op, int8_rec + (np.float64(1.5), 1) computed with (1, 1), and int8_rec == (np.float64(1.5), 2) was True. A record of a bytes and an int8 field holding (b"cd", 1) had (b"cd", 1.5) == it True, and a user's op stored (b"cd", 1.5) as (b"cd", 1); a record with a date field the same. - main and 0e1ad03c: all of these as above, silently. - now every leaf is typed. A numpy number or bool is strong, as alone or beside a built-in type: the leaf takes its dtype, promoted with any Python numbers in it, and a numpy number may stand for a whole nested record. Text or a date in a text or date field takes the field's own type when it converts there and back unchanged, so ops still pair the field with itself (rec + (b"ab", 1) is (b"abab", 2), and (np.timedelta64(5, "m"), 1) adds 300 s to a field of seconds, as on main); otherwise it keeps its own type, which does not fit: text at its own length, a numpy date or duration in its unit. Other dates (datetime, str) take the field's type. So the user's op raises "does not fit" for np.float64(1.5) or 1.5 in an int8 field next to text or a date, for text longer than its field (it was cut short) and for a date or duration in a finer unit that loses value (5 ms in seconds); np.int64(2) converts with the DeprecationWarning (exact, but not a safe cast) and np.int16(2) fits an int16 field; the lifted op gives a float64 field (2.5, 3), and + with 5 ms in a field of seconds raises KeyError (main added 0 s); and the comparisons are False. In a comparison, an int beyond its integer leaf's range now becomes infinity in that leaf only, not in the whole literal, so a text leaf beside it still pairs: rec == (b"ab", 300) is False (it raised). A text date is compared as a date in the exactness check. A sequence for an array UDT keeps its rule, the array numpy makes of it. * Refuse UDT ops on float16 elements by name; compare them in numpy for Scalar == Behavior fix. Numba has a float16 type but no code for it, so every op compiled on a UDT with a float16 leaf (a field, an array's element, or a bare float16 UDT) failed with a bare NotImplementedError: float16. - A record with a float16 field, + (1.0,) or == (1.5,): NotImplementedError on main, at 0e1ad03c and before this commit; now KeyError. - Its Scalar == (1.5,): True on main, NotImplementedError at 0e1ad03c and before; now True again. - A bare float16 UDT, Vector == or isequal: TypeError on main and at 0e1ad03c, NotImplementedError before; now KeyError. - An int8 record + (np.float16(1.5), 1) or == (np.float16(1.5), 2): main and 0e1ad03c cut 1.5 to 1 and answered (2, 3) and True, both wrong; before, KeyError ("no common type") and NotImplementedError; now KeyError. Every UDT op's _compile_udt (unary, binary, indexunary, select, indexbinary) now refuses a float16 leaf up front with the KeyError an op gives for a type it does not take: "binary.eq does not work with (...): Numba cannot compute on float16, which ... holds." A Scalar's == on a UDT with a float16 leaf compares in numpy, as for a UDT that is neither a record nor an array. A store that would cast to or from a UDT with a float16 leaf (another record or array UDT, or dup(dtype=...)), which raised the same NotImplementedError on main, at 0e1ad03c and before, now raises the DomainMismatch of a store that cannot cast, saying why: "... Numba cannot compute on float16, which ... holds." Widening float16 to float32 was not done: the result type would change, and results stored back as float16 would round. * Type a record's array field as an array UDT types its literal Behavior fix. A sequence for an array UDT is typed by the rule #591 gives it: Python numbers alone are weak, each taking the element's type when its kind holds them, and a numpy array, or a sequence holding a numpy value, is the array numpy makes of it. The same sequence for an array field of a record was typed element by element, numpy values strong and Python numbers weak beside the field, and a numpy array was read as Python numbers. So, with an int8[3] array UDT and a record with an int8[3] field: - (np.int8(1), 300, 3): the array UDT gives int64[3] and computes [2, 302, 6]; the record raised OverflowError (main and 0e1ad03c too). - np.array([1, 300, 3]): the array UDT gives int64[3]; the record raised OverflowError (0e1ad03c), and main wrapped 300 to 44 silently. - (np.float64(1.5), 2, 3): main and 0e1ad03c cut 1.5 to 1; the commit "Type every leaf of a record literal, numpy numbers, text and dates too" already gave a float64 field. A record's array field now takes the array UDT's rule: an array, or a sequence holding a numpy number or bool, is the array numpy makes of it, broadcast to the field's shape. So the record gives an int64[3] field and [2, 302, 6], as the array UDT does. Python numbers alone, and a numpy number for the whole field, are typed as before. A comparison keeps a Python int beyond the element's range exact, as it does everywhere (int8_vec == 300 is False), also inside such an array: the int is infinity in its place before numpy makes the array. So rec == ((np.int8(1), 2**70, 3), 4) and int8_array_udt == (np.int8(1), 2**70, 3) are False and != True. The array UDT raised numpy's OverflowError there (older than this commit), and the record field, False before, would have raised it under the array rule alone. A user's op with such a literal still raises OverflowError. * Refuse == between text, dates and numbers with eq's KeyError Behavior fix. binary.eq and ne compared each pair of record leaves with Numba, which cannot compare bytes with a number, so such a pair raised UdfParseError carrying Numba's typing error: - a record with a bytes and a float64 field, eq with a record of two float64 fields: GrB_DOMAIN_MISMATCH on main, UdfParseError at 0e1ad03c and before this commit; - the same record == 1 or != (1.0, 1.0), and its Scalar == (1.0, 1.0): main converted the number into the bytes field (1 became b"1") and compared the converted value; UdfParseError at 0e1ad03c and before; - a nested record == (b"x", np.float64(1)), whose number stands for a nested record holding a bytes field: False on main and at 0e1ad03c, UdfParseError before this commit. Text, a date, a duration and a number now compare only with their own kind; any other pair of leaves raises the KeyError binary.eq gives for a pair it does not take: "binary.eq does not work with (...): elements of |S4 and float64 do not compare." Scalar == raises it with or without Numba. This follows the rules #591 already has: plus finds no common type for such leaves (a KeyError), and a UDT that is neither a record nor an array pairs only with its own type. Answering False instead (a number for a text leaf can never be equal) would turn the record pair, an error since main, into an answer, and would differ from main anyway where main compared b"1" with 1 as equal. * Test the text-date and duration-for-a-date branches of record literal typing Tests only. Two branches of the record literal typing had no row: - a date given as text shorter than numpy's own date string ("1970-01-01" for a field of seconds), which _converts_back compares as a date: with a number that converts exactly, the literal converts with the DeprecationWarning, and without the branch it raised "does not fit"; - a numpy duration given for a date field, which keeps its own type: numpy converts np.timedelta64(5, "s") to a date by value, so without the guard it took the date field's type and failed in numpy's own conversion; with it, it raises "does not fit". * Let a UDF return an operand of a UDT that is neither a record nor an array Behavior fix. A user's op on a UDT of bytes, str or the like that returns an operand as it is (lambda x, y: x) failed to compile: Numba types the element as a char sequence, which no DataType is registered for, so the return type was not found. - main: UdfParseError "UDT UDF returned an unsupported type [char x 8]". - 0e1ad03c: TypeError 'NoneType' object is not subscriptable (the literal beside it was read as an array UDT). - before this commit: the UdfParseError again. - now: the return takes the operand's UDT when its Numba type is that UDT's, so the op computes; [b"abc"] for x, the literal for y. * Keep text that a bare text UDT would cut short out of it, and compare text widths Behavior fix. Text given beside a UDT of bytes or str (not a record) had no type, so numpy cut it to the UDT's width: with an S4 element b"abcd", b"abcdef" became b"abcd". - v == b"abcdef": TypeError on main and at 0e1ad03c; True before this commit; now False (!= is True). - Scalar == and Scalar.isequal with b"abcdef": True on main (isequal True at 0e1ad03c too), now False. - a user's op with b"abcdef": main and the commits before this one cut it silently (0e1ad03c raised TypeError); now ValueError "does not fit". Such text is now typed as a record's text field is: the UDT itself when it converts into it and back unchanged (b"ab", "ab" into S4, so bare_s4 == b"ab" works as before), else its own type at its own length, which does not fit. For eq and ne, that literal is text of the same kind at another width, so this commit also loosens, there and only there, the pairing rule of "Compare UDTs that are neither records nor arrays in eq and ne": two UDTs of text of the same kind (bytes with bytes, str with str) at different widths now pair under eq and ne and compare as text, as numpy compares them and as a record's S4 and S2 fields already do. An S4 and an S6 Vector compare element by element (they raised KeyError before; TypeError on main and at 0e1ad03c). Bytes against str, a number, float16 or any other kind still raise the pairing KeyError, as does every other op on such a pair. Stores are unchanged: v[0] = b"abcdef" still stores b"abcd", as every literal store converts by numpy's cast (int8_vec[0] = 1.5 stores 1). * Keep a NaT duration NaT in another unit on numpy before 2.3 Behavior fix. numpy before 2.3 converts a NaT timedelta64 scalar into another unit as the integer NaT is stored as: NaT ms assigned into timedelta64[s] is 9223372036854774 s, and NaT minutes is 0 s. An array of durations (astype) and a date convert NaT correctly, and numpy 2.3 and later convert it correctly every way. On numpy 2.2 (numpy 1.26 converts NaT the same way), with a UDT of timedelta64[s] or a record with a timedelta64[s] field: - Scalar.from_value, v[0] = ..., Scalar.value = ..., Vector.from_coo, and the Scalar a literal becomes, given NaT ms or NaT minutes: stored 9223372036854774 s or 0 s, as on main; now NaT. - A NaT ms literal beside the bare UDT: ValueError "does not fit", since it converted to a number and so not exactly; now it converts with the DeprecationWarning, as on numpy 2.4. - (NaT ms, 1) beside the record: typed as timedelta64[ms], so binary.plus raised KeyError "no common type"; now it takes the field's type, as on numpy 2.4. The conversion into an element, and the checks of whether a literal converts exactly, now hand numpy a NaT of no unit in place of each NaT duration (alone, or in a tuple or list), which every numpy converts to NaT in any unit. numpy 2.4 gives the same results before and after. * Keep the JIT layout test's record short enough to name on SuiteSparse 8 SuiteSparse before 9 has no GrB_NAME setter, so registration names a UDT by its numpy repr, which must fit in 128 bytes, and warns when it does not; pytest makes the warning an error. test_record_layouts_c_cannot_declare_skip_jit registered its record with an array field with align=True, which gives the dict form of the repr: 143 bytes. Its float64 fields have the same offsets packed or aligned, so it is now registered packed (a 71-byte repr); a C struct still cannot declare it, and the test runs the same lines. Every other UDT the tests register with a repr over 128 bytes is in a test that ignores that warning. * Convert a NaT duration in its leaf's unit, not as a NaT of no unit "Keep a NaT duration NaT in another unit on numpy before 2.3" handed numpy a NaT of no unit, np.timedelta64("NaT"), in place of each NaT duration. numpy 2.5 deprecates that generic unit ("The 'generic' unit for NumPy timedelta is deprecated"), so on numpy 2.5 every store or literal of a NaT duration into a duration UDT or field warned with a DeprecationWarning, which the test suite's filterwarnings makes an error: test_udt_nat_duration_in_another_unit_stays_nat, test_udt_sequence_and_array_literals_type_like_numbers and test_udt_literal_converts_only_when_exact failed. Each NaT duration is now converted into the unit of the leaf it fills by converting it as an array (astype), which keeps NaT on every numpy, before numpy assigns it. The value is walked beside the dtype: a tuple of a record's length as its fields, a list (or a tuple for any other dtype) as a sequence of elements, and an array UDT by its element type. Nothing makes a generic unit. numpy 2.5.4: DeprecationWarning before, NaT with no warning now. numpy 1.26, 2.2 and 2.4: NaT in the target unit before and after. New rows: a nested record's duration field, and an array UDT of durations given a list, where Numba is absent (Numba cannot type an array of durations, so registering one fails with it). * Skip the JIT part of the one-name UDT layouts test without a C compiler test_udt_layouts_under_one_name_get_their_own_jit_c_names set jit_c_control to 'on' and compiled kernels for both layouts, so on a host whose JIT cannot compile (no C compiler, or a broken one) it failed with JitError instead of skipping. CI always installs a compiler, so CI never saw it. It now asks _enable_jit_for_udt, which probes with a real compile once per process, and skips the kernel part when the answer is no. The C name checks and the register_new check need no compiler, so they come first and run everywhere. * Expand an iso array UDT's value without an extra axis in exports Behavior fix. SuiteSparse exports the iso value of an array UDT with shape (1, *subarray shape), and normalize_values broadcast it to shape + values.shape, keeping that axis of length 1. With the suitesparse backend, for an iso Vector or Matrix of an array UDT such as INT64[4] (as v[:] = [0, 1, 2, 3] makes): - Vector.to_dense: TypeError "unsupported operand type(s) for +: 'int' and 'tuple'" on main and before this commit (it passed its size as an int); now an array of shape (size, 4). - Matrix.to_dense: shape (nrows, ncols, 1, 4); now (nrows, ncols, 4). - Matrix.to_csr, to_csc, to_dcsr and to_dcsc: values of shape (nvals, 1, 4); now (nvals, 4). - Matrix.to_dicts, which uses to_dcsr or to_dcsc: each value as [[0, 1, 2, 3]]; now [0, 1, 2, 3]. Vector.to_coo, Matrix.to_coo, the vanilla backend and records were already right. normalize_values now broadcasts to shape + values.shape[1:], and Vector.to_dense passes (size,). CI found it through #652's test_udt_mixed_jit_c_name_is_its_own (17 jobs, e.g. job 122633175811): SuiteSparse before 10.5 makes a vector built from equal values iso, so that test's constant inputs gave an iso result, and to_dense raised. With 10.5 the vectors are not iso. * Refuse a zero-size UDT, so a str literal raises TypeError on SuiteSparse 7 Behavior fix. register_anonymous passed a dtype's itemsize to GrB_Type_new without a check. SuiteSparse 8 and later refuse a size of 0, but SuiteSparse 7 takes it as 1. lookup_dtype auto-registers what it does not know, so on SuiteSparse 7 a str became the UDT "<U0", which holds nothing. - register_anonymous of str, bytes, np.dtype("U"), an empty record or a subarray of length 0: InvalidValue with no message on SuiteSparse 8 and later, a UDT on 7; now ValueError "dtype must not have a size of 0" on every version. - lookup_dtype(str): ValueError "Unknown dtype" on 8 and later, the "<U0" UDT on 7; now ValueError "Unknown dtype" on every version. - Scalar.from_value("x"), Scalar + "x" and v.apply(binary.plus, right="x"): on SuiteSparse 7, ValueError "got more than 0 values to unpack" from cffi; now the TypeError they raise on 8 and later. CI found it in wave 2 through #654's test_bad_ewise case (s1, "x"), which expects the TypeError, in its one job with python-suitesparse-graphblas 7.4.2 (122632718294). * Note that SuiteSparse:GraphBLAS 10.5.2 fixed select with a value op SuiteSparse:GraphBLAS 10.5.2 fixed the bug _value_select_in_own_type works around (DrTimothyAldenDavis/GraphBLAS#463): select with a VALUE* op of another type than the input's now casts both to the op's type. Its MAINT note said to delete the function once every supported version has the fix, but that would change two results that test_select_value_thunk_of_another_type pins: with the function bypassed on 10.5.2, an INT64 thunk of 2**53 + 1 beside FP32 or FC32 values of 2**53 rounds to 2**53 in the op's type and keeps that element. The other 61 cases of the test match. The note now says so, and gives the versions with the bug as 8.0.2 through 10.5.1. Comments only. * Say which array Numba cannot type when registering a UDT Behavior change. Registering a UDT that holds an array of bool, datetime64, timedelta64, str or bytes (as the UDT, or as a field of a record) raised Numba's own error, which does not say what to change: dtypes.register_anonymous(np.dtype((np.bool_, (2,)))) # was AttributeError: 'Boolean' object has no attribute 'bitwidth' # now ValueError: Numba, which python-graphblas uses for UDTs, cannot # type an array of bool, as in ('?', (2,)); use an array of uint8 # instead The ValueError, the type of the other errors register_anonymous raises for a dtype it refuses, is chained from Numba's AttributeError. The np_type of dtypes.ss.register_new gets the same error. Without Numba these dtypes still register, as before. Numba now types the dtype before GrB_Type_new, so a dtype it cannot type no longer leaves an unused GrB_Type behind. lookup_dtype, which registers a numpy dtype it has not seen, still says "Unknown dtype" for these, as for any dtype it cannot register. * Type an IndexBinaryOp bound to theta when it runs, with theta of its own type Binding theta to an IndexBinaryOp of no type, op(theta), took the op's type from theta alone, and the inputs were cast to it. Now op(theta) returns a BinaryOp whose type is chosen each time it runs, as any other BinaryOp's is. Theta is typed by its value when it is given, as apply types an IndexUnaryOp's thunk: a Python int is INT64, a float FP64 and a bool BOOL, and a numpy scalar or a Scalar keeps its type. A numpy array is typed as its array layout, as other array literals are. The default theta, False, changes no type. An op registered without is_udt runs at the type that holds both inputs and theta. One with is_udt=True is compiled for the types of x, y and theta as they are, and SuiteSparse gets theta's type as the op's theta_type, which GxB_IndexBinaryOp_new takes apart from x's and y's. So theta is never converted. The compiled ops are cached per (x, y, theta) types, and op[X, Y, T] names all three; op[X, Y] and op[T] give theta y's type, as before. A Semiring built from op(theta) is polymorphic in the same way. op[T](theta) and op(theta, dtype=T) still bind at T. IndexBinaryOp has not been released, so nothing is deprecated. Behavior changes (before -> after): - A.ewise_mult(B, op()) on INT64: a BOOL op, so every value became True -> INT64, the values op computes. - u.ewise_mult(w, op(2)) on FP64: computed in INT64 (1.5 + 2 * 2.25 gave 5) -> FP64 (6.0). op(np.int8(2)): computed in INT8, where 300.0 became 127 and wrapped -> FP64. - A Semiring from op(0.5) with INT64, FP32 or INT8 matrices: KeyError -> computed in FP64, with or without is_udt. - The SuiteSparse user guide's argmax (an is_udt multiplier that makes the pair (k, A(i, k)), theta unused) on a matrix of pairs: iop(), iop(False), iop(0) and iop(0.0) were ops on BOOL, INT64 or FP64 values, which failed to compile for the pairs -> all four run, compiled for (pair, BOOL, BOOL), (pair, BOOL, INT64) and (pair, BOOL, FP64). - An is_udt op on INT64 data: op(0.5) or an FP32 Scalar theta made the op FP64 or FP32 and cast the inputs to it -> the inputs stay INT64, theta keeps its type, and Numba types the result (FP64 for x + y + theta). - An is_udt op on int32[3] data: op(np.array([10, 20, 30], np.int32)) raised "Cannot infer a dtype for theta" -> theta is an int32[3] value; an int64 array is an int64[3] one. op(5) gave GrB_DOMAIN_MISMATCH -> theta is an INT64. op((10, 20, 30)) still raises TypeError, now saying to give theta as a Scalar of its UDT; an op without is_udt raises KeyError for an array theta, as for a UDT Scalar. - A JIT op (indexbinary.ss.register_new) is registered for its x, y and theta types. op(theta) took the one registered at theta's type T for x, y and theta, and cast the inputs to T -> it takes the one for the inputs' types and theta's, without casting, as a JIT IndexUnaryOp takes its thunk: with an op registered at FP64, op(10.0) on FP64 data runs as before, op(10.0) on FP32 data ran in FP64 -> KeyError naming (FP32, FP32, FP64), and op(10) raised KeyError when bound -> when it runs. op.types is keyed (x, y, theta), so registering the same x and theta types for another y no longer raises "already defined", and op["FP64", "FP32"] is x and y, no longer x and theta: op["FP64", "FP64", "FP32"] is the op with an FP32 theta. - A bound op pickles with all its types: op["INT64", "FP64"](1) came back as op["INT64"](1), whose y is INT64. - op(theta).type: the bound type -> AttributeError, as for any untyped op; op(theta)[T] is op[T](theta). repr(op(2)): binary.f[INT64] -> indexbinary.f(2). op(theta)(A & B) works, as for other BinaryOps. - A Scalar theta is copied when given, so a later change to it does not reach the op, as when the op bound it at once. find_opclass now reads an opclass attribute on an OpBase subclass, so op(theta) is still a BinaryOp there. The user guide's examples drop the op[float](0.5) that hid the old typing. The comments on IndexUnaryOp's and SelectOp's UDT caches now say they are keyed by (dtype, thunk_type). * Refuse a record literal whose array field part does not broadcast to the field Behavior fix. Where a literal for a record UDT is typed (a lifted op such as +, ==, a user's op, an IndexUnaryOp thunk, an ewise_union default, and == on a Scalar), a part for an array field must now broadcast to the field's shape, or ValueError names both shapes. An array UDT's element takes the shape it broadcasts to with a literal, as numpy does, so INT8[3] plus np.array([[1, 300, 3]]) is INT64[1, 3]; that is unchanged. A record's field keeps its shape, so for the record the typing gave up, and the literal was converted into the field as numpy assigns it: the leading axis was dropped and 300 wrapped to 44, silently, even beside a user's op, which refuses a literal that does not fit. Stores (r[0] = ...) still convert as numpy assigns, as they do for an array UDT. With r a Vector of a record with an int8[3] field a and an int64 field b, holding ([1, 2, 3], 0) (before -> after): - r + (np.array([[1, 300, 3]]), 0): ([2, 46, 6], 0) -> ValueError "array([[ 1, 300, 3]]) has shape (1, 3), which does not fit field 'a' of R, of shape (3,): ...". - r.apply(user_op, right=(np.array([[1, 300, 3]]), 0)), with user_op returning y: ([1, 44, 3], 0) -> the same ValueError. - r == (np.array([[1, 300, 3]]), 0): False -> the same ValueError; with np.array([[1, 2, 3]], np.int8) it was True. - r + ([[1, 300, 3]], 0), and the other paths with it: numpy's "setting an array element with a sequence" ValueError -> this ValueError. - r + ([1, 2], 0): numpy's "could not broadcast input array from shape (2,) into shape (3,)" -> this ValueError. - r[0] = (np.array([[1, 300, 3]]), 0): ([1, 44, 3], 0), unchanged. * Cover more literals a UDT refuses and the Numba error of a lifted op Literal typing had arms no test reached: a comparison with a record literal of the wrong length (alone and nested beside an int beyond its leaf), a part holding numpy values that numpy cannot make an array of or that is not numbers, and text a bytes field cannot encode. An array of records now pins the UdfParseError a lifted op gives when Numba cannot compile it, and the NoJITWarning is checked to stay silent for a pair it warned about. Two arms of the monoid identity are excluded from coverage: the binary op compiles first, and it refuses what would reach them. * Give each type signature of a JIT IndexBinaryOp its own C name indexbinary.ss.register_new compiled every type signature of one op under the op's own name. SuiteSparse finds a compiled kernel in memory by its op's name and the types in the kernel's encoding, and an ewise or mxm kernel's encoding has x's, y's and z's types but not theta's. So two signatures that differ only in theta's type ran whichever kernel was compiled first. Now each signature's C function is compiled as name_N_x_M_y_theta, with N and M the lengths of the x and y parts (built-in types by name, UDTs by their C name): the C definition is the user's with the function renamed. A name SuiteSparse would cut at 127 characters is the op name's first 100 and a digest of the whole. The op's jit_c_name and jit_c_source show the C name and the source SuiteSparse compiles; jit_c_definition is still the definition given. JIT IndexUnaryOps and SelectOps are not affected: the thunk's type is in their kernels' encodings. Behavior change (before -> after): with add_theta registered at FP64 for x, y and theta, and then at an FP32 theta with (*z) = (*x) + (*y) + 2 * (double)(*theta), the FP32 signature on FP64 data with theta 10 gave [2, 2, 4, 0] (the FP64 kernel, reading the float as a double) -> [22, 22, 24, 20]. op.jit_c_name: None -> add_theta_4_FP64_4_FP64_FP32. * Name each op registered from C by its full name and definition The ss.register_new functions of unary, binary, indexunary (and select) gave SuiteSparse the op's last name as its C name, and indexbinary's the last name and types. SuiteSparse finds a compiled kernel in memory by its op's name, and checks an op's definition only when it loads a kernel from disk. So, in one process, an op registered again under its name with another definition (after delattr, as when a notebook cell is run again), or an op of the same last name in another namespace (ss.f and ss.nested.f), ran the kernel compiled first. Now every op registered from C source is compiled under a C name made of its last name, the types for an IndexBinaryOp (whose theta type is not in its kernels' encoding), and a 12-hex digest of these with the full dotted name and the definition: f_3a1c90e2d4b7, or add_theta_4_FP64_4_FP64_4_FP32_... for an IndexBinaryOp. A name SuiteSparse would cut at 127 characters is the last name's first 100 characters and a 16-hex digest. The function is renamed in the definition wherever it is named, except as a struct member (after "." or "->"). The same name and definition give the same C name, so a kernel is compiled once for them. jit_c_name and jit_c_source, None for these ops before (except IndexBinaryOp, from the previous commit), show the C name and the renamed source; jit_c_definition is still the definition given. The helper is ss._utils._jit_c_source. Behavior changes (before -> after), in one process: - binary.ss.register_new("f", x + y) and ("nested.f", x - y), both FP64: nested.f computed x + y -> x - y. - unary.ss.register_new("u", x * x), delattr(unary.ss, "u"), then register_new("u", x + 10): u computed x * x -> x + 10. * Refuse an empty Scalar theta for an IndexBinaryOp SuiteSparse binds theta without checking that it holds a value: it reads the Scalar's value array. Binding an empty Scalar crashed the interpreter, and op(s) with no type, which binds a copy of s, ran with theta 0. Both now raise EmptyObject, as apply does for an empty IndexUnaryOp thunk, in every bind: typed or not, is_udt or not, a JIT op's too. Behavior change (before -> after), with s = Scalar(int): - op[int](s) and op(s, dtype=int): segmentation fault -> EmptyObject. - op(s): an op that ran with theta 0 -> EmptyObject, at the bind. With s a C scalar (is_cscalar=True): segmentation fault -> EmptyObject. * Leave a dtypes.ss.register_new name free when the registration fails register_new recorded the name as a JIT C type name as soon as SuiteSparse made the type, before the numpy and Numba types were made. When one of those raised (an np_type with an array of bool, which Numba cannot type), the name stayed reserved, and a correct second call raised "name for dtype is unavailable". Main allowed the second call; the check against C type names came with "Give each UDT layout its own JIT C name". The name is now recorded after the type is registered. The type made by the failed call is never used, so no JIT kernel knows its name. Behavior change (before -> after), with d the typedef of pt_t: - register_new("pt_t", d, np_type=np.dtype((np.bool_, (2,)))) raises ValueError (unchanged), then register_new("pt_t", d, np_type=np.dtype([("x", "f8"), ("y", "f8")])): ValueError "'pt_t' name for dtype is unavailable" -> registers dtypes.ss.pt_t. * Type a Python date or duration in a record literal as numpy does A record literal's date or duration field takes the field's type only when its value converts into the field and back unchanged; otherwise it keeps its own type, so == compares it whole. A numpy date did this already, but a Python datetime.date, datetime.datetime or datetime.timedelta, or a date given as text, always took the field's type. So it was cut to the field's unit before == compared it: noon compared equal to its day in a field of days, and 5 s and 1 us equal to 5 s in a field of seconds. Main gave the same wrong answers (it gave them for numpy dates too). Now such a value is first made the numpy value it stands for (a datetime in microseconds, a timedelta in microseconds, a date as text in the unit it is written to), and is typed as that numpy value is. Behavior change (before -> after), with v a Vector of a record of an M8[D] and an m8[s] field holding (2020-01-01, 5 s), and s = v[0].new(): - v == (datetime(2020, 1, 1, 12), np.timedelta64(5, "s")), and with "2020-01-01T12:00" for the date: True -> False; the same for s ==. - v == (np.datetime64("2020-01-01"), timedelta(seconds=5, microseconds=1)): True -> False. - v.apply(user_op, right=(datetime(2020, 1, 1, 12), np.timedelta64(5, "s"))): stored the day, silently -> ValueError "does not fit", as for np.datetime64("2020-01-01T12:00"). - v + (datetime(2020, 1, 1, 12), ...): UdfParseError from Numba -> KeyError naming datetime64[D] and datetime64[us]. - A datetime at midnight, a date, and text such as "2020-01-01" still compare equal and fit the field. * Compare a UDT of dates or durations with one in another unit as numpy does A UDT that is a bare datetime64 or timedelta64 (not a record or an array) converted a date or duration literal into its own unit before == compared it, so noon compared equal to its day in a UDT of days. Main raised TypeError here for a Vector, and gave True for a Scalar; "Compare UDTs that are neither records nor arrays in eq and ne" made the Vector give True too. A numpy duration in another unit was a KeyError instead, even when it was the same duration. Now such a literal is typed as in a record's date field: the UDT itself when it converts into the UDT and back unchanged, else numpy's type for it (a datetime is in microseconds, a timedelta too, a date as text in the unit it is written to). Under eq and ne, a UDT of dates or durations pairs with one in another unit, as text of one kind pairs at another width, and they compare as numpy compares them. Behavior change (before -> after), with day a Vector of an M8[D] UDT holding 2020-01-01 and five one of an m8[s] UDT holding 5 s: - day == np.datetime64("2020-01-01T12:00"), datetime(2020, 1, 1, 12) or "2020-01-01T12:00": True -> False; != the opposite; the same for day[0].new() ==. - five == np.timedelta64(5000, "ms"): KeyError -> True; 5001 ms: KeyError -> False. five == timedelta(seconds=5, microseconds=1): True -> False. - binary.eq(five & a Vector of m8[ms] holding 5000 ms), and five.isequal of it: KeyError -> True. - day.apply(user_op, right=datetime(2020, 1, 1, 12)): stored the day, silently -> ValueError "does not fit". - five.apply(user_op, right=np.timedelta64(5000, "ms")): converted with a DeprecationWarning -> converts with none, as in a record's field. - day == timedelta(5): numpy's ValueError -> binary.eq's KeyError. * Type a literal beside a typed op in the op's own input type A literal operand beside a UDT became an element of that UDT even when the op was typed and takes that operand in another type: op[A, INT64] takes its thunk, or its second input, as an INT64. Apply and select pass a UDT thunk or operand to SuiteSparse as a pointer to its bytes, so the op read the bytes of the A element as an int64: silently wrong answers. Main did the same. A Scalar of the op's type gave the right answer. Now a typed op's own input type on the literal's side decides: its first input type for a left operand, else its second input or thunk type. GraphBLAS converts a value to a built-in input type, so the literal is typed as beside a built-in typed op (its own type: 2.5 for an INT64 thunk is converted to 2, as for an op typed on built-in types); a UDT input type takes only a literal that fits it. _literal_operand takes left=True from apply's left operand and from ewise_union's left default. Behavior change (before -> after), with v a Vector of a record A of two float64 fields holding (1.0, 2.0) and (3.0, 4.0), and user ops typed [A, INT64] (or [INT64, A]) that compare or add field a with the other: - v.apply(iu[A, INT64], 2): [False, False] -> [False, True], as with Scalar.from_value(2); the same for thunks 2.5 and np.int8(2). - v.select(se[A, INT64], 2), on a Vector or Matrix: empty -> index 1. - v.apply(f[A, INT64], right=2) and v.apply(g[INT64, A], left=2): about 4.6e18 for each element -> [3.0, 5.0]. - With B a record of two int64 fields: v.apply(h[A, B], right=(2, 3)): about 4.6e18 -> [3.0, 5.0]; v.apply(iu_rec[A, B], (2, 0)): [False, False] -> [False, True], and with (2.5, 0): [False, False] -> ValueError "does not fit". - Scalar(A).ewise_mult(2, f[A, INT64]): GraphBLAS's DomainMismatch -> 3.0. * Accumulate into a UDT Scalar as into a built-in Scalar The accumulator is typed on the object's dtype, and for a built-in Scalar GraphBLAS converts the value to that type before it accumulates: an INT8 Scalar of -1 with s += 0.5 (Scalar.__iadd__ is s(binary.plus) << 0.5) adds 0 and stays -1, as v(accum=binary.plus) << 0.5 does for a Vector of either kind. A UDT Scalar used the untyped op instead (_scalar_accum_op), so it computed -1 + 0.5 in FP64 and truncated the sum to 0. That was meant to match a Vector's v += 0.5, but that is v << v + 0.5, not an accumulate. Main gave the built-in answer. Now a UDT Scalar accumulates with the typed op, and the value is converted to the Scalar's type first, as storing it would convert it: a literal by Scalar.from_value, a Scalar or an element of another UDT by the store cast. A built-in Scalar into a UDT Scalar still raises DomainMismatch, as a store of it does, into a Vector too. Behavior change (before -> after), with s an INT8[3] UDT Scalar of [-1, -1, -1]: - s += 0.5 and s(accum=binary.plus) << 0.5: [0, 0, 0] -> [-1, -1, -1] (an INT8 Scalar gives -1; a Vector of the UDT gives [-1, -1, -1]). - s(accum=binary.plus) << (0.5, 1.5, 2.5): [0, 0, 1] -> [-1, 0, 1], as for a Vector of the UDT. - s(accum=binary.plus) << v[1], with v a Vector of an FP64[3] UDT holding [0.5, 1.5, 2.5]: [0, 0, 1] -> [-1, 0, 1], as with the same value in a Scalar of that UDT, which was cast first already. * Pair an array literal with an array UDT when their shapes broadcast udt.rst says a numpy array beside a UDT pairs as an array UDT of its own shape, and array UDTs pair when their shapes broadcast. A numpy array paired only when it broadcast to the UDT's shape, or had that shape apart from leading 1s. Any other array, even one whose shape broadcasts with the UDT's, was converted into the UDT, where numpy raised "could not broadcast", while the same values as a UDT operand worked. Main raised too (a KeyError for the UDT pair). Now such an array is typed in its own shape and pairs as an array UDT operand does; the result has the shape numpy's would. An array that broadcasts to the UDT's shape is typed in the UDT's shape, as before, which a literal that must become an element needs. Behavior change (before -> after), with v a Vector of an INT8[3] UDT holding [1, 2, 3] and u one of an INT8[1] UDT holding [1]: - v + np.array([[1, 2, 3], [4, 5, 6]]), and apply(binary.plus) with it on either side: ValueError "could not broadcast input array from shape (2,3) into shape (3,)" -> INT64[2, 3] [[2, 4, 6], [5, 7, 9]], as with the same values in an INT64[2, 3] UDT. - v == np.array([[1, 2, 3], [1, 2, 3]]), on a Vector or a Scalar: the same ValueError -> True. - u + np.array([1, 2, 3]), and u + (1, 2, 3): ValueError -> INT64[3] [2, 3, 4]. - v + [[1, 2, 3], [4, 5, 6]]: numpy's "setting an array element with a sequence" ValueError -> INT64[2, 3], as for the array. - v.ewise_union(v, binary.plus, a 2-by-3 array, 0): numpy's ValueError -> ValueError "does not fit". * Give IndexBinaryOp is_positional, so dtype in op and op[dtype] work OpBase.__getitem__ reads is_positional for a type the op does not have, and IndexBinaryOp had none, so asking an IndexBinaryOp (registered without is_udt) for a type it does not take raised AttributeError. Main did the same; IndexBinaryOp has not been released. It is now False, as on a bound IndexBinaryOp and every other op class. Behavior change (before -> after), with op an IndexBinaryOp of x + y + theta and R a record UDT: - R in op: AttributeError -> False, as for an IndexUnaryOp. - op[R] and op(Scalar.from_value((1.0, 2), R), dtype=R): AttributeError -> KeyError "... does not work with R". * Refuse a bound IndexBinaryOp in reduce with the usual TypeError reduce turns a BinaryOp that is part of a Monoid into that Monoid, and refuses any other BinaryOp with a TypeError naming it. It reads the op's monoid for that, and the error message its parent's; neither a bound IndexBinaryOp nor IndexBinaryOp had the attribute, so reduce raised AttributeError, for ibo[T](theta) on main and for the new untyped ibo(theta) too. Both now have monoid = None, as a BinaryOp of no Monoid has. IndexBinaryOp has not been released. Behavior change (before -> after), with v an INT64 Vector and ibo an IndexBinaryOp of x + y + theta: - v.reduce(ibo(1)) and v.reduce(ibo[int](1)): AttributeError "'_BoundIndexBinaryOp' object has no attribute 'monoid'" -> TypeError "Bad type for argument `op` in Vector.reduce(...) ... is not known to be part of a Monoid", as for binary.minus. - A.reduce_rowwise(ibo(1)): AttributeError -> NotImplementedException, as for binary.minus.
f659be8 to
159e411
Compare
159e411 to
9235727
Compare
9db47f8 to
b562062
Compare
45f8a75 to
db2f16b
Compare
Behavior fix. Since 2023.1.0, `x & s` and `x | s`, with a literal x and
a Scalar s, built the expression with s on the left and x on the right,
so the op applied to it ran with its operands swapped, and every op that
does not commute gave wrong values, silently:
s = Scalar.from_value(5)
binary.minus(1 & s) # was 4, now -4
binary.truediv(1 | s) # was 5.0, now 0.2
binary.first(1 & s) # was 5, now 1
binary.lt(1 & s) # was False, now True
So did rminus, cdiv, rdiv, pow, the other comparisons and
is-comparisons, and the rest of the ops that do not commute. So did the
defaults of an ewise_union, each of which went with the other operand:
with an empty Scalar e, `binary.minus(1 | e, left_default=0,
right_default=10)` was -1 and is now -9. And so did chains:
`binary.minus(1 & (2 & s))` was 2 and is now 4.
The expression now keeps the literal where it was written, as it is, and
the op applied to it types it: Scalar's ewise_mult, ewise_add and
ewise_union take a literal on the left as they take one on the right,
typed beside the Scalar by literals._literal_operand. A Python number is
weak, as beside `-`: `binary.minus(2 & int8_s)` is INT8 and
`binary.minus(0.5 & int8_s)` is FP64, as `2 - int8_s` and `0.5 - int8_s`
are. Beside a UDT, a lifted op such as minus types the literal as numpy
would and `first` makes it an element of the UDT.
How a literal is typed depends on the op (300 beside INT8 is an
OverflowError for `-` and is compared as it is by `<`), so the literal
is not converted when `&` or `|` builds the expression. Converting it
then, with its own dtype, would get the typing wrong: the 2 of
`int8_s & 2` would be INT64 under every op, where `int8_s - 2` keeps it
INT8, and `0.5 & int8_udt_s` would cut 0.5 to 0 in the UDT.
`op(x & y)` now asks the expression to call its left operand's method
(InfixExprBase._call_left), so a Scalar expression can pass a literal
there.
Two more errors with a literal operand are fixed:
- Computing a BOOL expression with a literal by itself, as
`(s & True).new()` and `(False | s).new()` do, raised AttributeError:
'bool' object has no attribute 'dtype'. It now types the literal for
LAND or LOR and computes, or raises the usual TypeError for a dtype
other than BOOL.
- `s & object()` gave an expression that failed only when an op was
applied; it now raises TypeError at once, as `s + object()` does. The
check types the literal as `==` would and drops it. Beside an infix
expression, as in `(s & t) & object()`, it raises when the op is
applied.
Behavior fix. Since 2023.1.0, a Scalar's reflected operators (`lit - s`,
`lit / s`, `lit ** s` and so on, with a literal on the left) computed
the op's commutes_to with the Scalar first, rminus(s, lit). That fails
where no such op exists:
1 - udt_s, 1 / udt_s, 1 // udt_s, (1, 2) - udt_s
# KeyError: 'rminus does not work with ...'; no reflected op
# takes a UDT, though `udt_s - 1` and `1 - udt_vector` work
7 % s # TypeError for every dtype: numpy.mod has no reflected op
2 ** s # TypeError without Numba: rpow is a Numba function
Now the literal becomes a Scalar on the left, typed beside s as in
`binary.minus(1 & s)`, and the op itself is applied: `1 - udt_s` is
[-4, -6] for udt_s = [5, 7], and `7 % int8_s` is 2 for 5.
`lit ** s` now gives what `binary.pow(lit, s)` and `lit ** vector` give.
That changes four kinds of result, where rpow, a Numba function,
differed:
- an integer power out of range saturates, as GraphBLAS's pow does,
where rpow wrapped: `3 ** int8_s` for 5 was -13 and is now 127, and
`0 ** int8_s` for -3 was 0 and is now 127;
- `False ** bool_s` is BOOL, not INT8;
- `0 ** complex_s` is 0, not nan (for 1+2j), and other complex powers
may differ in the last bit;
- a Python int beyond INT64 beside a BOOL Scalar, as in
`(2**64 - 1) ** bool_s`, raises OverflowError, as
`bool_s ** (2**64 - 1)` and `(2**64 - 1) ** bool_vector` do, where it
gave a UINT64.
Nothing else changes: for `+ - * / // % ** ^`, nine Scalars (INT8,
INT64, UINT8, FP32, FP64, BOOL, empty) and eight literals, every result
the previous code gave is the same but those of `**`; and for the seven
that are not empty, `lit OP s` now matches `binary.op(lit, s)` for each
of `+ - * / // % **` (it did not in 66 of 392 cases).
Behavior fix. Since 2023.1.0, showing a Scalar `&` or `|` expression
raised in two cases:
repr(s & 1), repr(1 | s), repr(t & True), and their _repr_html_
# AttributeError: 'int' object has no attribute 'name'
repr(s & s) # s not BOOL, with autocompute on (the default)
# TypeError: Bad dtypes for `x & y`! Automatic computation ...
A literal operand now shows its value, as an unnamed Scalar does in an
expression, and its own type where a dtype goes, since it has no dtype
until an op types it:
gb.ScalarEwiseMultExpr left_dtype right_dtype
s_1 & 1 INT8 int
The text repr now computes the expression to show its result only where
the HTML repr, and the reprs of Vector and Matrix infix expressions,
do: for `x @ y`, and for `x & y` and `x | y` of BOOL operands. A literal
counts as BOOL when it is a bool, so `t & True` shows its result.
A chain such as `(s & t) & u` still raises in a repr, as it does for
Vector and Matrix.
… infix expression A behavior fix. When an operand of `&`, `|` or `@` was itself an infix expression and the sizes did not match, building the error crashed in _mismatched, which called _dummy(obj) while _dummy took a second parameter, obj_type, that it never used (since #517). Before: >>> (v & v) & w # v of size 3, w of size 4 TypeError: _dummy() missing 1 required positional argument: 'obj_type' After: >>> (v & v) & w DimensionMismatch: ... The same held with the expression on the right (w & (v & v)), for `|`, for Matrix operands ((A & A) & B), and for `@` ((A @ A) @ w). _dummy now takes only obj. Tests cover Vector and Matrix with the expression on either side, for `|`, `&` and `@`.
Behavior change. A numpy scalar or 0-d array on the left of an infix
operator was computed by numpy, which converted a Scalar with __array__
and could not convert a Vector or Matrix. graphblas objects now set
__array_priority__, so numpy's operators return NotImplemented and Python
calls the graphblas reflected method, which takes the numpy value as a
literal operand, typed as it is on the right:
s = Scalar.from_value(5, dtypes.INT8)
np.int16(2) - s # was np.int16(-3), now an INT16 ScalarExpression
np.int16(2) < s # was np.True_, now a BOOL ScalarExpression
np.int16(5) == s # was np.True_, now True (Scalar's ==)
np.int16(2) - v # was a TypeError, now an INT16 VectorExpression
np.int16(2) == A # was a TypeError, now a BOOL MatrixExpression
The same holds for expressions, A.T and index expressions. `&`, `|` and
`^` now go to graphblas too: `np.int16(2) & s` was numpy's bitwise and,
np.int16(0), and is now the TypeError of `2 & s` (only BOOL computes).
With an empty Scalar, numpy computed with nan; the result is now what a
Python float on the left gives:
e = Scalar(float)
np.float64(5) + e # was nan, now 5.0, as 5.0 + e is (and `-`)
np.float64(5) * e # was nan, now an empty Scalar, as 5.0 * e is
np.float64(5) < e # was np.False_, now an empty Scalar
np.float64(5) == e # was np.False_, now False
What numpy computed and graphblas does not now raises TypeError:
np.array([1, 2]) - s # was array([-4, -3])
arr -= s # was array([-4, -3]), in place
np.int16(2) << s # was np.int16(64); graphblas has no `<<`
On numpy < 2, where a size-1 array beside a Scalar is a literal (on the
right too), `np.array([1]) - s` is now an INT64 ScalarExpression.
Ufuncs still use __array__: np.sin(s), np.subtract(np.int16(2), s) and
np.asarray(s) give numpy values as before.
_literal_scalar types a literal on either side of a Scalar op, as in
op(1 & s) or 1 - s, through _literal_operand, which now takes left=True
for a left operand: a typed op takes each operand in its own input type
("Type a literal beside a typed op in the op's own input type", in #591).
_literal_scalar did not pass it, so with an op typed for two different
types, a literal on the left was typed as the right operand. Every
left-side caller passes argname="left", so _literal_scalar passes
left=argname == "left". An untyped op, as every infix operator is, types
the literal the same either way.
Behavior change (before -> after), with A a record of two float64 fields,
s = Scalar.from_value((3.0, 4.0), A) and add_a a UDT BinaryOp of
x + y.ll_a:
- add_a[INT64, A](2 & s): GraphBLAS's DomainMismatch (2 became an A element
where the op takes an INT64) -> 5.0.
db2f16b to
7b13e55
Compare
No behavior change. - ScalarInfixExpr._call_left is a method that falls back to InfixExprBase._call_left, in place of a module function assigned on ScalarEwiseAddExpr and ScalarEwiseMultExpr. - _typed_operands asks _is_literal of each operand, without a check for a Scalar expression first: a Vector or Matrix infix expression never holds a literal. - formatting._is_bool, which had one caller, is folded into _computes_itself. - call_op names the method for an error only where a literal is on the left.
|
This all seems pretty reasonable. Merging. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.