GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — Type UDT ops like built-in ops; fix UDT literals, casts and comparisons by eriknw · Pull Request #591 · python-graphblas/python-graphblas Conversation
|
| Expression | main (numpy 2) | this PR |
|---|---|---|
int8_vec + 1 |
INT64 | INT8 |
int8_vec * 2, with a 100 |
INT64, 200 | INT8, wraps to -56 (as numpy) |
int8_vec ** 2, with a 100 |
INT64, 10000 | INT8, 127 (GraphBLAS saturates pow; numpy wraps to 16) |
int8_vec + 300, int8_vec % 300, apply(binary.minus, left=1000), ewise_union(..., 1000, 0), a UDF's apply(udf, right=1000) |
computed in INT64 | OverflowError (as numpy) |
uint8_vec - 1, with a 0 |
INT64, -1 | UINT8, 255 |
uint64_vec + 1, at 2**63 + 1 |
FP64, rounded | UINT64, exact |
uint64_vec // 3 |
FP64 | UINT64 |
fp32_vec * 0.5 |
FP64 | FP32 |
fp32_vec * 1j |
FC64 | FC32 |
fp32_vec + 1e300 |
FP64, 1e300 | FP32, inf with a RuntimeWarning (as numpy; fp32_vec == 1e300 warns too) |
fp32_vec == 0.1, at 0.1 |
False | True (compared in FP32, as numpy) |
Scalar(int8) + 1 |
INT64 | INT8 |
int8_vec.apply(binary.atan2, right=1) |
FP64 | FP32 |
int8_vec * 0.5, int8_vec / 300, int8_vec < 300 |
FP64, FP64, BOOL | unchanged |
int64_vec < 2**63, Scalar(int64) == 2**63 |
OverflowError |
all True, False (exact, as numpy 2) |
uint64_vec[0] = 2**64 - 1, Scalar.from_value(2**63, "UINT64") |
OverflowError |
stored |
To keep the old widening, make the operand strong: int8_vec + np.int64(1), a
Scalar (Scalar.from_value(1)), a typed operator
(int8_vec.apply(binary.plus["INT64"], right=1)), or convert first
(int8_vec.dup("INT64") + 1). numpy scalars, Scalars and typed operators were
strong before and still are. A comparison keeps no result type, so an int out of
an integer dtype's range compares exactly rather than raise, as does an op whose
result is another type anyway (int8_vec / 300 is FP64). to_dense(fill_value=...)
widens to hold a fill value that does not fit (fp32_vec.to_dense(fill_value=1e300)
is float64). dtypes.unify's is_left_scalar and is_right_scalar no longer change
anything (it used them to stand a value-0 array in for a Python number); this is
visible only on numpy 1.
Two more changes for built-in dtypes:
== computes eq in the type binary.eq is typed for,Scalar(INT64 2**53 + 1) == 2.0**53 wasisequal(check_dtype=False) still says False).Scalar.ewise_mult(op=binary.eq).select with a value op and a thunk of another type compares in the values'int8_vec.select("==", 1.0); it keeps the 1. "Fix select on complex valuestruediv is float64; int8 times float32 is float32).DomainMismatch, namingdocs/user_guide/udt.rst has the rules as two tables ("How results are typed" and
"Storing results"); types.rst, udf.rst and operators.rst have the rest.
From "Type UDT ops like built-in ops, cast UDT stores, broadcast array UDTs",
"Cast UDT stores without temporaries, type UDT literals, fold in #592-#595" and
the commits after them. Checked on main and on this branch:
i8x3 * 2.5 was an INT8[3]i8x3 / 2 is FP64[3];i8x3 + (0.5, 1.5, 2.5) is FP64[3] (main added 0, 1, 2); rec8 * 0.5 has+ 1 is INT64; i8x3 + np.int64(1) is INT64[3]; i8x3.apply(monoid.plus, right=0.5) is FP64[3]. A.ewise_mult(int64_vec, binary.plus) has an int64r8.inner(r16) is R16 (main: R8).ewise_add on Vectors and Matrices of mixed UDTs, or with truediv on anDomainMismatch that says why: an entry present in onetruediv case and truncated the quotients. Infix +ewise_union and ewise_mult work.w << v, v.dup(dtype=udt2) and fused stores such asw << v.apply(...) work between UDTs whose layouts correspond (main:f64x3_vec[0] = Scalar(i8x3) andi8x3_vec[0].new(dtype=F64x3) read the int8 bytes as float64 on main (valuescomputes in FP64 and stores with GraphBLAS's cast (truncate, saturate), andi8x3_vec += 0.5` truncates (main converted 0.5 to 0 and left it unchanged).apply(udf, right=(0.5, 2, 3)) added 0 and now(1.0, 2, 3) for an int8 array or 0 for a bool field,literals._refuse_unless_exact). Test suites-W error will see it.isequal compares as np.array_equal and ==Scalar([1, 1, 1]).isequal(1) was True and is False;Scalar([0, 0, 0]) == 0.5 was True and is False; fp64x1.isequal(fp64x3) wasi8x3 == 300 was an OverflowError and is False.rec.ewise_mult(int_vec, binary.lt)), raises KeyError (main: numpy'sint_vec.select(select.valueeq, Scalar(rec)) likewise.monoid.plus[R8, R16] raises TypeError, where main returned monoid.plus[R16]._gbudt_NNN, so their JIT kernels are not shared (they read and wrote thedtypes.ss.register_new refuses a name already used as a== on a UDT works without Numba.min, max, floordiv, division and complex multiplicationCode that may break: result dtypes of UDT ops with float literals or mixed types
change ((i8x3 * 2.5).new() is an anonymous FP64[3], so a UDF typed for the
INT8[3] UDT no longer takes it); ewise_add with truediv on an integer UDT
raises; UDF literals that do not fit raise or warn; monoid.plus[R8, R16]
raises; a Scalar of bytes == 1 raises KeyError. Pickles made on main load and
compute. Private names moved: graphblas.core.operator.udt_utils is now
udt_codegen, and scalar._dict_to_record, _udt_bytes and _padding_ranges
are in core/udt.py.
Not done, deliberately: broadcasting inside record fields; ewise_add on mixed
UDT Vectors without copies; store casts that would need a temporary; stores
between a UDT and a built-in dtype; C JIT kernels for mixed-type ops and casts;
a casting-policy setting.
register_new (an np_type Numba cannot type) kept the name, so adatetime with a time of day, a timedelta finer than the field, or a date==, so noon equaled its day in aday == np.datetime64("2020-01-01T12:00") was True and is False, andseconds == np.timedelta64(5000, "ms") was a KeyError and is True.op[A, INT64] became an element of A, whose bytes the opv.apply(iu[A, INT64], 2) nowScalar.from_value(2) gives.s += 0.5 on ani8x3 + np.array([[1, 2, 3], [4, 5, 6]]) raised numpy's "could notR in ibo,ibo[R] and v.reduce(ibo(1)) raised AttributeError.CI: the workflow passes secrets.CODECOV_TOKEN to the codecov action. Codecov
refuses tokenless uploads from this repository's own branches with "Token required
because branch is protected", so this PR had no coverage report. (The input moved
here from #636, which now only pins the action's version in that step.)
.pre-commit-config.yaml listed the check-executables-have-shebangs hook twice; the
second is removed.
The CI cap that keeps an unpinned pandas below 3.1 (conda-forge's pandas
3.1.0 fails three test_formatting.py tests) is now on main, from #658.
Coverage: tests for lines this PR adds that CI never ran (literal typing,
select's thunk rewrite, UDT store and ewise_add casts, UDF return-type errors,
JIT layout checks); # pragma: no cover for Numba-compiled bodies and unreachable
guards; and removal of a few branches no caller can reach. Locally, coverage of
literals.py, udt.py, udt_codegen.py, udf.py and select.py went from 92-96% to
99-100%. A later commit adds tests for more literals a UDT refuses (a record
literal of the wrong length, parts numpy cannot make an array of, text a bytes
field cannot encode) and for the UdfParseError a lifted op gives when Numba
cannot compile it, and pragmas for two arms of the monoid identity that the
binary op's refusal keeps anything from reaching.
Test fix: test_dtype's test of two UDT layouts under one name compiled kernels with
the JIT forced on, so where the JIT cannot compile (no C compiler, or a broken one) it
failed with JitError instead of skipping; CI always has a compiler. It now asks the
one-compile probe the JIT tests use and skips only its kernel part, so its C name checks
still run everywhere.
Behavior fixes found along the way, each in its own commit with before and
after in its message:
select with a value op on a complex input and a thunk of another type kept==, != and isequal on a UDT that is neither a record nor an array== was a regression in== with a number, which main answered, raises== and !=, are two text UDTs of one kindMatrix.power with a type-changing semiring on a UDT raisedselect with a float64 thunk too large for FP32 warned about overflow (a==.(np.float64(1.5), 1) no longer becomes (1, 1) in an int8 record, as it== raisednp.float16 in an int8 record's literal, which main cut== compares them in numpy, asr + (np.array([[1, 300, 3]]), 0) gave [2, 46, 6].r[0] = ...) still convert as numpy assigns.== between leaves of different kinds (text and a number, for example)== 1 compared b"1" with 1, and nested == (b"x", np.float64(1)) wasnp.timedelta64("NaT", "ms") intoMatrix.to_dense had shape (nrows, ncols, 1, 4),to_csr, to_csc, to_dcsr and to_dcsc values of shape (nvals, 1, 4),to_dicts each value as [[0, 1, 2, 3]], and Vector.to_dense raisedregister_anonymous refuses a dtype of size 0 (str, bytes, an emptylookup_dtype(str) registered the UDT "<U0" and Scalar + "x" raised aAttributeError: 'Boolean' object has no attribute 'bitwidth' (main too).register_anonymous leaves no unused GrB_Type behind. Without Numba thesess.register_new of unary, binary,ss.f and ss.nested.f), ran the kernel compiled first in theadd_theta_4_FP64_4_FP64_4_FP32_ and 12 hex digits); jit_c_name andjit_c_source show it.Element stores still use numpy's assignment cast.
IndexBinaryOp typing. An untyped op(theta) took the op's type from theta
alone and cast the inputs to it. It now returns a BinaryOp typed each time it
runs, as any other BinaryOp is, at the type that holds both inputs and theta.
Theta is typed by its value, as apply types an IndexUnaryOp's thunk: a Python
int is INT64, a float FP64 and a bool BOOL, a numpy scalar or a Scalar keeps
its type, and a numpy array is typed as its array layout. An op with
is_udt=True is compiled for the types of x, y and theta as they are, and
SuiteSparse gets theta's type as the op's theta_type, so theta is never
converted; a JIT op takes the signature for the inputs' and theta's types, as
a JIT IndexUnaryOp takes its thunk. op[X, Y, T] names all three types;
op[X, Y] and op[T] give theta y's type, as before. IndexBinaryOp is on main
but not released, so nothing is deprecated. Before and after (the commit
message has the rest, including pickling and repr):
A.ewise_mult(B, op()) on INT64: a BOOL op, so every value became True ->u.ewise_mult(w, op(2)) on FP64: computed in INT64 (1.5 + 2 * 2.25 gave 5)op(0.5) with INT64, FP32 or INT8 matrices: KeyError ->is_udt multiplier that makes theiop(), iop(False), iop(0) andiop(0.0) failed to compile -> all four run.op(10.0) on FP32 data ran in FP64 -> KeyErrorop["FP64", "FP32"] is x and y, no longer x andStack created with GitHub Stacks CLI • Give Feedback 💬
Sorry, something went wrong.
There was an error while loading. Please reload this page.
7d55403 to
d4c4481
Compare
d4c4481 to
0bd9607
Compare
479c26c to
563b80a
Compare
563b80a to
123b148
Compare
683dcc9 to
ec062f1
Compare
ec062f1 to
b8e797f
Compare
b8e797f to
17bc219
Compare
17bc219 to
da41e30
Compare
da41e30 to
15b9955
Compare
15b9955 to
02464a8
Compare
02464a8 to
2f6fb4d
Compare
2f6fb4d to
cdefa0b
Compare
cdefa0b to
dc5dbaf
Compare
dc5dbaf to
3b4489d
Compare
"Keep a NaT duration NaT in another unit on numpy before 2.3" handed
numpy a NaT of no unit, np.timedelta64("NaT"), in place of each NaT
duration. numpy 2.5 deprecates that generic unit ("The 'generic' unit
for NumPy timedelta is deprecated"), so on numpy 2.5 every store or
literal of a NaT duration into a duration UDT or field warned with a
DeprecationWarning, which the test suite's filterwarnings makes an
error: test_udt_nat_duration_in_another_unit_stays_nat,
test_udt_sequence_and_array_literals_type_like_numbers and
test_udt_literal_converts_only_when_exact failed.
Each NaT duration is now converted into the unit of the leaf it fills
by converting it as an array (astype), which keeps NaT on every numpy,
before numpy assigns it. The value is walked beside the dtype: a tuple
of a record's length as its fields, a list (or a tuple for any other
dtype) as a sequence of elements, and an array UDT by its element type.
Nothing makes a generic unit.
numpy 2.5.4: DeprecationWarning before, NaT with no warning now. numpy
1.26, 2.2 and 2.4: NaT in the target unit before and after. New rows:
a nested record's duration field, and an array UDT of durations given
a list, where Numba is absent (Numba cannot type an array of durations,
so registering one fails with it).
test_udt_layouts_under_one_name_get_their_own_jit_c_names set jit_c_control to 'on' and compiled kernels for both layouts, so on a host whose JIT cannot compile (no C compiler, or a broken one) it failed with JitError instead of skipping. CI always installs a compiler, so CI never saw it. It now asks _enable_jit_for_udt, which probes with a real compile once per process, and skips the kernel part when the answer is no. The C name checks and the register_new check need no compiler, so they come first and run everywhere.
Behavior fix. SuiteSparse exports the iso value of an array UDT with shape (1, *subarray shape), and normalize_values broadcast it to shape + values.shape, keeping that axis of length 1. With the suitesparse backend, for an iso Vector or Matrix of an array UDT such as INT64[4] (as v[:] = [0, 1, 2, 3] makes): - Vector.to_dense: TypeError "unsupported operand type(s) for +: 'int' and 'tuple'" on main and before this commit (it passed its size as an int); now an array of shape (size, 4). - Matrix.to_dense: shape (nrows, ncols, 1, 4); now (nrows, ncols, 4). - Matrix.to_csr, to_csc, to_dcsr and to_dcsc: values of shape (nvals, 1, 4); now (nvals, 4). - Matrix.to_dicts, which uses to_dcsr or to_dcsc: each value as [[0, 1, 2, 3]]; now [0, 1, 2, 3]. Vector.to_coo, Matrix.to_coo, the vanilla backend and records were already right. normalize_values now broadcasts to shape + values.shape[1:], and Vector.to_dense passes (size,). CI found it through #652's test_udt_mixed_jit_c_name_is_its_own (17 jobs, e.g. job 122633175811): SuiteSparse before 10.5 makes a vector built from equal values iso, so that test's constant inputs gave an iso result, and to_dense raised. With 10.5 the vectors are not iso.
…rse 7
Behavior fix. register_anonymous passed a dtype's itemsize to
GrB_Type_new without a check. SuiteSparse 8 and later refuse a size of
0, but SuiteSparse 7 takes it as 1. lookup_dtype auto-registers what it
does not know, so on SuiteSparse 7 a str became the UDT "<U0", which
holds nothing.
- register_anonymous of str, bytes, np.dtype("U"), an empty record or
a subarray of length 0: InvalidValue with no message on SuiteSparse
8 and later, a UDT on 7; now ValueError "dtype must not have a size
of 0" on every version.
- lookup_dtype(str): ValueError "Unknown dtype" on 8 and later, the
"<U0" UDT on 7; now ValueError "Unknown dtype" on every version.
- Scalar.from_value("x"), Scalar + "x" and v.apply(binary.plus,
right="x"): on SuiteSparse 7, ValueError "got more than 0 values to
unpack" from cffi; now the TypeError they raise on 8 and later.
CI found it in wave 2 through #654's test_bad_ewise case (s1, "x"),
which expects the TypeError, in its one job with
python-suitesparse-graphblas 7.4.2 (122632718294).
SuiteSparse:GraphBLAS 10.5.2 fixed the bug _value_select_in_own_type works around (DrTimothyAldenDavis/GraphBLAS#463): select with a VALUE* op of another type than the input's now casts both to the op's type. Its MAINT note said to delete the function once every supported version has the fix, but that would change two results that test_select_value_thunk_of_another_type pins: with the function bypassed on 10.5.2, an INT64 thunk of 2**53 + 1 beside FP32 or FC32 values of 2**53 rounds to 2**53 in the op's type and keeps that element. The other 61 cases of the test match. The note now says so, and gives the versions with the bug as 8.0.2 through 10.5.1. Comments only.
c045eea to
7efd40b
Compare
Behavior change. Registering a UDT that holds an array of bool,
datetime64, timedelta64, str or bytes (as the UDT, or as a field of a
record) raised Numba's own error, which does not say what to change:
dtypes.register_anonymous(np.dtype((np.bool_, (2,))))
# was AttributeError: 'Boolean' object has no attribute 'bitwidth'
# now ValueError: Numba, which python-graphblas uses for UDTs, cannot
# type an array of bool, as in ('?', (2,)); use an array of uint8
# instead
The ValueError, the type of the other errors register_anonymous raises
for a dtype it refuses, is chained from Numba's AttributeError. The
np_type of dtypes.ss.register_new gets the same error. Without Numba
these dtypes still register, as before.
Numba now types the dtype before GrB_Type_new, so a dtype it cannot type
no longer leaves an unused GrB_Type behind.
lookup_dtype, which registers a numpy dtype it has not seen, still says
"Unknown dtype" for these, as for any dtype it cannot register.
…own type Binding theta to an IndexBinaryOp of no type, op(theta), took the op's type from theta alone, and the inputs were cast to it. Now op(theta) returns a BinaryOp whose type is chosen each time it runs, as any other BinaryOp's is. Theta is typed by its value when it is given, as apply types an IndexUnaryOp's thunk: a Python int is INT64, a float FP64 and a bool BOOL, and a numpy scalar or a Scalar keeps its type. A numpy array is typed as its array layout, as other array literals are. The default theta, False, changes no type. An op registered without is_udt runs at the type that holds both inputs and theta. One with is_udt=True is compiled for the types of x, y and theta as they are, and SuiteSparse gets theta's type as the op's theta_type, which GxB_IndexBinaryOp_new takes apart from x's and y's. So theta is never converted. The compiled ops are cached per (x, y, theta) types, and op[X, Y, T] names all three; op[X, Y] and op[T] give theta y's type, as before. A Semiring built from op(theta) is polymorphic in the same way. op[T](theta) and op(theta, dtype=T) still bind at T. IndexBinaryOp has not been released, so nothing is deprecated. Behavior changes (before -> after): - A.ewise_mult(B, op()) on INT64: a BOOL op, so every value became True -> INT64, the values op computes. - u.ewise_mult(w, op(2)) on FP64: computed in INT64 (1.5 + 2 * 2.25 gave 5) -> FP64 (6.0). op(np.int8(2)): computed in INT8, where 300.0 became 127 and wrapped -> FP64. - A Semiring from op(0.5) with INT64, FP32 or INT8 matrices: KeyError -> computed in FP64, with or without is_udt. - The SuiteSparse user guide's argmax (an is_udt multiplier that makes the pair (k, A(i, k)), theta unused) on a matrix of pairs: iop(), iop(False), iop(0) and iop(0.0) were ops on BOOL, INT64 or FP64 values, which failed to compile for the pairs -> all four run, compiled for (pair, BOOL, BOOL), (pair, BOOL, INT64) and (pair, BOOL, FP64). - An is_udt op on INT64 data: op(0.5) or an FP32 Scalar theta made the op FP64 or FP32 and cast the inputs to it -> the inputs stay INT64, theta keeps its type, and Numba types the result (FP64 for x + y + theta). - An is_udt op on int32[3] data: op(np.array([10, 20, 30], np.int32)) raised "Cannot infer a dtype for theta" -> theta is an int32[3] value; an int64 array is an int64[3] one. op(5) gave GrB_DOMAIN_MISMATCH -> theta is an INT64. op((10, 20, 30)) still raises TypeError, now saying to give theta as a Scalar of its UDT; an op without is_udt raises KeyError for an array theta, as for a UDT Scalar. - A JIT op (indexbinary.ss.register_new) is registered for its x, y and theta types. op(theta) took the one registered at theta's type T for x, y and theta, and cast the inputs to T -> it takes the one for the inputs' types and theta's, without casting, as a JIT IndexUnaryOp takes its thunk: with an op registered at FP64, op(10.0) on FP64 data runs as before, op(10.0) on FP32 data ran in FP64 -> KeyError naming (FP32, FP32, FP64), and op(10) raised KeyError when bound -> when it runs. op.types is keyed (x, y, theta), so registering the same x and theta types for another y no longer raises "already defined", and op["FP64", "FP32"] is x and y, no longer x and theta: op["FP64", "FP64", "FP32"] is the op with an FP32 theta. - A bound op pickles with all its types: op["INT64", "FP64"](1) came back as op["INT64"](1), whose y is INT64. - op(theta).type: the bound type -> AttributeError, as for any untyped op; op(theta)[T] is op[T](theta). repr(op(2)): binary.f[INT64] -> indexbinary.f(2). op(theta)(A & B) works, as for other BinaryOps. - A Scalar theta is copied when given, so a later change to it does not reach the op, as when the op bound it at once. find_opclass now reads an opclass attribute on an OpBase subclass, so op(theta) is still a BinaryOp there. The user guide's examples drop the op[float](0.5) that hid the old typing. The comments on IndexUnaryOp's and SelectOp's UDT caches now say they are keyed by (dtype, thunk_type).
…the field Behavior fix. Where a literal for a record UDT is typed (a lifted op such as +, ==, a user's op, an IndexUnaryOp thunk, an ewise_union default, and == on a Scalar), a part for an array field must now broadcast to the field's shape, or ValueError names both shapes. An array UDT's element takes the shape it broadcasts to with a literal, as numpy does, so INT8[3] plus np.array([[1, 300, 3]]) is INT64[1, 3]; that is unchanged. A record's field keeps its shape, so for the record the typing gave up, and the literal was converted into the field as numpy assigns it: the leading axis was dropped and 300 wrapped to 44, silently, even beside a user's op, which refuses a literal that does not fit. Stores (r[0] = ...) still convert as numpy assigns, as they do for an array UDT. With r a Vector of a record with an int8[3] field a and an int64 field b, holding ([1, 2, 3], 0) (before -> after): - r + (np.array([[1, 300, 3]]), 0): ([2, 46, 6], 0) -> ValueError "array([[ 1, 300, 3]]) has shape (1, 3), which does not fit field 'a' of R, of shape (3,): ...". - r.apply(user_op, right=(np.array([[1, 300, 3]]), 0)), with user_op returning y: ([1, 44, 3], 0) -> the same ValueError. - r == (np.array([[1, 300, 3]]), 0): False -> the same ValueError; with np.array([[1, 2, 3]], np.int8) it was True. - r + ([[1, 300, 3]], 0), and the other paths with it: numpy's "setting an array element with a sequence" ValueError -> this ValueError. - r + ([1, 2], 0): numpy's "could not broadcast input array from shape (2,) into shape (3,)" -> this ValueError. - r[0] = (np.array([[1, 300, 3]]), 0): ([1, 44, 3], 0), unchanged.
Literal typing had arms no test reached: a comparison with a record literal of the wrong length (alone and nested beside an int beyond its leaf), a part holding numpy values that numpy cannot make an array of or that is not numbers, and text a bytes field cannot encode. An array of records now pins the UdfParseError a lifted op gives when Numba cannot compile it, and the NoJITWarning is checked to stay silent for a pair it warned about. Two arms of the monoid identity are excluded from coverage: the binary op compiles first, and it refuses what would reach them.
indexbinary.ss.register_new compiled every type signature of one op under the op's own name. SuiteSparse finds a compiled kernel in memory by its op's name and the types in the kernel's encoding, and an ewise or mxm kernel's encoding has x's, y's and z's types but not theta's. So two signatures that differ only in theta's type ran whichever kernel was compiled first. Now each signature's C function is compiled as name_N_x_M_y_theta, with N and M the lengths of the x and y parts (built-in types by name, UDTs by their C name): the C definition is the user's with the function renamed. A name SuiteSparse would cut at 127 characters is the op name's first 100 and a digest of the whole. The op's jit_c_name and jit_c_source show the C name and the source SuiteSparse compiles; jit_c_definition is still the definition given. JIT IndexUnaryOps and SelectOps are not affected: the thunk's type is in their kernels' encodings. Behavior change (before -> after): with add_theta registered at FP64 for x, y and theta, and then at an FP32 theta with (*z) = (*x) + (*y) + 2 * (double)(*theta), the FP32 signature on FP64 data with theta 10 gave [2, 2, 4, 0] (the FP64 kernel, reading the float as a double) -> [22, 22, 24, 20]. op.jit_c_name: None -> add_theta_4_FP64_4_FP64_FP32.
The ss.register_new functions of unary, binary, indexunary (and select) gave
SuiteSparse the op's last name as its C name, and indexbinary's the last name
and types. SuiteSparse finds a compiled kernel in memory by its op's name,
and checks an op's definition only when it loads a kernel from disk. So, in
one process, an op registered again under its name with another definition
(after delattr, as when a notebook cell is run again), or an op of the same
last name in another namespace (ss.f and ss.nested.f), ran the kernel
compiled first.
Now every op registered from C source is compiled under a C name made of its
last name, the types for an IndexBinaryOp (whose theta type is not in its
kernels' encoding), and a 12-hex digest of these with the full dotted name
and the definition: f_3a1c90e2d4b7, or add_theta_4_FP64_4_FP64_4_FP32_...
for an IndexBinaryOp. A name SuiteSparse would cut at 127 characters is the
last name's first 100 characters and a 16-hex digest. The function is renamed
in the definition wherever it is named, except as a struct member (after "."
or "->"). The same name and definition give the same C name, so a kernel is
compiled once for them. jit_c_name and jit_c_source, None for these ops
before (except IndexBinaryOp, from the previous commit), show the C name and
the renamed source; jit_c_definition is still the definition given. The
helper is ss._utils._jit_c_source.
Behavior changes (before -> after), in one process:
- binary.ss.register_new("f", x + y) and ("nested.f", x - y), both FP64:
nested.f computed x + y -> x - y.
- unary.ss.register_new("u", x * x), delattr(unary.ss, "u"), then
register_new("u", x + 10): u computed x * x -> x + 10.
SuiteSparse binds theta without checking that it holds a value: it reads the Scalar's value array. Binding an empty Scalar crashed the interpreter, and op(s) with no type, which binds a copy of s, ran with theta 0. Both now raise EmptyObject, as apply does for an empty IndexUnaryOp thunk, in every bind: typed or not, is_udt or not, a JIT op's too. Behavior change (before -> after), with s = Scalar(int): - op[int](s) and op(s, dtype=int): segmentation fault -> EmptyObject. - op(s): an op that ran with theta 0 -> EmptyObject, at the bind. With s a C scalar (is_cscalar=True): segmentation fault -> EmptyObject.
register_new recorded the name as a JIT C type name as soon as
SuiteSparse made the type, before the numpy and Numba types were made.
When one of those raised (an np_type with an array of bool, which Numba
cannot type), the name stayed reserved, and a correct second call
raised "name for dtype is unavailable". Main allowed the second call; the
check against C type names came with "Give each UDT layout its own JIT C
name". The name is now recorded after the type is registered. The type
made by the failed call is never used, so no JIT kernel knows its name.
Behavior change (before -> after), with d the typedef of pt_t:
- register_new("pt_t", d, np_type=np.dtype((np.bool_, (2,)))) raises
ValueError (unchanged), then register_new("pt_t", d,
np_type=np.dtype([("x", "f8"), ("y", "f8")])): ValueError "'pt_t' name
for dtype is unavailable" -> registers dtypes.ss.pt_t.
A record literal's date or duration field takes the field's type only
when its value converts into the field and back unchanged; otherwise it
keeps its own type, so == compares it whole. A numpy date did this
already, but a Python datetime.date, datetime.datetime or
datetime.timedelta, or a date given as text, always took the field's type.
So it was cut to the field's unit before == compared it: noon compared
equal to its day in a field of days, and 5 s and 1 us equal to 5 s in a
field of seconds. Main gave the same wrong answers (it gave them for numpy
dates too). Now such a value is first made the numpy value it stands for
(a datetime in microseconds, a timedelta in microseconds, a date as text
in the unit it is written to), and is typed as that numpy value is.
Behavior change (before -> after), with v a Vector of a record of an
M8[D] and an m8[s] field holding (2020-01-01, 5 s), and s = v[0].new():
- v == (datetime(2020, 1, 1, 12), np.timedelta64(5, "s")), and with
"2020-01-01T12:00" for the date: True -> False; the same for s ==.
- v == (np.datetime64("2020-01-01"), timedelta(seconds=5, microseconds=1)):
True -> False.
- v.apply(user_op, right=(datetime(2020, 1, 1, 12), np.timedelta64(5,
"s"))): stored the day, silently -> ValueError "does not fit", as for
np.datetime64("2020-01-01T12:00").
- v + (datetime(2020, 1, 1, 12), ...): UdfParseError from Numba ->
KeyError naming datetime64[D] and datetime64[us].
- A datetime at midnight, a date, and text such as "2020-01-01" still
compare equal and fit the field.
… does
A UDT that is a bare datetime64 or timedelta64 (not a record or an array)
converted a date or duration literal into its own unit before == compared
it, so noon compared equal to its day in a UDT of days. Main raised
TypeError here for a Vector, and gave True for a Scalar; "Compare UDTs that
are neither records nor arrays in eq and ne" made the Vector give True too.
A numpy duration in another unit was a KeyError instead, even when it was
the same duration.
Now such a literal is typed as in a record's date field: the UDT itself
when it converts into the UDT and back unchanged, else numpy's type for it
(a datetime is in microseconds, a timedelta too, a date as text in the unit
it is written to). Under eq and ne, a UDT of dates or durations pairs with
one in another unit, as text of one kind pairs at another width, and they
compare as numpy compares them.
Behavior change (before -> after), with day a Vector of an M8[D] UDT
holding 2020-01-01 and five one of an m8[s] UDT holding 5 s:
- day == np.datetime64("2020-01-01T12:00"), datetime(2020, 1, 1, 12) or
"2020-01-01T12:00": True -> False; != the opposite; the same for
day[0].new() ==.
- five == np.timedelta64(5000, "ms"): KeyError -> True; 5001 ms: KeyError
-> False. five == timedelta(seconds=5, microseconds=1): True -> False.
- binary.eq(five & a Vector of m8[ms] holding 5000 ms), and five.isequal
of it: KeyError -> True.
- day.apply(user_op, right=datetime(2020, 1, 1, 12)): stored the day,
silently -> ValueError "does not fit".
- five.apply(user_op, right=np.timedelta64(5000, "ms")): converted with a
DeprecationWarning -> converts with none, as in a record's field.
- day == timedelta(5): numpy's ValueError -> binary.eq's KeyError.
A literal operand beside a UDT became an element of that UDT even when the op was typed and takes that operand in another type: op[A, INT64] takes its thunk, or its second input, as an INT64. Apply and select pass a UDT thunk or operand to SuiteSparse as a pointer to its bytes, so the op read the bytes of the A element as an int64: silently wrong answers. Main did the same. A Scalar of the op's type gave the right answer. Now a typed op's own input type on the literal's side decides: its first input type for a left operand, else its second input or thunk type. GraphBLAS converts a value to a built-in input type, so the literal is typed as beside a built-in typed op (its own type: 2.5 for an INT64 thunk is converted to 2, as for an op typed on built-in types); a UDT input type takes only a literal that fits it. _literal_operand takes left=True from apply's left operand and from ewise_union's left default. Behavior change (before -> after), with v a Vector of a record A of two float64 fields holding (1.0, 2.0) and (3.0, 4.0), and user ops typed [A, INT64] (or [INT64, A]) that compare or add field a with the other: - v.apply(iu[A, INT64], 2): [False, False] -> [False, True], as with Scalar.from_value(2); the same for thunks 2.5 and np.int8(2). - v.select(se[A, INT64], 2), on a Vector or Matrix: empty -> index 1. - v.apply(f[A, INT64], right=2) and v.apply(g[INT64, A], left=2): about 4.6e18 for each element -> [3.0, 5.0]. - With B a record of two int64 fields: v.apply(h[A, B], right=(2, 3)): about 4.6e18 -> [3.0, 5.0]; v.apply(iu_rec[A, B], (2, 0)): [False, False] -> [False, True], and with (2.5, 0): [False, False] -> ValueError "does not fit". - Scalar(A).ewise_mult(2, f[A, INT64]): GraphBLAS's DomainMismatch -> 3.0.
The accumulator is typed on the object's dtype, and for a built-in Scalar GraphBLAS converts the value to that type before it accumulates: an INT8 Scalar of -1 with s += 0.5 (Scalar.__iadd__ is s(binary.plus) << 0.5) adds 0 and stays -1, as v(accum=binary.plus) << 0.5 does for a Vector of either kind. A UDT Scalar used the untyped op instead (_scalar_accum_op), so it computed -1 + 0.5 in FP64 and truncated the sum to 0. That was meant to match a Vector's v += 0.5, but that is v << v + 0.5, not an accumulate. Main gave the built-in answer. Now a UDT Scalar accumulates with the typed op, and the value is converted to the Scalar's type first, as storing it would convert it: a literal by Scalar.from_value, a Scalar or an element of another UDT by the store cast. A built-in Scalar into a UDT Scalar still raises DomainMismatch, as a store of it does, into a Vector too. Behavior change (before -> after), with s an INT8[3] UDT Scalar of [-1, -1, -1]: - s += 0.5 and s(accum=binary.plus) << 0.5: [0, 0, 0] -> [-1, -1, -1] (an INT8 Scalar gives -1; a Vector of the UDT gives [-1, -1, -1]). - s(accum=binary.plus) << (0.5, 1.5, 2.5): [0, 0, 1] -> [-1, 0, 1], as for a Vector of the UDT. - s(accum=binary.plus) << v[1], with v a Vector of an FP64[3] UDT holding [0.5, 1.5, 2.5]: [0, 0, 1] -> [-1, 0, 1], as with the same value in a Scalar of that UDT, which was cast first already.
udt.rst says a numpy array beside a UDT pairs as an array UDT of its own shape, and array UDTs pair when their shapes broadcast. A numpy array paired only when it broadcast to the UDT's shape, or had that shape apart from leading 1s. Any other array, even one whose shape broadcasts with the UDT's, was converted into the UDT, where numpy raised "could not broadcast", while the same values as a UDT operand worked. Main raised too (a KeyError for the UDT pair). Now such an array is typed in its own shape and pairs as an array UDT operand does; the result has the shape numpy's would. An array that broadcasts to the UDT's shape is typed in the UDT's shape, as before, which a literal that must become an element needs. Behavior change (before -> after), with v a Vector of an INT8[3] UDT holding [1, 2, 3] and u one of an INT8[1] UDT holding [1]: - v + np.array([[1, 2, 3], [4, 5, 6]]), and apply(binary.plus) with it on either side: ValueError "could not broadcast input array from shape (2,3) into shape (3,)" -> INT64[2, 3] [[2, 4, 6], [5, 7, 9]], as with the same values in an INT64[2, 3] UDT. - v == np.array([[1, 2, 3], [1, 2, 3]]), on a Vector or a Scalar: the same ValueError -> True. - u + np.array([1, 2, 3]), and u + (1, 2, 3): ValueError -> INT64[3] [2, 3, 4]. - v + [[1, 2, 3], [4, 5, 6]]: numpy's "setting an array element with a sequence" ValueError -> INT64[2, 3], as for the array. - v.ewise_union(v, binary.plus, a 2-by-3 array, 0): numpy's ValueError -> ValueError "does not fit".
OpBase.__getitem__ reads is_positional for a type the op does not have, and IndexBinaryOp had none, so asking an IndexBinaryOp (registered without is_udt) for a type it does not take raised AttributeError. Main did the same; IndexBinaryOp has not been released. It is now False, as on a bound IndexBinaryOp and every other op class. Behavior change (before -> after), with op an IndexBinaryOp of x + y + theta and R a record UDT: - R in op: AttributeError -> False, as for an IndexUnaryOp. - op[R] and op(Scalar.from_value((1.0, 2), R), dtype=R): AttributeError -> KeyError "... does not work with R".
reduce turns a BinaryOp that is part of a Monoid into that Monoid, and refuses any other BinaryOp with a TypeError naming it. It reads the op's monoid for that, and the error message its parent's; neither a bound IndexBinaryOp nor IndexBinaryOp had the attribute, so reduce raised AttributeError, for ibo[T](theta) on main and for the new untyped ibo(theta) too. Both now have monoid = None, as a BinaryOp of no Monoid has. IndexBinaryOp has not been released. Behavior change (before -> after), with v an INT64 Vector and ibo an IndexBinaryOp of x + y + theta: - v.reduce(ibo(1)) and v.reduce(ibo[int](1)): AttributeError "'_BoundIndexBinaryOp' object has no attribute 'monoid'" -> TypeError "Bad type for argument `op` in Vector.reduce(...) ... is not known to be part of a Monoid", as for binary.minus. - A.reduce_rowwise(ibo(1)): AttributeError -> NotImplementedException, as for binary.minus.
|
This grew larger than expected, but has a lot of things that ought to go in. I expect to do some followups to clean some things up. Merging. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.