GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |
GeezerVison is showing github.com — Don't install graphblas=8.0 yet by eriknw · Pull Request #459 · python-graphblas/python-graphblas Conversation
Member
Fixes #458
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. All reactions
|
There was an error while loading. Please reload this page.
|
coverage: 86.77% (-13.0%) from 99.748% |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
* Pin the CI interpreter and its ABI in the update-env step conda-forge ships two ABIs for CPython 3.13+, and the "Update env" step did not restate ``python``, so the solver was free to replace the interpreter setup-miniconda installed. Three of the last sixteen python 3.14 draws on ubuntu-24.04-arm did exactly that: the log reads ``python 3.14.7-h6bfacdd_100_cp314 --> 3.14.7-hefd156b_0_cp314t`` and pytest then reports "Python: 3.14.7 free-threading build". Two of the three passed only because they also drew pandas 3.0. The third drew pandas 2.3, whose extension raises "The global interpreter lock (GIL) has been enabled to load module 'pandas._libs.pandas_parser'", and ``filterwarnings = error`` turns that into a collection error. It took test_formatting.py and test_recorder.py down with it, the second only because the first had left pandas half-imported. The window opened on 2026-08-19, when conda-forge uploaded python 3.14.7 for linux-aarch64 in both ABIs. Re-solving the failing job's own spec set per platform: linux-aarch64 and osx-arm64 both pick ``cp314t`` today, so macos-latest is on the same fuse and has simply not drawn 3.14 with an unlucky pandas yet. linux-64, osx-64 and win-64 pick the GIL build either way, which is why this has been an arm-only symptom. ``python-gil`` pins ``python_abi``; restating ``python=`` pins the version, since ``python-gil`` alone constrains only the ABI and its own version is left open. setup-miniconda's ``python-version`` is equally ABI-free, so this step is where both get settled. Re-solved with the pair added: every platform lands on a ``cp314`` build, and no draw becomes unsatisfiable or downgrades numpy, numba, pandas or psg, including python 3.11 with numpy 1.24 and scipy 1.9. python-gil has builds for 3.9 through 3.14, so one spec covers every version in the pool. The pypy guard matches the neighbouring lines: ``pypy python-gil`` is unsatisfiable, so an unguarded spec would hard-fail a pypy draw if one is ever added back. python-suitesparse-graphblas pins the same pair in ``.github/workflows/test.yml``, and asks for ``python-freethreading`` on the matrix entries that want the other ABI. An entry like that would be worth having here too, but it is a coverage decision rather than a fix, so it is not in this commit. * Bump the GitHub Actions and pin them to commit SHAs Every action moves to its newest release: checkout 7.0.1, setup-python 7.0.0, setup-miniconda 4.1.0, codecov-action 7.1.1, upload-artifact 7.0.2, gh-action-pypi-publish 1.14.2, and action-tmate 3.24 (pre-commit/action stays at 3.0.1, its newest). This supersedes the open Dependabot PRs #572, #573, #575, #577 and #578. Each is pinned to the commit of its release, with the version in a comment, since a tag can be moved to other code. Dependabot updates the pin and the comment together. The SHAs are the same ones python-suitesparse-graphblas and afar pin. Dependabot now waits seven days after a release before proposing it and groups all action bumps into one PR a week, as those two repositories do. None of the major bumps changes an input this repository uses. Most are the move to Node.js 24; setup-python 7 drops the ``pip-install`` input, checkout 7 refuses fork checkouts under pull_request_target and workflow_run, and upload-artifact 7 adds unzipped single-file uploads. One input changes: * setup-miniconda deprecated ``auto-activate-base`` in favor of ``auto-activate`` (already in 3.3), and every run logged a warning for it. * Update the pre-commit hooks pre-commit autoupdate: isort 9.0.2, pyupgrade 3.22.0, black 26.10.1, ruff 0.17.1, flake8 7.4.1, prettier 3.9.10 and check-jsonschema 0.38.2, with flake8's hand-pinned plugins moved to flake8-bugbear 26.9.30 and flake8-simplify 0.31.1 (and scripts/check_versions.sh with them). This supersedes pre-commit.ci's #579, which predates black 26.10.1, ruff 0.17 and prettier 3.9.10. pyroma stays on 5.0.1: autoupdate offers 5.1b1, and 5.1 is still in beta on PyPI (5.1b2), which is what the comment above it waits for. None of the new versions changes or flags any file, here or at the top of the current stack. * Add numba 0.68, awkward 2.14, networkx 3.7 and psg 10.5.1.1 to the CI pools Each is the newest release on conda-forge, and the audits named all four as newer than any pin. * numba 0.68 caps numpy at <2.6, the same as 0.67 (conda-forge's ``depends``). * awkward 2.14 needs awkward-cpp 57 and Python >=3.11, which every Python in the pool has. * networkx 3.7 requires Python >=3.12, so py3.11's pool stops at 3.6. It constrains numpy, scipy and pandas exactly as 3.5 and 3.6 do, so the existing ``>= (3, 5)`` rule covers it; audit_conda_versions.py now checks 3.7's constraints too. * psg 10.5.1.1 is the newest build of SuiteSparse:GraphBLAS 10.5.1, with numpy 2 builds for every Python we test. numpy (2.5), scipy (1.18), pandas (3.0), sparse (0.19) and pyyaml (6.0) have no newer series. scripts/check_versions.sh moves to the new minimums. The rest of the files with dependency versions need no change: pyproject.toml holds floors, environment.yml and dev-requirements.txt are unpinned, and binder/environment.yml and docs/env.yml pin Pythons that are still supported. `ci_pick_versions.py --validate`, audit_pypi_versions.py --samples 200 and audit_conda_versions.py --samples 100 all pass. * Support Python 3.15 conda-forge has Python 3.15.0 final (cp315 and cp315t), and every dependency CI installs has py315 builds: numba 0.68, numpy 2.5.2+, scipy 1.18.1, pandas 3.0.6, awkward 2.13+, psg 10.5.1.1, matplotlib 3.11.2 and coverage 7.16 (compiled, so the tracer works). PyPI has cp315 wheels of all of them except pyyaml, which pip builds from its sdist. Numba supports 3.15 from the start, so this is a single step rather than the usual two: the numba guards in pyproject.toml move to `python_version<'3.16'`. * pyproject.toml: the 3.15 classifier, the numba guards (all five, sparse's included), and py315 in black's target-version (black 26.10 knows it). * test_and_build.yml and imports.yml draw 3.15. The interpreter pin from the update-env step makes it the GIL build (cp315). * ci_pick_versions.py gets a 3.15 row in every per-Python table, and three rules for what has no py315 build: scipy <1.18, pandas <3.0, and awkward <2.13 (awkward is noarch, but each release pins one awkward-cpp, and 56 is the first with py315 builds). numba starts at 0.68 and psg at 10.5.1.1 for every source: its sdist is the first from after psg added 3.15 support. The validator learned the same three rules and now names the floor each failure needs; disabling any one of the new rules makes `--validate` fail. Checked: `--validate` passes (200000 combos), as do audit_pypi_versions.py and audit_conda_versions.py. A dry-run solve of the update-env install, built the way the workflow builds it, succeeds for 60 draws: 3.15 on all five CI platforms with every source and slowtask (numba skipped too), and 3.11-3.14 on four platforms. Every one lands on a GIL interpreter. The full test suite with --runslow passes on 3.15 with the newest pins (numpy 2.5.4, numba 0.68.0, psg 10.5.1.1, scipy 1.18.1, pandas 3.0.6, networkx 3.7, awkward 2.14.0, sparse 0.19.2). * Run zizmor in pre-commit and CI, and fix what it reports zizmor audits GitHub Actions workflows for security problems. It runs as a pre-commit hook (and so on pre-commit.ci, offline), and security.yml runs it again with a token for its online audits: advisories against the actions we use, pins to commits outside an action's repository, and version comments that no longer match their pin. security.yml also runs pysentry, which pre-commit.ci skips for lack of network. Both run on PRs that touch pyproject.toml, .pre-commit-config.yaml or .github/, and weekly for advisories published since. The setup follows afar's. pysentry now runs when pyproject.toml or .pre-commit-config.yaml changes rather than at every commit, where it fails offline, and with ``--direct-only`` it audits the floors in pyproject.toml instead of the newest versions uv resolves, which would hide a floor that allows a vulnerable release. It finds nothing either way today. What zizmor reported, and what changed: * excessive-permissions: test_and_build.yml, imports.yml, debug.yml and publish_pypi.yml had no ``permissions:``, so their jobs got this repository's default token, which is read-write on everything. They now ask for ``contents: read``, which is all checkout needs; uploads to Codecov and of artifacts do not use the token. * template-injection: the "Update env" and "Build extension module" steps pasted step outputs into their scripts with ``${{ }}``. The values come from fixed lists, so nothing could be injected, but they now reach the scripts as environment variables. That includes the ``format('python={0} python-gil', ...)`` spec, which zizmor does not flag but which is the same pattern. * use-trusted-publishing: suppressed inline on the token, with a TODO. A ``pypi`` environment with required reviewers already exists, but moving to trusted publishing needs a publisher registered on PyPI, which cannot be checked from here. * unpinned-uses and dependabot-cooldown were fixed by the previous commits. .github/zizmor.yml ignored all three of the first findings for whole files; nothing is left for it to ignore, so it is gone. Offline and online, zizmor now reports no findings (one ignored, the inline suppression). The default persona is the one afar uses; the pedantic one would also flag the matrix values in run steps, which are not attacker-controlled. * Re-date the numba#5395 notes: bool pointers still fail on Numba 0.68 The two MAINT notes on the BOOL-to-INT8 routing asked for a re-test. A cfunc that reads ``CPointer(boolean)`` still fails with "cond is not i1: i8", and one that writes it with "cannot store i1 to i8*", on Numba 0.66 and on 0.68, the release that adds Python 3.15. The routing stays; only the notes' dates and versions change. * Publish to PyPI with Trusted Publishing The publish step used a PyPI API token stored as a secret. A trusted publisher is now registered on PyPI for this repository, so the upload moves to its own job that exchanges a short-lived OIDC token instead: only that job gets `id-token: write`, and it runs in the `pypi` environment, which the PyPI registration names and which asks a maintainer to approve the upload. The build job is unchanged and hands the files over as the `releases` artifact it already uploads. This removes zizmor's last inline suppression. * Leave py315 out of black's target versions for now CI's bizarro job installs black from conda-forge, and on Python 3.15 the newest black it can install there is 26.1, because the pytokens >=0.4 that black 26.5 needs has no 3.15 build yet. Black 26.1 does not know "py315" and exits with an error, so the auto-generated code check failed on every 3.15 draw. Any other black older than 26.5 refused to run on this repository for the same reason. Nothing here uses syntax new in 3.15, so the formatting is the same; a MAINT note says when to add it back. * Install the drawn psg version on wheel and source draws The version picker draws a python-suitesparse-graphblas version for every source type, but ``psgver`` was a shell variable of the "Update env" step. "Build extension module", which pip-installs psg for wheel and source draws, runs in a new shell, where ``suitesparse-graphblas${psgver}`` expanded to plain ``suitesparse-graphblas``. So every wheel and source draw installed the newest psg on PyPI, whatever the "versions:" line in its log said; only conda-forge draws, which install psg in "Update env" itself, got the version they drew. It has been this way since wheel and source draws began in #385. It turned up in the CI logs read for why test_ssjit.py's JIT tests skip in CI. In all four wheel and source jobs there that drew a pin (psg 8.0.2.1, 9.4.5.0, 10.1.1.0 and 10.3.1.0), pip installed 10.5.1.1. "Update env" now passes psgver on as a step output, and "Build extension module" reads it through ``env:`` like the other step outputs (zizmor's template-injection audit). An output rather than ``$GITHUB_ENV``, so no other step sees it. Installing the drawn version brings two kinds of draw to life for the first time, and neither could have worked: * source: psg's sdist compiles against conda-forge's GraphBLAS, which was always the newest one (``graphblas>=7.4``). psg 9.4.5.0's sdist does not compile against GraphBLAS 10.5.1 (undeclared GxB_*_GPU_ID constants, changed GxB_Iterator and *_get signatures); against 9.4.5 it builds and imports. A source draw now asks for the GraphBLAS that conda-forge's build of the same psg depends on: python-suitesparse-graphblas X.Y.Z.W requires graphblas >=X.Y.Z,<X.(Y+1). An unpinned draw still gets the newest of both. * wheel: psg 7.4.3.2 has manylinux x86_64 wheels only, and the picker, which does not know the OS, offered it to every Python 3.11 wheel draw with numpy 1.x. It leaves the wheel pool; conda-forge draws still test 7.4. No other step after "Update env" reads a picked version. Checked: * PyPI has wheels on all five runner platforms (manylinux x86_64 and aarch64, macOS arm64 and x86_64, win_amd64) for every psg in the wheel pools, for every Python each one is offered on. * A dry-run solve of the "Update env" install, built the way the workflow builds it, succeeds for source draws of psg 9.3.1.0 (Python 3.11), 9.4.5.0 (3.13), 10.0.1.1 (3.12), 10.1.1.0 (3.13), 10.3.1.0 (3.14), 10.4.0.0 (3.11) and 10.5.1.1 (3.15) on all five platforms, each landing on the GraphBLAS that conda-forge pairs with it. * Locally (macOS arm64, Python 3.12), psg's 9.4.5.0 sdist builds and imports against conda-forge's graphblas 9.4.5 and fails against 10.5.1, while 10.4.0.0's builds against 10.5.1 too. * test_vector, test_matrix, test_scalar, test_op and test_ssjit (with --runslow) pass on psg's 8.0.2.1 wheel with numpy 1.25 (Python 3.11) and its 9.3.1.0 wheel with numpy 2.3 (Python 3.12). * ``ci_pick_versions.py --validate`` passes. * Run the JIT tests with --runslow in every Linux and macOS notebooks job Most of test_ssjit.py needs four things in one pytest run: the suitesparse backend, --runslow, Numba, and a GraphBLAS that can JIT-compile. The backend is drawn per pass, --runslow is on in one pass of each pytest job and in neither pass of a notebooks job, Numba is skipped at random outside the notebooks jobs, and psg's wheels never JIT. In one run in four or five no job had all four, so codecov reported about 210 lines of test_ssjit.py as missed on #591 even though the JIT works in CI. That came from reading CI logs for why those tests skip. The C compiler is fine: every Linux and macOS job on conda-forge's GraphBLAS (conda-forge, source and upstream draws) passes the JIT probe, and scripts/jit_diagnostics.py prints "Result: True". The skips come from psg's wheels, which build GraphBLAS with GRAPHBLAS_USE_JIT=OFF, and from Windows, where conda-forge's GraphBLAS names the build machine's cl.exe. A new step runs test_ssjit.py on its own, under coverage, with --runslow and --backend=suitesparse, in the ubuntu-latest and macos-latest notebooks jobs, before the coverage report and upload. Those jobs install Numba unless the version picker rules it out (some source draws on Python 3.11 and 3.12), and draw psg from all four sources. It mirrors the Windows-only "More tests for coverage" step, and runs the file in order (``-p no:randomly``), as local runs do. No condition on Numba or the JIT: where either is missing, the tests that need it skip themselves and the rest still run and count, so the step passes. Locally (macOS arm64), the step's command: * psg 10.5 with Numba and a working JIT: 48 passed, in about 45 s. * The same with Numba blocked: 19 passed, 29 skipped. * conda-forge's psg 9.3.1.0, 8.2.1.0 and 8.0.2.1, each with Numba and c-compiler as a notebooks job installs them: 47, 32 and 32 passed, the rest skipped for needing a newer SuiteSparse:GraphBLAS. * psg's 8.0.2.1 and 9.3.1.0 wheels, without Numba: all passed or skipped. * Cancel a CI run's identical twin, keyed on the PR's head commit When one push moves both a PR's head and its base branch, GitHub often delivers that PR's pull_request event twice, and Tests and Test imports each ran twice on the same merge commit. Restacking this repository's PR stack moves every branch in one push: in the last three restacks, 37 PRs had their base moved each time, and 48 of those 111 PR updates got a duplicate run (15, 13 and 20), the twins arriving up to about 50 s apart. It is GitHub's doing, not the push: gh-stack pushes once, with one ``git push`` and no API call after it, and an ``--atomic`` push got the same duplicates. The PRs that got one show ``base_ref_force_pushed`` beside ``head_ref_force_pushed`` on their timelines. This came out of tracing the 40 queued duplicates left after the last restack. The Security workflow, whose concurrency group cancels the first twin within seconds, never kept one. Both workflows now have a concurrency group keyed on the workflow, the PR number and the PR's head commit, with cancel-in-progress, so the later twin cancels the earlier one, which is nearly always still queued: * A run for an earlier commit of the same PR is in another group and is left to finish, as before. The group test_and_build.yml had commented out since #459, keyed on the branch, would cancel it; this replaces it. * A push to main has no pull_request, so the group falls back to the run id and the pushed commit: each push's run is a group of its own and is never cancelled. Neither workflow has another trigger; workflow_dispatch would key on its run id the same way. * A re-run keeps its run's group, so re-running a cancelled twin while the other is still going cancels that one. * Give the test and notebook steps a timeout A run that hangs holds its runner for GitHub's default six hours. One way to hang was seen locally: on macOS, a conda environment with a pip suitesparse-graphblas wheel loads two copies of libomp (the workflow sets KMP_DUPLICATE_LIB_OK=TRUE to allow that), and a full test run sat at 0% CPU for seven hours in SuiteSparse's OpenMP join barrier. "Unit tests" and "Unit tests (bizarro scalars)" now stop after 120 minutes (the slowest seen is 73.5 minutes, a full --runslow on macos-15-intel with the Tensor PR higher in the stack), and "Notebooks Execution check" after 30. * Keep conda-forge's libomp out of macOS jobs that use the psg wheel The suitesparse-graphblas wheel for macOS bundles its own libomp. When conda-forge's libomp is in the same process too (KMP_DUPLICATE_LIB_OK=TRUE lets both start), the two runtimes call into each other's barrier code: a GraphBLAS parallel region (a large calloc's memset) can wait forever in its join barrier, or a worker crashes. A macOS notebooks job hung that way in CI, and locally, with the wheel, numpy 2.1, scipy 1.13 and numba 0.68, the Tensor test modules hung at the same test in every run (2 of 2), also with NUMBA_THREADING_LAYER=workqueue (1 of 1). conda-forge's libomp gets in two ways: OpenBLAS's openmp build (loaded with numpy) and Numba's default "omp" threading layer. With both replaced, by OpenBLAS's pthreads build and Numba's "workqueue" layer, the same runs finished (3 of 3, two seeds; 529 passed), and the wheel's libomp was the only OpenMP runtime the process loaded. With the pthreads build alone, Numba's omp layer still loaded conda-forge's libomp, and the run crashed in the first parallel Numba kernel. So "Update env" adds libopenblas=*=*pthreads* to the conda install and sets NUMBA_THREADING_LAYER=workqueue for later steps on macOS draws of the wheel. conda-forge has macOS pthreads builds from libopenblas 0.3.30 on only, and scipy 1.9-1.11 require an older libopenblas, so draws pinning those keep the openmp build. Dry-run solves of the install line with the pin resolve on osx-arm64 and osx-64 for Python 3.11 to 3.15 (libopenblas 0.3.34 pthreads). * Give the other test steps a 30-minute timeout, as the notebooks step has "Miscellaneous tests", "More tests for coverage" and "JIT tests for coverage" run graphblas too, so the hang the unit-test timeouts guard against (two libomp copies on macOS) could hold their runner for the 6-hour limit as well. In the two green CI runs of 50-tensor on 2026-10-27, "Miscellaneous tests" took 0.1 to 0.5 minutes and "More tests for coverage" 1.6 to 2.1. "JIT tests for coverage" has not run in CI yet; with a cold JIT cache at the top of the stack it took 62 seconds locally, compiling 227 kernels. * Add python-suitesparse-graphblas 10.5.2.0 to CI's version draws psg 10.5.2.0 is on PyPI and conda-forge. PyPI has cp311 to cp315 wheels for every platform CI runs on (manylinux x86_64 and aarch64, macOS arm64 and x86_64, win_amd64), and the sdist. conda-forge has builds for py311 to py315 on linux-64, linux-aarch64, osx-64, osx-arm64 and win-64, all depending on graphblas >=10.5.2,<10.6, which conda-forge has on all five, so a "source" draw's graphblas pin solves too. The builds need numpy >=1.24 (py3.11) or >=1.25 (py3.12 on), which the numpy 2.x pool this list feeds always meets. No Python floor moves: 10.5.2.0 builds for every Python 3.11 to 3.15 on every source, so PSG_MIN_PYTHON stays as it is. * Give a pinned psg the GraphBLAS it was built for on conda-forge and source draws python-suitesparse-graphblas X.Y.Z.W wraps SuiteSparse:GraphBLAS X.Y.Z, and psg's own test_version_tracks_libgraphblas asserts that the two match. conda-forge's psg builds accept any graphblas >=X.Y.Z,<X.(Y+1), and so did this workflow's pin for source draws, so once conda-forge published graphblas 10.5.2 (2026-10-28), a conda-forge draw of psg 10.5.1.1 or 10.5.0.0 solved to graphblas 10.5.2, and a source draw compiled the 10.5.1.1 or 10.5.0.0 sdist against 10.5.2's headers. The "python-suitesparse-graphblas tests" step then reports `assert (10, 5, 1) == (10, 5, 2)`, and the graphblas tests run against a library the psg in the draw was not built for. Pin graphblas to >=X.Y.Z,<X.Y.(Z+1) for a pinned psg on conda-forge and source draws. Every psg pin in the pools (7.4.0 to 10.5.2.0) depends on graphblas >=X.Y.Z on conda-forge, and graphblas X.Y.Z is there for linux-64, linux-aarch64, osx-64, osx-arm64 and win-64. Unpinned psg, upstream and wheel draws are unchanged. Checked with dry-run solves of CI's spec list, 17 of 17 landing on graphblas X.Y.Z: conda-forge draws of psg 10.5.1.1 on py3.15 on all five platforms, of an old psg on py3.11 on each (7.4.0 on linux-64 and osx-arm64, 7.4.3.2 on linux-aarch64, 8.0.2.1 on osx-64, 7.4.1 on win-64), of 8.2.1.0, 9.3.1.0, 9.4.5.0 and 10.5.2.0, and source draws of 9.4.5.0, 10.3.1.0 and 10.5.1.1. Without the pin, a py3.13 osx-arm64 conda-forge draw of 10.5.1.1 solves to graphblas 10.5.2. * Bump pypa/gh-action-pypi-publish to v1.14.3 for GHSA-5xjx-3w67-w9x9 v1.14.2 and earlier trust a certificate authority that Sigstore has revoked, which Trusted Publishing and digital attestations rely on. zizmor's known-vulnerable-actions audit fails on the old pin. * Update the pre-commit hooks ruff 0.17.3, flake8 7.5.0 (with flake8-bugbear 26.10.27), zizmor 1.32.0, validate-pyproject 0.27 and check-jsonschema 0.39.0. The isort and pyroma updates pre-commit autoupdate offers are betas (10.0.0b2, 5.1b1), so they stay where they are. pre-commit run --all-files passes.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
bug: deprecating python 3.8 support might resolve it
There was an error while loading. Please reload this page.