A C++17 header-only, cross-platform tensor library with zero third-party dependencies — CPU (own SIMD kernels) and GPU (own Metal / CUDA / WGSL kernels) on macOS, Linux, Windows and the browser, from a single #include <tensorlib.h>. No BLAS, no cuBLAS/cuDNN, no CUTLASS: every accelerated kernel — GEMM, GEMV, attention, quantized (int4) matmul — is hand-written in this repo, so the only thing a consumer links against is the OS itself (and the CUDA driver, which is dlopen'd at runtime, not linked).

As a full end-to-end proof of that promise, the library also drives a real local-LLM chat pipeline — GGUF loader, BPE tokenizer, and a Qwen2.5 decoder, all on the same own-kernel, zero-dependency foundation (see LLM inference below).

• Header-only, C++17 minimum (builds cleanly through C++23) — #include <tensorlib.h>, nothing to link or build separately.
• Cross-platform: macOS (Accelerate + Metal), Linux/Windows (own BLIS-style CPU microkernels — scalar/AVX2/NEON — and own CUDA kernels), and WebAssembly (own WGSL kernels over WebGPU). One dispatch seam, no platform #ifdefs in consumer code.
• Zero third-party dependencies: no OpenBLAS, cuBLAS, cuDNN, or CUTLASS anywhere in the accelerated paths (see Dependency policy).
• MLX-style lazy evaluation: ops build a graph; tl::eval() evaluates multiple arrays in one topological pass, with build-time peephole fusion (every node carries an affine epilogue, so scalar chains and dot-then-scale collapse to a single dispatch).
• Switchable backend via tl::use_cpu() / tl::use_gpu() / tl::use_auto() (auto picks CPU or GPU per op from measured per-kernel-class thresholds).
• Zero-copy views (transpose / reshape / slice), numpy broadcast rules.
• Data types: float (primary), plus bf16/int4 storage dtypes for the bandwidth-bound LLM decode path (see the milestones).

Design informed by silarray (the macOS-only experiment this succeeds), rebuilt for three platforms with a single dispatch seam the backends plug into. Serves as the matrix foundation for culebra's Tensor type (culebra keeps autograd/VJP; this library owns the graph, fusion, and execution).

#include <tensorlib.h>

auto a = tl::array::ones({1000, 1000});
auto b = tl::array::ones({1000, 1000});

auto c = a.dot(b) * 0.5f + 1.0f;   // lazy: one fused GEMM+epilogue dispatch
auto d = (a + b).relu();           // also lazy

tl::eval(c, d);                    // evaluate both in one pass

tl::use_gpu();                     // Metal (macOS), CUDA (Linux/Windows), WebGPU (wasm)
auto e = a.dot(b);
tl::use_auto();                    // CPU or GPU per op, by measured size

#include <tensorlib.h> pulls in the whole library; everything lives in namespace tl. The CMake target is INTERFACE, so it deliberately does not impose a standard on you — set C++17 or newer yourself.

tokenizer.h and gguf.h are not in the umbrella header; include them directly if you want them.

Shapes are tl::shape_t (= std::vector<int64_t>), so braced literals work. {} is a valid shape — it makes a rank-0 scalar.

shape(), strides(), rank(), size(), contiguous(), defined(), dt(). Note it is rank(), not ndim().

Reading elements: at({i, j}) (no bounds checking), item() for a single-element array, raw() for a const float*, data() for a mutable float*. All of these evaluate the graph first, and flush any pending GPU work — mixing them with lazy ops is safe.

There is no printing. No operator<<, no to_string — format arrays yourself from raw().

For tests: allclose(a, b, rtol = 1e-5f, atol = 1e-6f) and array_equal(a, b) (the same thing with zero tolerance). Both return false on a shape mismatch, and, as numpy's do, where either holds a NaN; an infinity matches only itself. max_err(worst, err) is the larger of two errors with a NaN in either kept, for a running worst that a NaN cannot slip through (std::max drops one).

Everything below is lazy unless noted.

Broadcasting follows numpy's trailing-dimension rules; an incompatible pair throws. attn_decode takes q [H,D], K/V [H,ctx,D] and returns [H,D].

Ops build a graph. Nothing runs until something forces it:

auto c = (a.dot(b) * 0.5f + 1.0f).relu();   // nothing computed yet
c.eval();                                    // now it runs
tl::eval(c, d, e);                           // or: several roots, one pass

.eval() (member) is the idiomatic form; the free tl::eval(...) is variadic and evaluates multiple roots in a single topological pass, which is what you want when their graphs share subexpressions.

Evaluation also happens implicitly on data(), raw(), item(), at(), the scalar reductions, add_(), and when a view is taken of a lazy source.

tl::use_cpu();    // default
tl::use_gpu();    // Metal / CUDA / WebGPU, per platform
tl::use_auto();   // per op, by measured size thresholds
bool ok = tl::gpu_available();   // compiled in AND a device is present

Selection is a process-wide switch, and which GPU backend you get is decided at compile time. When no device is present, use_gpu() and use_auto() silently run on the CPU. The auto thresholds are measured per kernel class and per backend; the matmul one is re-derived on the host at the first auto-mode graph eval (a four-point CPU-vs-GPU census, a few ms; TL_AUTO_MATMUL pins it, TL_AUTO_TRACE=1 prints the census) and TL_BATCH_MATMUL_BIAS overrides the batched-matmul one. The CPU gemm splits across threads only when each would get a floor of multiply-adds that is derived from the pool's measured wake-up on first use (TL_CPU_MIN_WORK pins it; misc/census_pool_latency.cpp shows the arithmetic); below that it runs on the calling thread.

The GPU ops are written once, in gpu_ops.h, over views (gpu::span: a device handle and a byte offset) and a kernel ABI every backend shares (gpu_abi.h). A backend is a device core — memory, a kernel table, one dispatch — plus kernel source, and the ops it runs its own way as members of its own struct; an op a backend has no kernel for answers false and the evaluator falls back to the CPU. gpu::census(kernel) counts launches, so a test can tell the two apart. No machine here runs CUDA kernels, so tools/cuda_trace records what the CUDA backend asks of the driver (kernel, grid, every argument) against a stand-in libcuda in a Linux container, and tools/cuda_trace/compare.sh <ref> diffs that across a change to the backend's host side. -DTENSORLIB_HOST_GPU=ON selects a reference backend whose device is the CPU (gpu_host.h), which runs that whole layer and its conformance tests with no GPU. docs/backends.md covers the layers, the kernel ABI, and how to add an op or a backend.

tl::profile (profile.h) says where evaluation goes: a profile::scope labels a block, the evaluator opens one per op under it, and every kernel launch, transfer and blocking wait a backend performs lands under the innermost open scope. profile::start() / stop() bracket a session, rows() returns it, report(stderr) prints it as a table; TL_PROFILE=1 does all three for a whole process. CUDA stamps each launch with events, so its rows carry a device time; Metal knows the GPU time per command buffer (summarize().batch_device_us); WebGPU counts launches. A row's device_timed says how many of its launches the device time covers.

Compute and results are always f32. bf16 and q4 are weight-container formats for the bandwidth-bound decode path, not general dtypes:

auto w = weights.to_bf16();   // f32 -> bf16 (round-to-nearest-even)
auto q = weights.to_q4();     // f32 [K,N] -> grouped int4; needs K % 32 == 0
auto f = q.to_f32();          // back to f32 (a no-op if already f32)

Ops other than the native decode GEMV widen transparently. Direct element access requires f32 — raw(), data(), at() and item() throw on a bf16/q4 array, so call to_f32() first.

• data() requires a contiguous array; raw() does not. Calling data() on a transposed view throws — clone() it first.
• Views alias their base. transpose/reshape/slice share storage, so a write through data() is visible through every view of it — including from unevaluated graphs still holding that array.
• add_() mutates shared storage and is not safe on an array that is still feeding a lazy graph.
• Evaluation is single-threaded by design. There is no documented support for sharing an array across threads.

#include <tokenizer.h> gives tl::tokenizer(gguf_path) with encode(), decode(), bos_id(), eos_id(), read from the model's own GGUF metadata: GPT-2 byte-level BPE with the pre-tokenizer the file names (Qwen2's, Qwen3.5's or Llama 3's; another is refused), or SentencePiece. The checks pin it against HF's Qwen2.5 tokenizer and llama.cpp's on Qwen3.5, Llama 3 and Gemma 3. #include <gguf.h> gives tl::gguf::model(path) — an mmap'd reader with tensor(name), tensors(), kv(key), metadata(). Both are used by the LLM pipeline below.

Everything reduces to strides through one index walker, so views, broadcast and transposed operands share a single code path. Accelerated backends replace it per-op at the graph::eval_one dispatch seam; the seam carries no platform #ifdefs (non-Apple builds get inline stubs).

All three GPU backends cover the array surface — GEMM, elementwise unary/binary, rank-2 broadcast, row reductions (softmax / row sum / row max) — each with the same fused affine epilogue, and each validated against the reference implementation as an oracle. The LLM kernels (GEMV, attention, quantized matmul, DeltaNet) are on CUDA and Metal. WebGPU stops at the array surface by design: it serves tensors in the browser, and in-browser LLMs are better served by engines built for them. It does run the small pieces of the shared decode contract (argmax_to, embed_row on f32 tables, read_later), and its argmax is one workgroup, which is enough for array use but slow on a vocabulary-sized vector. Any op a backend lacks declines at the seam and runs on the CPU.

The WebGPU backend is built with emcc --use-port=emdawnwebgpu -DTENSORLIB_WEBGPU, and differs from the native two in two ways worth knowing before you target it:

• Chrome in practice. It needs JSPI to keep flush() and sync_to_host() synchronous. Firefox has it behind a flag and Safari has not shipped it; there, device acquisition fails, available() stays false, and every op routes to CPU — correct, just not accelerated. That is also the fallback if the page hands in no device.
• No STEEL-equivalent tiling. The auto-mode thresholds are calibrated against measurements, but the WGSL SGEMM is a straightforward tiled kernel; macOS gets a considerably more tuned one.

CI runs the wasm suite on every push — test/wasm/ builds the same oracle suite as the native backends and runs it headless via Deno (a browser is not required, locally either), asserting a per-kernel dispatch census so a silent all-CPU fallback fails the build rather than passing green.

Zero third-party dependencies, on every platform. No OpenBLAS, no cuBLAS/cuDNN, no CUTLASS — every accelerated kernel (CPU SIMD, CUDA GEMM/ GEMV/attention, int4 dequant-matmul) is hand-written in this repo. What each platform links against:

• macOS — only OS frameworks (Accelerate, Metal, Foundation).
• Linux / Windows (CPU) — nothing; own microkernels compiled straight into the consumer binary.
• Linux / Windows (CUDA) — nothing linked. Kernels are compiled to PTX at build time (nvcc, build-time only — not a runtime dependency) and the CUDA driver is dlopen'd (LoadLibrary on Windows) at run time, so a binary built with -DTENSORLIB_CUDA=ON still runs — falling back to CPU — on a machine with no NVIDIA driver at all.
• WebAssembly — nothing linked. The WGSL kernels are committed as a C string and the WebGPU entry points come from Emscripten's bundled emdawnwebgpu port, so the toolchain is the only requirement and it is a build-time one.
• Tests only — doctest is vendored (not a runtime dependency of the library itself).

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure   # cpu / gpu / auto modes
./build/tensorlib_bench                       # micro-benchmarks

On Linux/Windows with an NVIDIA GPU, add -DTENSORLIB_CUDA=ON (needs nvcc on PATH at build time only — see Dependency policy):

cmake -B build -DCMAKE_BUILD_TYPE=Release -DTENSORLIB_CUDA=ON
cmake --build build
ctest --test-dir build --output-on-failure

Requires C++17 or newer — the headers use inline variables, std::optional, and structured bindings, but nothing from C++20/23, so any g++ >= 11 / clang++ >= 13 works (Apple Clang too). The bundled CMake build compiles the tests at C++23, but consuming the headers only needs C++17. Exception: the macOS/Metal backend #embeds its shader source, so building on macOS additionally needs a #embed-capable compiler (Clang 19+); non-Apple builds never reach that #embed. The WebGPU backend has the same problem and solves it the other way — its WGSL is committed as a generated C string (kernels/tensorlib_webgpu_wgsl.inc, regenerated by kernels/gen_wgsl_inc.sh when the .wgsl changes), so it needs no #embed and no build step.

For a WebAssembly build, see test/wasm/build.sh — a flat emcc line with --use-port=emdawnwebgpu -DTENSORLIB_WEBGPU, plus -sJSPI=1, which is what lets flush() keep a synchronous signature.

bench/models/chat_qwen.cpp is the end-to-end proof of the header-only, zero-dependency, cross-platform claims above: it loads a real Qwen2.5, Qwen3, Qwen3.5, Gemma 3 or Llama 3.2 GGUF model (F16, BF16, or quantized as Q4_K / Q5_K / Q6_K / Q8_0 / Q5_0 / Q4_0 / Q4_1 blocks, which stay quantized on the device: Qwen3.8-27B Q4_K_M, 17.7 GiB, runs on a 24 GB RTX 3090) and chats, using nothing but this library's own kernels — own GGUF v3 reader (include/gguf.h), own GPT-2-byte-BPE and SentencePiece tokenizers (include/tokenizer.h), and own CUDA prefill/decode kernels. Gemma 3 runs as the 1B and the 4B, their local layers attending a sliding window of 512 and 1024 tokens. Llama 3.2 runs as the 1B and the 3B, with Llama 3's per-pair RoPE frequency factors (rope_freqs). No GGML, no llama.cpp, no third-party inference runtime anywhere in the path.

cmake -B build -DCMAKE_BUILD_TYPE=Release -DTENSORLIB_CUDA=ON
cmake --build build --target tensorlib_chat_qwen
./build/tensorlib_chat_qwen path/to/qwen2.5-0.5b-instruct.gguf "What is the capital of France?"

Decoding is greedy unless sampling is asked for, as Qwen recommends it: append temp=0.7 top_k=20 top_p=0.8 seed=1 (after the dtype, q4 spec and nothink arguments). The token is drawn on the GPU, so it costs a decode step little; the same seed draws the same text.

On an RTX 3090, Qwen2.5-0.5B (F16 GGUF, bf16 weight storage):

Decode started at 3.5 tok/s and reached ~500 by removing host-side graph construction, then per-launch overhead, then GEMV inefficiency. Against llama.cpp on the same files and GPU, decode is ahead for Qwen2.5, Qwen3 and Qwen3.5 (1.05-1.4x), with or without 2048 tokens cached, and prefill is ahead at 512 tokens (1.1-1.3x) and with 2048 cached on all but Qwen3, where it is level (both sum the prefill's products in f16 by default): docs/llama-cpp-comparison.md has the numbers, what differs between the two runs, and how to reproduce them. Quantized files prefill their Q4_K and Q8_0 projections on the int8 tensor cores, the activation rounded to 8 bits as llama.cpp's MMQ does (TL_NO_MMQ=1 takes an f16 path instead), and decode with the Q4_K and Q5_K GEMVs' activation in f16, on the tensor cores, and the other formats' in f32, where llama.cpp rounds it to int8 (their logits ~10x closer to an f32 forward's on Qwen); they are ahead of or level with llama.cpp, Qwen3.8-27B included (its prefill 0.98-1.02x, its decode 0.98-1.09x).

Prefill is a different regime and was slower than decode for a long time, because the prompt was fed through the decode path one token at a time. Its tokens share one set of weights, so batching turns the projections into GEMMs rather than GEMVs and the attention into one tiled causal pass — worth ~50x over replaying the decode graph per prompt token, and ~170x over the array-graph path chat used originally.

bench/cuda/speed/bench_qwen_decode.cpp and bench_qwen_prefill.cpp produce these numbers and also guard them: every prefill path is checked against the array-graph reference for the same first generated token, and the decode comparison is greedy-exact. Measure with min-of-rounds — a single run on WSL2 swings ±20 tok/s with clock ramp.

MIT license (c) 2026 yhirose