GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing github.com — Tags · bitsandbytes-foundation/bitsandbytes
Tags
continuous-release_mainCache the hipBLASLt int8 matmul algo per problem on ROCm (#2100) * Cache the hipBLASLt int8 matmul algo per problem on ROCm hipBLASLt's heuristic query takes ~430 ms per call on MI300X, while the int8 matmul itself takes well under 1 ms, and igemmlt ran it on every call. Cache the algo that ran successfully per (handle, m, n, k, lda, ldb, ldc), including the problems where no workspace-free algo runs, which keep taking the fp32 fallback. Also destroy the heuristic preference and the descriptors on the fallback path, which leaked before. * Share hipBLASLt int8 algos across token-count buckets Check algos with matmulIsAlgoSupported before launching them and key the cache on a bucket of n, so a new sequence length reuses an algo instead of running the heuristic again. An n the shared algo doesn't fit gets its own entry, and a cached algo that fails re-queries instead of erroring. Signed-off-by: Kashif Rasul <[email protected]> --------- Signed-off-by: Kashif Rasul <[email protected]> Co-authored-by: Kashif Rasul <[email protected]> PreviousNext
|
| You are visitor 01402520 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |