Precedence¶

Every NVFP4 entry point that used to read the environment now takes an nvfp4_4over6= keyword typed Optional[NVFP44Over6Config]. There are three cases, but only one public type:

The four FLASHINFER_NVFP4_4OVER6* variables are documented in the repository’s CLAUDE.md. They remain supported as a compatibility shim but are deprecated: being process-wide, they cannot express two models served in one process under different recipes. When the shim is retired the default of nvfp4_4over6= becomes None and no signature changes.

Resolution happens in exactly one place, resolve_nvfp4_4over6(). Below it, every kernel driver sees the two-state Optional[NVFP44Over6Config] in which None means off, and nvfp4_4over6_code() packs that resolved value into the int64 the custom ops take. nvfp4_4over6_from_code() inverts the packing, which is how logs and repro tooling recover the recipe a captured call actually ran with.

FP4 KV Cache Quantization¶

GPU-accelerated quantization / dequantization for KV-cache data using the linear (non-swizzled) block-scale layout.

• nvfp4_kv_dequantize(): SM80+ (Ampere and later)

• nvfp4_kv_dequantize_paged(): SM80+ (Ampere and later)

• nvfp4_kv_quantize(): SM100+ (Blackwell and later)

• nvfp4_quantize_paged_kv_cache()

NVFP4 KV Cache Materialization¶

Materialize the active pages of a paged NVFP4 KV cache as FP8 for attention kernels that do not read NVFP4 K/V directly.

class flashinfer.quantization.kernels.nvfp4_kv_materialize.NVFP4KVMaterializer(num_kv_heads: int, page_size: int, head_dim: int, device: str | int | device = 'cuda', uses_shared_paged_kv_idx: bool = True)¶

Validate once, launch many times: nvfp4_kv_materialize_fp8() for serving loops.

Frameworks materialize once per layer per prefill step, always with the same layout. The functional API validates every tensor and allocates the output block table on each call, which takes several times longer on the host than the launch itself. This class checks the static layout and the device and loads the compiled kernel at construction. run() then launches with caller-owned outputs.

run() allocates nothing, does not synchronize with the host and launches on the current CUDA stream. It can be captured in a CUDA graph and replayed after new metadata is copied into the captured tensors.

Parameters:

• num_kv_heads (int) – Number of KV heads per page.

• page_size (int) – Tokens per page. Must be divisible by 4.

• head_dim (int) – Head dimension. Must be divisible by 64.

• device (Union[str, int, torch.device]) – CUDA device that holds the tensors passed to run().

• uses_shared_paged_kv_idx (bool) – Whether K and V share page indices. block_tables is [batch_size, max_num_pages] if so, else [batch_size, 2, max_num_pages].

Examples

>>> materializer = NVFP4KVMaterializer(num_kv_heads, page_size, head_dim)
>>> # Per step: one workspace and one zeroed table, shared by every layer.
>>> fp8_block_tables.zero_()
>>> for layer in layers:
...     materializer.run(
...         k_cache, v_cache, k_block_scales, v_block_scales,
...         block_tables, seq_lens, kv_page_indptr,
...         out=kv_cache_fp8, fp8_block_tables=fp8_block_tables,
...     )

run(k_cache: Tensor, v_cache: Tensor, k_block_scales: Tensor, v_block_scales: Tensor, block_tables: Tensor, seq_lens: Tensor, kv_page_indptr: Tensor, *, out: Tensor, fp8_block_tables: Tensor) → Tuple[Tensor, Tensor]¶

Write the active pages to out and their indices to fp8_block_tables.

Same result as nvfp4_kv_materialize_fp8() called with the same kv_page_indptr and out, for the pages and table entries it writes.

Parameters:

• k_cache – As in nvfp4_kv_materialize_fp8(), in this object’s layout.

• v_cache – As in nvfp4_kv_materialize_fp8(), in this object’s layout.

• k_block_scales – As in nvfp4_kv_materialize_fp8(), in this object’s layout.

• v_block_scales – As in nvfp4_kv_materialize_fp8(), in this object’s layout.

• block_tables – As in nvfp4_kv_materialize_fp8(), in this object’s layout.

• seq_lens – As in nvfp4_kv_materialize_fp8(), in this object’s layout.

• kv_page_indptr (torch.Tensor) – int32 [batch_size + 1] prefix sum of the pages each request reserves in out, starting at 0. Request b must get at least ceil(seq_lens[b] / page_size) pages. For the functional API’s default layout pass arange(0, (batch_size + 1) * m, m) with m = max_pages_per_seq.

• out (torch.Tensor) – float8_e4m3fn [num_out_pages, 2, num_kv_heads, page_size, head_dim] with num_out_pages >= kv_page_indptr[-1] + 1. It may be larger than the step needs, e.g. one workspace sized for the largest step; the kernel stops at kv_page_indptr[-1], so the pages past it are neither written nor visited. Request b’s page j is written to kv_page_indptr[b] + j + 1 and page 0 is zeroed.

• fp8_block_tables (torch.Tensor) – Contiguous int32 [batch_size, max_pages_per_seq]. block_tables must have at least max_pages_per_seq columns and seq_lens[b] <= max_pages_per_seq * page_size. Only the first ceil(seq_lens[b] / page_size) entries of each row are written. Zero the table once per step if the consumer reads the other entries; every layer of a step writes the same entries.

Returns:

• kv_cache_fp8 (torch.Tensor) – out.

• fp8_block_tables (torch.Tensor) – fp8_block_tables.

Note

The compiled kernel’s launcher checks dtypes, ranks, inner strides, 16-byte alignment and the extents shared between arguments. Tensor values (lengths, page indices, kv_page_indptr) are not checked, and all tensors must be on this object’s device.