GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing docs.flashinfer.ai — FlashInfer 0.7.4 documentation

Welcome to FlashInfer’s documentation!¶

FlashInfer is a library and kernel generator for Large Language Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA. FlashInfer focuses on LLM serving and inference, and delivers state-of-the-art performance across diverse scenarios.

Get Started

Installation
- Python Package
Command Line Interface
Logging
fi_trace — Operation Schema Extraction
Trace Apply
Autotuning
Experimental APIs and Backends

Tutorials

Attention States and Recursive Attention
- Applications
- Related APIs
KV-Cache Layout in FlashInfer
FlashInfer on JAX with TVM FFI
- Execution requirements
Cake SiTU backend
- Prepared weights and scales
- Caller-owned execution

PyTorch API Reference

FlashInfer Attention Kernels
- Experimental Task-Scheduled Attention
- flashinfer.decode
- flashinfer.prefill
- flashinfer.mla
flashinfer.attn_scores
- Paged MQA Logits
- Scheduling and Setup
flashinfer.cake_fmha
- flashinfer.cake_fmha.cake_batch_decode_with_kv_cache
- flashinfer.cake_fmha.cake_batch_context_with_kv_cache
- flashinfer.cake_fmha.cake_fmha_manifest
- flashinfer.cake_fmha.get_cake_fmha_module
flashinfer.cake_fused_qk_rope_append
- flashinfer.cake_fused_qk_rope_append.cake_fused_qk_rmsnorm_rope_append_paged_kv_cache
- flashinfer.cake_fused_qk_rope_append.launch_plan
flashinfer.cake_fused_qk_rope_fp8_append
- flashinfer.cake_fused_qk_rope_fp8_append.cake_fused_qk_rmsnorm_rope_quantize_fp8_append_paged_kv_cache
- flashinfer.cake_fused_qk_rope_fp8_append.launch_plan
Cake ragged BF16 grouped GEMM (experimental)
- Stable API opt-in
- One-shot functions
- Autograd
- Prepared launches
- Names before the deterministic-family rename
Cake ragged BF16 grouped GEMM with a selectable numerical mode (experimental)
- Dispatch
- Numerical contract of deterministic=False
- Weight-gradient accumulation
- Functions
- Names
- Features by architecture
flashinfer.fused_moe.cake / cake_cute (MXFP4 SiTU routed MoE)
- Host plan
- Generated kernels
- Manifest
flashinfer.cake_sampling
- Launch forms
- Dispatch and fallback
- Source product
- Correctness contract
- Benchmark
flashinfer.experimental.cake_stepfun_moe
- Unified MoE API opt-in
- One-shot functions
- Prepared launches
flashinfer.gemm
- BF16 GEMM
- FP4 GEMM
- SVDQuant NVFP4 GEMM (SM100)
- Quantized-base GEMM with runtime BF16 LoRA (SM90, SM100, SM103)
- BF16 x FP4 GEMM (W4A16)
- MXFP8 GEMM
- FP8 GEMM
- Mixed Precision GEMM (fp8 x fp4)
- Router GEMM (DeepSeek-V3 / Mistral / GLM / Kimi-K2 / Kimi-K3)
- Blackwell SM100 GEMM
- cuTile GEMM
- Grouped GEMM (CuTe-DSL, Blackwell)
- Grouped GEMM (Ampere/Hopper)
- PrimsTS dense FP8 and NVFP4 GEMM (SM100, SM103, SM107)
flashinfer.grouped_mm
- BF16 / FP16
- FP8
- MXFP8
- FP4 (NVFP4 / MXFP4)
- MXFP8 MoE GEMM (SM120 cute backend)
- FP8 MoE GEMM (SM120 cute backend)
flashinfer.fused_moe
- Types and Enums
- Unified MoE API
- Utility Functions
- AlphaMoE Router (SM100/SM103)
- Multi-LoRA MoE (BGMV)
- CUTLASS Fused MoE
- cuTile Fused MoE
- TensorRT-LLM Fused MoE
- AlphaMoE FP8 Block-Scaled MoE (SM100/SM103)
- AlphaMoE NVFP4 (SM100/SM103)
- AlphaMoE NVFP4 prepared weight scales
- AlphaMoE NVFP4 prepared gate/up data
- AlphaMoE NVFP4 adjacent gate/up panels
- Cake NVFP4 Warp Decode (SM100/SM103)
- Prims-TS Unified MoE (SM100/SM103)
- Prims-TS Fused MoE
- Standalone TRT-LLM Gen Routing
- CuteDSL Fused MoE
- MonoMoE (Single-Kernel Block-FP8, SM90a)
- cuDNN Grouped-GEMM MoE
flashinfer.megamoe
- Buffer and execution
- Weight preparation
flashinfer.moe_ep: Expert-Parallel Communication
- flashinfer.moe_ep.BootstrapConfig
- flashinfer.moe_ep.MoEEpCommunication
- flashinfer.moe_ep.MoEEpCommParams
- flashinfer.moe_ep.MoEEpDispatchResult
- flashinfer.moe_ep.NVLinkOneSidedAlltoAll
- flashinfer.moe_ep.NVLinkOneSidedConfig
- flashinfer.moe_ep.CakeAlltoAll
- flashinfer.moe_ep.CakeAlltoAllConfig
- flashinfer.moe_ep.NVLinkTwoSidedAlltoAll
- flashinfer.moe_ep.NVLinkTwoSidedConfig
- flashinfer.moe_ep.create_communication
flashinfer.cascade
- Merge Attention States
- Cascade Attention
- Cascade Attention Wrapper Classes
flashinfer.comm
- CUDA IPC Utilities
- DLPack Utilities
- Mapping Utilities
- All-Gather Matmul
- Fused FP8 Row-wise GEMM + All-Reduce
- TensorRT-LLM AllReduce
- Unified AllReduce Fusion API
- Cake NVFP4-Transport AllReduce
- FP8 Quantized AllReduce
- vLLM AllReduce
- PCIe IPC AllReduce
- PCIe IPC AllGather and ReduceScatter
- Ulysses Context-Parallel All-to-All
- MNNVL (Multi-Node NVLink)
- TensorRT-LLM MNNVL AllReduce
- DCP All-to-All (Context-Parallel Attention Reduction)
- NCCL LSA DCP All-to-All + LSE Reduce
- TP MoE Finalize + RMSNorm
- MNNVL CuTe DSL Fused Collectives
- Mixed Communication
flashinfer.sparse
- BlockSparseAttentionWrapper
- VariableBlockSparseAttentionWrapper
- Cake VSA
- Token-Sparse Attention Inputs
- SDSA Indexer
flashinfer.msa_ops
- flashinfer.msa_ops.msa_proxy_score
- flashinfer.msa_ops.msa_proxy_score_fp4
- flashinfer.msa_ops.MSASparseAttentionWorkspace
- flashinfer.msa_ops.supports_packed_kv
- flashinfer.msa_ops.msa_sparse_attention
- flashinfer.msa_ops.msa_sparse_decode_attention
- flashinfer.msa_ops.msa_topk_select
- flashinfer.msa_ops.prepare_msa_nvfp4_sparse_decode
- MiniMax-M3 ICP indexer
flashinfer.qsa
- flashinfer.qsa.qsa_forward
- flashinfer.qsa.qsa_indexed_forward
flashinfer.pod
- PODWithPagedKVCacheWrapper
- BatchPODWithPagedKVCacheWrapper
flashinfer.cudnn
- flashinfer.cudnn.cudnn_batch_decode_with_kv_cache
- flashinfer.cudnn.cudnn_batch_prefill_with_kv_cache
- Ragged prefill: cached geometry and warmup
- Linear attention
flashinfer.cute_dsl
- Availability
- RMSNorm + FP4 Quantization
- Attention Wrappers
- Block Sparse Attention
- HCA Decode
- Fused MoE
flashinfer.concat_ops
- flashinfer.concat_ops.concat_mla_k
flashinfer.mla_kv_pack
- flashinfer.mla_kv_pack.concat_mla_kv_quant_fp8
flashinfer.diffusion_ops
- flashinfer.diffusion_ops.minimax_h3_bf16_pre_attention
- flashinfer.diffusion_ops.minimax_h3_fp8_pre_attention
- flashinfer.diffusion_ops.minimax_h3_nvfp4_pre_attention
- flashinfer.diffusion_ops.quantize_minimax_h3_qkv_weight_fp8
- flashinfer.diffusion_ops.quantize_minimax_h3_qkv_weight_nvfp4
- flashinfer.diffusion_ops.minimax_h3_fp8_out_proj
- flashinfer.diffusion_ops.minimax_h3_nvfp4_out_proj
- flashinfer.diffusion_ops.quantize_minimax_h3_o_weight_fp8
- flashinfer.diffusion_ops.quantize_minimax_h3_o_weight_nvfp4
- flashinfer.diffusion_ops.minimax_h3_sm120_varlen_attention_fp8
- flashinfer.diffusion_ops.minimax_h3_sm120_varlen_attention_nvfp4
- flashinfer.diffusion_ops.minimax_h3_sm120_varlen_attention_nvfp4_nodelta
- flashinfer.diffusion_ops.minimax_h3_fc1_swiglu_fp8
- flashinfer.diffusion_ops.prepare_minimax_h3_fc1_weight_fp8
- flashinfer.diffusion_ops.minimax_h3_mlp_fp8_sm120
- flashinfer.diffusion_ops.minimax_h3_mlp_nvfp4_sm120
- flashinfer.diffusion_ops.prepare_minimax_h3_fc2_weight_fp8
- flashinfer.diffusion_ops.prepare_minimax_h3_fc2_weight_nvfp4_sm120
- flashinfer.diffusion_ops.fused_qk_rmsnorm_rope
- flashinfer.diffusion_ops.fused_dit_residual_layernorm_scale_shift
- flashinfer.diffusion_ops.fused_dit_gate_residual_layernorm_scale_shift
- flashinfer.diffusion_ops.fused_dit_gate_residual_layernorm_gamma_beta
flashinfer.page
- Append new K/V tensors to Paged KV-Cache
flashinfer.sampling
- flashinfer.sampling.sampling_from_probs
- flashinfer.sampling.sampling_from_logits
- flashinfer.sampling.softmax
- flashinfer.sampling.top_p_sampling_from_probs
- flashinfer.sampling.top_k_sampling_from_probs
- flashinfer.sampling.min_p_sampling_from_probs
- flashinfer.sampling.top_k_top_p_sampling_from_logits
- flashinfer.sampling.top_k_top_p_sampling_from_probs
- flashinfer.sampling.top_p_renorm_probs
- flashinfer.sampling.top_k_renorm_probs
- flashinfer.sampling.top_k_mask_logits
- flashinfer.sampling.chain_speculative_sampling
flashinfer.topk
- Top-K Selection
- Utility Functions
flashinfer.logits_processor
- Pipeline Construction
- Pipeline
- Processors
- Types
- Customization Features
flashinfer.norm
- flashinfer.norm.rmsnorm
- flashinfer.norm.rmsnorm_quant
- flashinfer.norm.fused_add_rmsnorm
- flashinfer.norm.fused_add_rmsnorm_quant
- flashinfer.norm.fused_add_rmsnorm_fp8_block_quant
- flashinfer.norm.gemma_rmsnorm
- flashinfer.norm.gemma_fused_add_rmsnorm
- flashinfer.norm.layernorm
- flashinfer.norm.layernorm_quant
- flashinfer.norm.fused_rmsnorm_silu
- flashinfer.norm.fused_qk_rmsnorm_rope
- flashinfer.norm.fused_dit_residual_layernorm_scale_shift
- flashinfer.norm.fused_dit_gate_residual_layernorm_scale_shift
- flashinfer.norm.fused_dit_gate_residual_layernorm_gamma_beta
- Fixed-Width Fused Norms
- Training RMSNorm (Cake)
- Training RMSNorm, opt-in non-deterministic weight gradient (Cake)
- Training SiLU-gated RMSNorm (Cake)
- Training SiLU-gated RMSNorm, opt-in non-deterministic weight gradient (Cake)
flashinfer.rope
- flashinfer.rope.apply_rope_inplace
- flashinfer.rope.apply_llama31_rope_inplace
- flashinfer.rope.apply_rope
- flashinfer.rope.apply_llama31_rope
- flashinfer.rope.apply_rope_pos_ids
- flashinfer.rope.apply_rope_pos_ids_inplace
- flashinfer.rope.apply_llama31_rope_pos_ids
- flashinfer.rope.apply_llama31_rope_pos_ids_inplace
- flashinfer.rope.apply_rope_with_cos_sin_cache
- flashinfer.rope.apply_rope_with_cos_sin_cache_inplace
- flashinfer.rope.rope_quantize_fp8
- flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache
- flashinfer.rope.mla_rope_quantize_fp8
flashinfer.activation
- Up/Gate output activation
flashinfer.gdn_decode
- flashinfer.gdn_decode.gated_delta_rule_decode
- flashinfer.gdn_decode.gated_delta_rule_decode_pretranspose
- flashinfer.gdn_decode.gated_delta_rule_mtp
- flashinfer.gdn_decode.gated_delta_rule_replayssm_commit
- Rubin T=1 bf16-state decode (SM107)
- ReplaySSM speculative verification (SM100/SM103, SM120)
flashinfer.gdn_fused_decode_step
- flashinfer.gdn_fused_decode_step
- flashinfer.gdn_fused_decode_step_supported
flashinfer.gdn_prefill
- flashinfer.gdn_prefill.chunk_gated_delta_rule
- Cake GDN non-CP backend: host-free launch
- Cake GDN CP context-parallel backend
flashinfer.gdn_paged
- flashinfer.gdn_paged.chunk_gated_delta_rule_paged
- flashinfer.gdn_paged.gated_delta_rule_decode_paged
- flashinfer.gdn_paged.gdn_paged_prefill_slots
- flashinfer.gdn_paged.gdn_paged_decode_slots
- flashinfer.gdn_paged.GDNPagedPrefillSlots
- flashinfer.gdn_paged.GDNPagedDecodeSlots
- Page table
- Slot arguments of chunk_gated_delta_rule
flashinfer.gdn2_prefill
- flashinfer.gdn2_prefill.chunk_gated_delta_rule2
flashinfer.gdp_prefill
- flashinfer.gdp_prefill.chunk_gated_delta_product
- Decode and MTP verification (Cake, SM90 / SM100 / SM103)
Kimi Delta Attention (KDA)
- flashinfer.kda
- flashinfer.kda_decode
- flashinfer.kda_prefill
flashinfer.mamba
- flashinfer.mamba.selective_state_update
- flashinfer.mamba.checkpointing_ssu
- flashinfer.mamba.ssd_combined_fwd
- flashinfer.mamba.replayssm_materialize
- flashinfer.mamba.causal_conv1d_fn
- flashinfer.mamba.causal_conv1d_update
- Inference causal convolution
- Training causal conv1d (Cake)
- Context-parallel training causal conv1d (Cake)
- Training causal conv1d with a selectable numerical mode (Cake, experimental)
flashinfer.mhc
- flashinfer.mhc.mhc_post
- flashinfer.mhc.mhc_pre_big_fuse
- flashinfer.mhc.mhc_pre_big_fuse_with_prenorm
- Training mHC (Cake)
- Training mHC with a selectable numerical mode (Cake, experimental)
flashinfer.quantization
- Types and Enums
- Packbits Utilities
- NVFP4 4over6 Recipes
- FP4 Quantization
- CSF Block Scales
- FP8 Quantization
- Fused gated-activation MXFP8 quantization
flashinfer.green_ctx
- Green context creation
- uGPU localization (multi-die GPUs)
flashinfer.fp4_quantization
- See Also
flashinfer.testing
- Test Environment Setup
- Performance Analysis
- GPU Benchmarking

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.