Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

cuda quantization inference-optimization kv-cache vllm local-llm llm-inference qwen speculative-decoding qwen3 consumer-gpu rtx-3090

• Updated Nov 10, 2026
• Python

High-efficiency floating-point neural network inference operators for mobile, server, and Web

cpu neural-network inference multithreading simd matrix-multiplication neural-networks convolutional-neural-networks convolutional-neural-network inference-optimization mobile-inference

• Updated Nov 12, 2026
• C

Efficient Deep Learning Systems course materials

machine-learning deep-learning cuda pytorch performance-optimization distributed-training ml-infrastructure mlops inference-optimization efficient-deep-learning ml-systems

• Updated May 28, 2026
• Jupyter Notebook

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

machine-learning deep-learning neural-network compiler tensorflow pytorch inference-optimization mlir

• Updated Dec 30, 2024
• C++

The Tensor Algebra SuperOptimizer for Deep Learning

deep-neural-networks deep-learning inference-optimization

• Updated Jan 26, 2023
• C++

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

transformer inference-engine inference-optimization mixture-of-experts cpu-inference large-language-models gpu-inference llm-inference high-performance-inference hybrid-inference gguf-model-support llama-cpp-alternative

• Updated Nov 12, 2026
• C++

Everything you need to know about LLM inference

inference-optimization llm llm-inference inference-handbook inference-infrastructure

• Updated Nov 12, 2026
• TypeScript

Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

inference-optimization large-language-models llm llms llm-inference llm-engineering inference-engineering

• Updated Nov 9, 2026
• Markdown

[MLSys 2021] IOS: Inter-Operator Scheduler for CNN Acceleration

acceleration cnn parallelism inference-optimization

• Updated Apr 27, 2022
• C++

Batch normalization fusion for PyTorch. This is an archived repository, which is not maintained.

deep-neural-networks deep-learning pytorch batch-normalization inference-optimization

• Updated Apr 6, 2020
• Python

Ultrafast Qwen3-TTS: sub-50 ms time-to-first-audio at 10 requests per second.

text-to-speech streaming ai inference streaming-audio inference-optimization audio-generation speech-generation qwen3-tts

• Updated Aug 19, 2026
• Python

LLM infrastructure cost reduction via NUMA-aware weight banking: 147 t/s (8.8x stock llama.cpp) on refurbished enterprise POWER8. Self-hosted inference, no cloud APIs. Part of the Proof of Physical AI stack.

self-hosted transformer numa powerpc energy-efficiency ppc64le cost-optimization power8 inference-optimization edge-ai hebbian depin ai-inference llm llmops llama-cpp llm-inference ai-infrastructure proof-of-physical-ai vec-perm

• Updated Nov 7, 2026
• Python

Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

macos rust ai metal cuda inference nvidia mlx inference-engine model-serving inference-optimization blackwell aimodels

• Updated Nov 11, 2026
• Rust

Optimize layers structure of Keras model to reduce computation time

keras inference-optimization

• Updated Jul 18, 2020
• Python

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

ai-agents context-api inference-optimization prompt-engineering context-engineering openclaw hermes-agent

• Updated Oct 29, 2026
• Python

the agi compiler: records llm agent behavior, proves what repeats, and compiles it into verified, sandboxed wasm binaries that run for microdollars. nothing figured out twice, paper: https://arxiv.org/abs/2607.04542

agent compiler webassembly program-synthesis distillation conformal-prediction inference-optimization llm ai-infrastructure

• Updated Jul 7, 2026
• Rust

A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.

machine-learning awesome ai openai awesome-list claude cost-optimization inference-optimization kv-cache large-language-models llm prompt-engineering generative-ai prompt-caching context-window prompt-compression llm-cost token-optimization token-efficiency model-routing

• Updated Nov 8, 2026
• Shell

Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.

react self-hosted quantization model-deployment inference-optimization fastapi llm llm-serving llama-cpp comfyui vllm local-llm llm-inference gguf speculative-decoding sglang

• Updated Nov 6, 2026
• Python

[CVPR 2025] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models

pruning inference-optimization multi-modality llm token-pruning vision-language-model llava multimodal-large-language-models

• Updated Apr 16, 2026
• Python

Run 70B+ LLMs on a single 4GB GPU — no quantization required.

inference inference-optimization llm qwen airllm

• Updated Feb 28, 2026
• Python