GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing github.com — Build software better, together
#
inference-engineeringHere are 21 public repositories matching this topic...Language: All
Filter by language
Sort: Most stars
Sort options
Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
inference-optimization large-language-models llm llms llm-inference llm-engineering inference-engineering
A curated list of awesome inference engineering tools, papers and resources.
Rust + cuTile research prototype for paged latent-cache LLM decode attention, validated on an RTX 4060.
rust gpu cuda attention kv-cache ml-systems llm-inference paged-attention inference-engineering cutile
A curated, continuously verified map of inference engineering: tools, techniques, hardware, and knowledge for serving AI models.
A practical handbook for software engineers to learn AI, Large Language Models (LLMs), and Inference Engineering—from fundamentals to production systems.
CPU-only conversational voice assistant with VAD-based speech segmentation, streaming LLM response generation overlapped with concurrent TTS synthesis via producer-consumer queues. Batch STT (Whisper). Fully instrumented with per-stage latency benchmarking.
text-to-speech speech-to-text voice-assistant gemini-api voice-ai cpu-inference silero-vad whisper-ai piper-tts inference-engineering latency-benchmarking
KI ohne Cloud: a hands-on inference engineering workshop. Run, measure, optimize and operate open models with vLLM on your own infrastructure (nextacademy.io)
it's me as a repository
LLM inference benchmarking dashboard: Python FastAPI backend with async orchestration, WebSocket live TTFT/TBT/throughput comparison across configs (512/128 to 4096/1024 tokens), Grafana + Docker Compose stack, GitHub Actions CI; 21/21 pytest passing.
python docker benchmarking docker-compose gpu websocket grafana prometheus pytest throughput observability mlops github-actions fastapi ttft llm vllm llm-inference mi300x inference-engineering
Trace-Aware Serving Controller: eval-gated inference policy optimization
Deterministic simulator for KV-cache admission, placement, movement, eviction, and recomputation policies
Private AI on NeoClouds - Large Open Weight models need a lot of VERY large GPUs (aka massive infra)
amd tokens inference-optimization tokenomics inference-engines kv-cache nvidi hybrid-intelligence llms vllm genai llm-inference hybridai sglang inference-scaling inference-engineering tokens-per-second kv-cache-optimization tokens-per-watt
Evidence-gated benchmark, comparison and promotion control plane for LLM inference serving changes.
benchmarking reproducibility model-serving local-first ai-engineering llmops vllm llm-inference sglang inference-engineering
An interactive playground for learning inference engineering—explore LLM serving concepts, tune the stack, and graduate to production incidents.
Deep Agents and SvelteKit harness for authoring verifier-gated Bonsai workflow packs.
Offline, evidence-first bottleneck hypotheses for LLM inference traces
Executable field guide and deterministic labs for LLM inference engineering across kernels, scheduling, KV cache, placement, and control.
distributed-systems performance-engineering gpu kv-cache ml-systems llm-inference inference-engineering
Hands-on LLM inference labs on free T4 GPUs with vLLM: prefix caching, AWQ quantization, throughput knee, chunked prefill, KV-cache pressure. Every number reproducible.
benchmarking gpu quantization kv-cache llm-serving vllm llm-inference qwen sglang prefix-caching inference-engineering
Interactive inference-engineering lessons and a bilingual AI mentor, built with Next.js and the Vercel AI SDK.
Add this topic to your repoTo associate your repository with the inference-engineering topic, visit your repo's landing page and select "manage topics." |
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |