The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

python machine-learning deep-learning model-serving multimodal mlops ml-engineering ai-inference llm generative-ai llmops llm-serving model-inference-service llm-inference inference-platform

• Updated Nov 2, 2026
• Python

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

kubernetes inference huggingface llm modelscope llamacpp vllm text-generation-inference ollama sglang inference-platform

• Updated Aug 27, 2026
• Go

A tool for developers and researchers to design FHE pipelines and execute them on cloud-based GPUs

cryptography inference post-quantum-cryptography homomorphic-encryption post-quantum fhe ai-security accelerated-computing fully-homomorphic-encryption confidential-computing post-quantum-security post-quantum-algorithms ai-tools secure-inference post-quantum-encryption inference-platform post-quantum-cryptography-

• Updated Nov 12, 2026
• Python

Local AI workflow orchestration and runtime management framework

workflow-engine embeddings tts stt observability model-serving 3d-generation depth-mapping llm node-based-ui generative-ai audio-diffusion local-ai inference-platform image-diffusion

• Updated Nov 6, 2026
• Rust

GPU-aware LLM serving platform with multi-worker routing, dynamic batching, model placement, streaming, admission control, and reproducible performance benchmarks.

distributed-systems pytorch model-serving fastapi huggingface-transformers dynamic-batching gpu-scheduling llm-serving inference-platform ai-infrastructure

• Updated Oct 23, 2026
• Python

Prepaid asynchronous LLM inference platform: FastAPI, PostgreSQL-backed queues, deadline-aware scheduling, modular GPU workers, vLLM integration, token metering, ledger billing, Stripe test payments, Next.js dashboard, Python SDK, and Docker. Includes a mock mode and recorded walkthrough; real GPU validation pending.

docker machine-learning multi-tenant stripe asynchronous gpu nextjs scheduler postgresql nvidia developer-tools python-sdk job-queue model-serving fastapi inference-serving usage-based-billing vllm llm-inference inference-platform

• Updated Nov 12, 2026
• Python

artificial-intelligence inference-platform web333

• Updated Jan 7, 2026
• Rust

A conceptual framework for a high-scale Agentic AI orchestrator, inspired by enterprise-grade inference platforms.

distributed-systems ml-ops generative-ai inference-platform agentic-ai

• Updated Mar 15, 2026

Local AI inference platform built with FastAPI, React, Ollama, and SQLite. Features multi-model benchmarking, structured generation, experiment tracking, JWT authentication, and performance analytics for small language models running entirely offline.

react machine-learning typescript jwt-authentication fastapi ai-engineering llm ollama offline-ai inference-platform

• Updated Jun 16, 2026
• TypeScript

AI inference platform architecture lab demonstrating admission control, fairness scheduling, bounded queues, and graceful degradation under burst traffic.

distributed-systems microservices load-testing prometheus observability system-design platform-architecture inference-platform ai-infrastructure rag-architecture

• Updated Mar 15, 2026
• Python