An open-source, cloud-native, high-performance gateway unifying multiple LLM providers
📖 Documentation · 🚀 Getting Started · 💬 Discussions · 🐛 Issues
Inference Gateway is a proxy server that provides a unified API to interact with multiple large language model (LLM) providers - from local solutions like Ollama to major cloud providers like OpenAI, Anthropic, Groq, Cohere, Cloudflare, and DeepSeek.
Stop managing multiple SDKs and API keys. Route all your LLM traffic through a single, production-ready gateway.
📋 Copy the command
curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "openai/gpt-5.5", "messages": [{"role": "user", "content": "Hello!"}]}'
| Feature | Description |
|---|---|
| 🔀 Unified API | One OpenAI-compatible endpoint for 16 LLM providers, plus Anthropic Messages and OpenAI Responses compatibility |
| 🔧 Tool-use | Function calling across supported providers with a unified API |
| 🔌 MCP | Native Model Context Protocol support for automatic tool discovery, and the gateway itself as an MCP server on POST /mcp |
| 🚦 Guardrails | OPA/Rego policies, secret and PII detection, and an optional external guardrail service - applied to requests, responses and MCP tool calls |
| 🤖 A2A | Agent-to-Agent coordination across specialized agents |
| 🌊 Streaming | Real-time token streaming from all supported providers |
| 🖼️ Multimodal | Vision input plus image, speech, sound effect, music and video generation endpoints |
| 🎯 Decisions | OpenAI Decisions API compatibility on POST /v1/decisions: typed questions answered with native probabilities by decision models, relayed to OpenAI and translated for Ollama, Ollama Cloud and llama.cpp |
| 🛡️ Enterprise | OIDC authentication, configurable timeouts, and TLS support |
| ☸️ Kubernetes | First-class K8s support with Operator and HPA scaling |
| 📊 Observability | OpenTelemetry Prometheus metrics following the GenAI semantic conventions, tracing, and an OTLP push endpoint |
| 🔒 Privacy First | Self-hosted, zero data collection, Apache 2.0 licensed |
| 🌿 Lightweight | ~10MB binary with minimal resource footprint |
📋 Copy the commands
# Run with Docker docker run -p 8080:8080 \ -e OPENAI_API_KEY=your-key \ ghcr.io/inference-gateway/inference-gateway:latest # Or install the CLI curl -fsSL https://raw.githubusercontent.com/inference-gateway/cli/main/install.sh | bash infer init && infer chat
👉 Full setup guide: docs.inference-gateway.com/getting-started
| Repository | Description |
|---|---|
| inference-gateway | The core gateway server |
| operator | Kubernetes Operator for lifecycle management |
| cli | A Git-first CLI coding agent that turns ideas, issues, and tasks into real code changes |
| infer-action | GitHub Action for long-horizon tasks |
| desktop | Desktop App for computer-use |
| opentask | Browser extension for browser-use |
| schemas | MCP, A2A, and OpenAPI schemas |
| docs | Documentation site |
| Repository | Language |
|---|---|
| sdk | Go |
| typescript-sdk | TypeScript |
| rust-sdk | Rust |
| python-sdk | Python |
| Repository | Description |
|---|---|
| adk | Agent Development Kit (Go) |
| typescript-adk | Agent Development Kit (TypeScript) |
| rust-adk | Agent Development Kit (Rust) |
| Repository | Description |
|---|---|
| google-calendar-agent | Google Calendar scheduling & automation |
| browser-agent | Browser automation via Playwright |
| documentation-agent | Context7-style documentation access |
| grafana-agent | Grafana dashboards automation |
| n8n-agent | n8n workflow generation & automation |
| mock-agent | Mocking and testing |
| Repository | Description |
|---|---|
| adl | Agent Definition Language for declarative agent definitions |
| adl-cli | Scaffold and manage A2A-powered enterprise agents |
| a2a-debugger | A2A agents troubleshooter |
| registry | Registry for A2A agents and skills |
| agents | Agents repository |
| skills | Skills repository |
| tokenless | A E2E testing library for calling AI applications without wasting tokens |
| awesome-a2a | Curated list of A2A-compatible agents |
We welcome contributions of all kinds - bug reports, feature requests, documentation improvements, and code!
- ⭐ Star the main repo to show your support
- 🐛 Report bugs via GitHub Issues
- 💬 Join discussions in GitHub Discussions
- 🔧 Submit PRs - see
CONTRIBUTING.mdin each repository
Released under the Apache 2.0 License · Built with ❤️ in Go