LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
-
Updated
Nov 12, 2026 - Python
GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing github.com — Build software better, together
#
inference-serverHere are 278 public repositories matching this topic...Language: All
Filter by language
Sort: Most stars
Sort options
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
rust machine-learning transformers api-server developer-tools llama command-line-tool inference-server webgpu rust-crate huggingface huggingface-transformers huggingface-models webgpu-shaders llamacpp llm-inference local-ai gguf ollama-api openai-compatible
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
macos mcp inference-server mlx apple-silicon local-llm llm-inference local-ai speculative-decoding llm-server coding-agents continuous-batching tool-calling anthropic-api mlx-lm openai-compatible claude-code ollama-alternative omlx-alternative lm-studio-alternative
Open-source inference server and production cluster for all the models your agent needs.
python nlp natural-language-processing information-retrieval deep-learning retrieval ml inference embeddings bge semantic-search inference-server data-pipeline reranking mlops vector-search colbert llm splade retrieval-augmented-generation
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
Turn any computer or edge device into a command center for your computer vision projects.
python docker machine-learning computer-vision deployment inference classification object-detection vit agents inference-server jetson tensorrt instance-segmentation onnx inference-api yolov5 yolov8 yolo11 yolov12
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
macos text-to-speech mcp openai speech-to-text inference-server mlx apple-silicon openai-api llm anthropic vision-language-model vllm local-llm continuous-batching tool-calling anthropic-api multimodal-ai openai-compatible claude-code
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
rust self-hosted calibration typesafe decision-model inference-server model-serving laya systemone onnxruntime local-inference llama-cpp ollama mcp-server decision-models jev system-one typed-decisions jev-alternative jev-compatible
The simplest way to serve AI/ML models in production
open-source machine-learning packaging artificial-intelligence falcon easy-to-use whisper inference-server model-serving inference-api stable-diffusion wizardlm
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
ocr grpc nvidia text-recognition text-detection inference-server fp16 tensorrt rag fastapi pdf-extraction paddleocr easyocr document-ai document-parsing qwen-vl gpu-ocr
An open-source computer vision framework to build and deploy apps in minutes
python machine-learning cloud video computer-vision deep-learning gstreamer ffmpeg multimedia inference pipeline-framework artificial-intelligence stream-processing video-processing perception yolo object-detection inference-server vision-framework multimedia-applications
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
python nlp data-science machine-learning ai computer-vision deep-learning tensorflow transformers inference pytorch artificial-intelligence inference-server predict paddlepaddle model-deployment model-serving serving huggingface modelserver
A REST API for Caffe using Docker and Go
PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.
macos machine-learning ai deep-learning metal transformers tui lora quantization ane inference-server mlx fine-tuning distillation apple-silicon llm llm-training llm-inference qlora gguf
Work with LLMs on a local environment using containers
llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.
golang metal vulkan cuda self-hosted moe inference-server multi-gpu openai-api llm llamacpp llama-cpp local-llm gguf speculative-decoding localllama ollama-alternative
This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.
api docker dockerfile computer-vision deep-learning neural-network gpu rest-api inference yolo deeplearning inference-server bounding-boxes no-code yolov3 alexeyab-darknet yolo-gui detection-inference-api inference-gui yolov4
This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.
api docker opencv deep-neural-networks cpu computer-vision deep-learning neural-network rest-api inference object-detection inference-server bounding-boxes no-code yolov3 detection-inference-api cpu-inference-api inference-gui yolov4 yolov4-darknet
ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.
machine-learning ai deep-learning cuda inference-server nueral-networks contributions-welcome onnx onnxruntime
This is a repository for an object detection inference API using the Tensorflow framework.
api docker cpu computer-vision deep-learning docker-container tensorflow rest-api docker-image inference deeplearning object-detection tensorflow-framework predictions inference-server bounding-boxes computervision docker-ce inference-engine detection-inference-api
Add this topic to your repoTo associate your repository with the inference-server topic, visit your repo's landing page and select "manage topics." |
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |