inference-server

Open repos and models their makers tagged inference-server, most starred first. The same for everyone.

AI agentsRepo

Rapid-MLX

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

  • 4.2k
  • Python
  • Apache-2.0
  • 20 min ago
raullenchaiTake it
AI on my own machineRepo

ramalama

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

  • 3.1k
  • Python
  • MIT
  • 5 d ago
containersTake it
Run my own servicesRepo

ollaya

Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

  • 1.5k
  • Rust
  • Apache-2.0
  • 5 h ago
ollaya-devTake it
AI on my own machineRepo

vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

  • 1.7k
  • Python
  • Apache-2.0
  • 11 d ago
waybarriosTake it
AI on my own machineRepo

pmetal

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

  • 334
  • Rust
  • 2 wk ago
EpistatesTake it
AI on my own machineRepo

ggrun

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.

  • 285
  • Go
  • MIT
  • 1 d ago
raketenkaterTake it
Voice and audioRepo

modelship

Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.

  • 54
  • Python
  • Apache-2.0
  • 10 h ago
modelship-aiTake it
Computer visionRepo

CVAuto

👁 零代码零标注 CV AI 自动化测试工具 🚀 免除大量人工画框和打标签等,直接零代码快速自动化测试 CV 计算机视觉 AI 人工智能图像识别算法:行人检测、动植物分类、人脸识别、OCR 车牌识别、旋转校正、舞蹈姿态、抠图分割 等,还可一键 下载测试报告、导出训练和测试数据集

  • 44
  • JavaScript
  • Apache-2.0
  • 6 wk ago
TommyLemonTake it
AI on my own machineRepo

euron-local-inference

One-command local model serving: hardware detection, a GGUF model manager with quantization presets, an OpenAI-compatible API and a throughput dashboard. Terraform for AWS, Floci for local.

  • 0
  • Python
  • MIT
  • 1 d ago
Search GitHub and Hugging Face for inference-server

Get , open it once and sign in. Then come back here and press the button again.