model-serving

Open repos and models their makers tagged model-serving, most starred first. The same for everyone.

AI agentsRepo

flyte

Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)

  • 7.8k
  • Go
  • Apache-2.0
  • 1 h ago
flyteorgTake it
AI on my own machineRepo

LightLLM

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

  • 4.4k
  • Python
  • Apache-2.0
  • 2 h ago
ModelTCTake it
GitHubRepo

FedML

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

  • 4.1k
  • Python
  • Apache-2.0
  • 6 d ago
FedML-AITake it
AutomationRepo

mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

  • 1.7k
  • Python
  • Apache-2.0
  • 14 h ago
mlrunTake it
Run my own servicesRepo

ollaya

Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

  • 1.5k
  • Rust
  • Apache-2.0
  • 6 h ago
ollaya-devTake it
AI on my own machineRepo

ome

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

  • 527
  • Go
  • Apache-2.0
  • 13 h ago
ome-projectsTake it
AI on my own machineRepo

JetStream

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

  • 474
  • Python
  • Apache-2.0
  • 11 mo ago
AI on my own machineRepo

flama

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

  • 307
  • Python
  • Apache-2.0
  • 2 d ago
vorticoTake it
Computer visionRepo

tamserve

Live, multi-camera object tracking with SAM3, SAM 2 and EdgeTAM, on one GPU or many. One model per GPU serves every camera: faster than the original code, about 4x less GPU memory, identical masks.

  • 87
  • Python
  • Apache-2.0
  • 2 d ago
aryamandardaTake it
AI on my own machineRepo

multi-turboquant

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simulation, Godzilla KVarN/TriAttention with SM86/SM89 qualification, CUDA weight sharing, and multi-GPU planning.

  • 30
  • Python
  • MIT
  • 6 wk ago
aivrarTake it
Search GitHub and Hugging Face for model-serving

Get , open it once and sign in. Then come back here and press the button again.

model-serving: open source repos and models on Take My Repo