llm-serving

Open repos and models their makers tagged llm-serving, most starred first. The same for everyone.

AI on my own machineRepo

TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

  • 15k
  • Python
  • 48 min ago
NVIDIATake it
AI on my own machineRepo

gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

  • 6.1k
  • Python
  • Apache-2.0
  • 4 h ago
gpustackTake it
AI on my own machineRepo

trnrt

The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.

  • 1.6k
  • Python
  • Apache-2.0
  • 22 h ago
AI on my own machineRepo

flama

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

  • 307
  • Python
  • Apache-2.0
  • 9 h ago
vorticoTake it
AI on my own machineRepo

model-serving-minefield

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.

  • 178
  • Python
  • MIT
  • 3 d ago
BlackwellboyTake it
AI on my own machineRepo

ReadyLLM

Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.

  • 101
  • Python
  • MIT
  • 13 d ago
htobtyTake it
AI on my own machineRepo

vllm-windows-build

Native Windows vLLM 0.33.0 (Python 3.13/CUDA 13.0), plus a Python 3.14/CUDA 13.2 build; older versions on the Releases page. PyTorch 2.14, FlashAttention/Rust, OpenAI serving, 10 KV formats, Multi-TurboQuant, experimental CPU/NVMe KV offload. No WSL or Docker.

  • 68
  • Python
  • MIT
  • 14 h ago
aivrarTake it
Search GitHub and Hugging Face for llm-serving

Get , open it once and sign in. Then come back here and press the button again.

llm-serving: open source repos and models on Take My Repo