awesome-llm-routing-cascading
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
- 5
- 5 wk ago
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
A high-throughput and memory-efficient inference and serving engine for LLMs
Structured multi-perspective deliberation for hard decisions. Run full councils, focused triads, or duo debates across Claude Code, Codex, Gemini CLI, and OpenCode.
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
MBART Quality Estimator proposed in IST-Unbabel 2021 Submission for the Quality Estimation Shared Task
In-browser LLM router over 12 models from 6 providers, with a cheapest-first cascade that grades and escalates
Open-source AI gateway you run yourself. One endpoint for 40+ providers, with virtual keys, budgets, and usage tracking.
Awesome Jev: a source-backed field guide to TypeSafe's System One model, with SDKs, live demos, agent tools, and independent evaluations.
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
| Name | Stars | License | Language | Last change | In common | Compare |
|---|---|---|---|---|---|---|
| ymoslem/CRE-Router | 4 | Apache-2.0 | Python | 21 h ago | ||
| ymoslem/awesome-llm-routing-cascading | 5 | None declared | Unknown | 5 wk ago | llm-cascade, llm-routing | Compare |
| katanemo/plano | 7.1k | Apache-2.0 | Rust | 4 wk ago | llm-routing | Compare |
| vllm-project/vllm | 97k | Apache-2.0 | Python | 7 h ago | llm-serving | Compare |
| Unbabel/OpenKiwi | 236 | AGPL-3.0 | Python | 4 y ago | quality-estimation | Compare |
| 0xNyk/council-of-high-intelligence | 4.9k | MIT | Shell | 6 d ago | llm-routing | Compare |
| ray-project/ray | 44k | Apache-2.0 | Python | 1 h ago | llm-serving | Compare |
| Unbabel/wmt21-qe-task | 9 | Apache-2.0 | Python | 5 y ago | quality-estimation | Compare |
| sugeerth/llm-router | 1 | MIT | JavaScript | 11 d ago | llm-cascade, llm-routing | Compare |
| mozilla-ai/otari | 959 | Apache-2.0 | Python | 10 h ago | llm-routing | Compare |
| liguodongiot/llm-action | 25k | Apache-2.0 | HTML | 5 wk ago | llm-serving | Compare |
| AbdelStark/awesome-typesafe-jev | 619 | MIT | HTML | 3 wk ago | llm-routing | Compare |
| NVIDIA/TensorRT-LLM | 15k | None declared | Python | 10 h ago | llm-serving | Compare |