awesome-llm-routing-cascading
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
- 5
- 5 wk ago
Cascaded Framework for Cost-Aware LLM Serving
Repo ymoslem/CRE-Router, read from GitHub 1 min ago. ymoslem has not joined Take My Repo.
ymoslem/CRE-Router is a public repo on GitHub by ymoslem. Cascaded Framework for Cost-Aware LLM Serving. It declares the Apache-2.0 license on GitHub. It is written mainly in Python, has 4 stars and last changed on 2026-11-21. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-21.
In the maintainer's words. Shown from GitHub; Take My Repo does not keep a copy.
Implementation of the paper, Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving.
Production deployment of large language models forces a trade-off between accuracy and cost. CRE-Router framework routes each query to the most cost-effective model in a pool, then escalates low-quality outputs to a stronger model:
Error(m, c) + lambda * Cost_norm(m), where lambda ($\lambda$, the cost weight) is
tuned once to lambda* ($\lambda^{\ast}$) to satisfy a latency budget (TPOT or E2EL).cre fit plans this stage from the routing: the target is
the most accurate model the routing uses, every cluster served by a cheaper
model is gated, and each gated model gets its own classifier.Both stages train only on task-correctness labels obtainable from standard benchmark evaluation; no extra annotation is required.
Stage 1 scores each model with Error(m, c) + lambda * Cost_norm(m), where Cost is
either time per output token (--cost-metric tpot, the default) or end-to-end
request latency (--cost-metric e2el).
TPOT and E2EL rank the models the same way while pool members emit similar
numbers of tokens, and diverge as soon as they do not. A reasoning model and a non-reasoning one can differ by less
than a millisecond in TPOT while differing several-fold in E2EL, because
E2EL = TTFT + TPOT * L and TPOT divides the output length L out. Use E2EL
whenever the pool mixes thinking and non-thinking members, or verbose and brief
ones.
The full workflow is driven by the cre CLI, one command per step:
cre cluster → cre evaluate → cre fit → cre qe-data → cre qe-train → cre serve
Serving (cre serve) switches between the live vLLM backends through LiteLLM.
Three commands sit outside this chain and work offline.
cre qe-cascade replays a trained estimator over saved generations, to check it
before serving.
cre stats builds the stats file cre fit reads from saved captures, with no GPU.
cre compose measures a routing's Stage 1 and Stage 1 + 2 accuracy, TPOT and
E2EL on the batches that were served; use it for accurate results, and the
paper's numbers come from it.
cre batches writes the batches that serve a fitted routing with each model
receiving exactly its routed traffic, as one batch (see
Serving a routing as deployed).
The main commands and options are summarized in the following table. For full flags for any command, run:
cre <command> --help
| Command | What it does | Input | Output | Key options |
|---|---|---|---|---|
cre cluster |
Embeds training queries and fits k-means centroids (k chosen by Silhouette) | JSONL of training queries |
centroids.npy and router.json; train_assignments.jsonl
|
--k, --embedding-model
|
cre evaluate |
Runs each model per cluster through vLLM's benchmark, scores answers, averages per-cluster error and TPOT. This is the slowest step; cost scales with model size, output length, and --runs, and it runs once per model. |
dataset JSONL, a running vLLM server, fitted centroids | per-model entry in the stats JSON; raw runs under results/
|
--runs, --concurrency, --save-generations, --artifacts
|
cre fit |
Pareto-prunes the pool, sweeps $\lambda$, selects $\lambda^{\ast}$ under the cost budget, and plans Stage 2: the escalation target and the clusters to gate | stats JSON, budget B | routing table, $\lambda^{\ast}$ and the Stage 2 plan in router.json; prints which estimators to train next |
--budget (required), --cost-metric, --error-tol, --escalate-to, --selection, --output
|
cre qe-data |
Builds a gated model's estimator training data from its cre evaluate generations |
the gated model's *_generations.jsonl on train and held-out questions |
train.jsonl and test.jsonl for cre qe-train
|
--train, --test, --out, --task
|
cre qe-train |
Fine-tunes ModernBERT-base as the accept/escalate QE classifier | HF dataset of model outputs with correctness labels | QE classifier checkpoint |
--train-split, --eval-split, --learning-rate, --max-length
|
cre qe-cascade |
Replays the trained QE over an efficient model's saved generations, composing per-cluster cascade accuracy and escalation counts | generations JSONL, the strong model's outcomes, a QE checkpoint | per-cluster cascade accuracy and escalations per run |
--clusters, --accept-threshold
|
cre stats |
Rebuilds per-cluster stats from saved captures, regrading every answer; prices a one-batch capture under any clustering of its questions | pool spec, captures directory or the released captures dataset | stats JSON for cre fit
|
--pool, --captures, --dataset, --clusters
|
cre batches |
Writes one batch per model holding exactly its routed traffic, then the strong model's batches with each run's escalations, and the routing spec cre compose reads |
router.json from cre fit, each model's questions, accept probabilities |
batch JSONL files, manifests with SHA-256, routing spec |
--router, --dataset, --prefix, --name, --probs
|
cre compose |
Measures a routing's Stage 1 and Stage 1 + 2 accuracy, TPOT and E2EL from the batches that served it | routing spec, captures, accept probabilities | accuracy, TPOT, E2EL, escalations per run |
--routing, --dataset, --probs, --json
|
cre serve |
Runs the live router: sends each incoming query to its cluster's assigned model, and escalates weak answers to a stronger model | serving config, running backends | live HTTP router on port 4000 |
--config, --port
|
From PyPI:
pip install "cre-router[full]"From source, for changes not yet released:
pip install "cre-router[full] @ git+https://github.com/ymoslem/CRE-Router.git@main"From a clone, to edit the code or to use the files in configs/ and data/,
which the wheel does not include:
git clone https://github.com/ymoslem/CRE-Router.git cd CRE-Router pip install -e ".[full]"
full installs the whole pipeline on one machine. For a narrower install, pick
from the extras below.
| Extra | Adds | For |
|---|---|---|
| (core) | numpy, scikit-learn, sentence-transformers, pyyaml, anyio | cre cluster, cre fit; always installed |
data |
+ pyarrow | cre stats, cre compose on the released datasets |
serve |
+ litellm, fastapi, uvicorn, httpx, torch, transformers | cre serve, the Stage 1 + Stage 2 cascade router (loads ModernBERT in-process) |
qe |
+ torch, transformers, datasets, accelerate | cre qe-train, cre qe-eval |
eval |
+ vllm | cre evaluate; also provides vllm serve for backends |
full |
data + qe + serve + eval | the whole pipeline on one machine |
The vLLM backends the router talks to are separate processes started with
vllm serve <model> (cf.
Quickstart: serve CRE-Router); a
serve-only host still needs vLLM installed wherever those backends run.
Efficient QE training also needs FlashAttention
(flash-attn), installed as a second step once torch is already present
(e.g. after pip install "cre-router[qe]"):
pip install flash-attn --no-build-isolation
(the paper used flash-attn==2.8.3). The flag matters: without it, pip's
isolated build environment can't see your installed torch/CUDA, so the build
targets the wrong version. Without flash-attn at all, pass
cre qe-train --attn-implementation sdpa to fall back.
A standalone, zero-setup demo of the routing math itself, no serving and no
GPU involved. Running it reproduces the paper's routing table and $\lambda^{\ast}$
selection directly from the checked-in stats, so you can see how Stage 1
decides which model handles which cluster before setting up any backends.
It is not a prerequisite for the end-to-end GPU-based serving,
which fits its own routing table as one of its steps (cf.
Quickstart: serve CRE-Router).
The per-cluster
stats measured in the paper are checked in under
configs/, so the routing table and budgeted $\lambda^{\ast}$ reproduce
without any GPU:
cre fit --stats configs/aime_stats_1xA100_Sep2026.json --budget 30
This prints the Pareto analysis, the $\lambda$ sweep (routing regions), and the $\lambda^{\ast}$ selection. Regenerating the stats from scratch instead, by clustering and measuring each model yourself, needs a GPU: see Measuring your own pool below.
The section above reproduces a routing table from stats checked into the repo, and the one below serves a router built from one. This is the step that produces those stats in the first place: every model in the pool is run over your dataset and measured per cluster. It is the experiments path, and the slowest part of the pipeline.
Each model is measured while a vLLM server is serving it. Start the server in its own terminal, since it keeps running, with prefix caching off: every run resends the same prompts, so with caching on, runs after the first skip their prefill and the measured cost comes out lower.
vllm serve WeiboAI/VibeThinker-1.5B --port 8000 --max-model-len 42000 --no-enable-prefix-caching
Then, in another terminal:
# 1. embed and cluster the training queries cre cluster --input data/aime_train.jsonl --output artifacts/aime # 2. measure the served model (repeat per model, restarting the server for each) cre evaluate --task aime --model WeiboAI/VibeThinker-1.5B \ --dataset data/aime_train.jsonl --artifacts artifacts/aime \ --stats-out configs/aime_stats.json --runs 5 --save-generations # 3. compute the routing table, lambda* and the Stage 2 plan; it prints the # escalation target and each gated model, which needs an estimator cre fit --stats configs/aime_stats.json --budget 20 --output artifacts/aime # 4. for each gated model, build its estimator data from the generations step 2 # saved, on the training questions and on held-out ones cre qe-data --task aime --out data/qe-vibethinker \ --train <VibeThinker train *_generations.jsonl> --test <held-out *_generations.jsonl> # 5. train that estimator cre qe-train --dataset data/qe-vibethinker --output-dir qe-vibethinker
Each checkpoint from step 5 goes under qe.classifiers in the serving config,
keyed by its gated model (see the next section).
Each cre evaluate call appends that model's entry to the stats JSON, so a pool
is measured by repeating step 2 once per model. --save-generations also writes
the per-question outputs Stage 2 trains on, and is worth passing even if you only
want Stage 1 today. REPRODUCE.md carries the full worked sequence,
including the QE classifier stages.
By default cre evaluate serves each cluster as its own batch. A deployed model
receives every cluster routed to it at once, and its cost per token rises with
the context its server holds, so a batch of one cluster runs lighter than the
traffic it stands for. With --one-batch, cre evaluate serves the whole file
as one batch in the row order given, keeping each row's true cluster, and every
cost is then measured per request under that load. The sequence below fits and
measures a routing this way:
# 1. each model serves the whole training set as one batch; rows need "stage": 1 cre evaluate --one-batch --task telemath --model MODEL --dataset train_shuffled.jsonl ... # 2. per-cluster stats from those captures; --clusters prices another clustering cre stats --pool POOL --captures DIR --out stats.json cre stats --pool POOL --captures DIR --clusters train_k1.jsonl --out stats_k1.json # 3. fit; --selection exhaustive searches every assignment under the budget cre fit --stats stats.json --budget 20 --output artifacts/rt # 4. the Stage 1 batches: one per model, exactly its routed clusters cre batches --router artifacts/rt --dataset MODEL=test.jsonl ... \ --prefix p --name rt --out batches/ # 5. serve each with cre evaluate --one-batch, and score the gated models' answers # 6. the strong model's batches: its own clusters plus each run's escalations cre batches --router artifacts/rt --dataset MODEL=test.jsonl ... \ --prefix p --name rt --out batches/ --probs scores.jsonl # 7. serve them, then measure the system on every batch it served cre compose --routing batches/rt_routing.json --captures DIR --probs scores.jsonl
Step 6 needs the scores of step 5's answers, since those are the answers the system escalates from.
cre fit lists, under "Near-ties", any routing decision the training data
cannot resolve. Two options control how such ties are handled.
| Option | Compares | On a tie | Default |
|---|---|---|---|
--error-tol |
two models within one cluster, also in Pareto pruning | the cheaper model is chosen | 0.001, unless the stats file sets "error_tol" |
--crossover-tol |
the switching points of two clusters | the routing that switches the clusters in the other order is also considered, and the budget picks between them | 0 (off) |
--crossover-tol takes a number, the largest gap between two clusters' lambda
crossovers treated as a tie, or auto, which ties every gap below twice its
standard error. When the selected routing comes from a tie, no lambda reaches
it, and lambda* is reported as none. Pass --error-tol 0 for an exact
comparison within clusters.
cre fit --stats configs/teleqna_stats_1xA100_Oct2026.json --budget 20 --crossover-tol auto
This walks through standing up the live router end to end, using the paper's AIME pool as a concrete, runnable example. Swap in your own models, dataset, and config the same way once you see the shape of it.
Start one vLLM server per pool model. --max-model-len must cover input +
the task's generation length (40,960 for AIME), or long outputs truncate:
vllm serve WeiboAI/VibeThinker-1.5B --port 8001 --max-model-len 42000 vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 --port 8002 --max-model-len 42000
Cluster the training queries and write the routing table into an artifacts directory. Reusing the checked-in stats skips the GPU measurement step:
python data/download.py --dataset ymoslem/AIME-clustered --split train --output data/aime_train.jsonl cre cluster --input data/aime_train.jsonl --embeddings-field embeddings --output artifacts/aime cre fit --stats configs/aime_stats_1xA100_Sep2026.json --budget 30 --output artifacts/aime
Write a serving config. It carries only deployment wiring — the model pool
(each model's endpoint) and the QE classifier checkpoints. The routing
table, $\lambda^{\ast}$, the escalation target and the gated clusters are read from
the artifacts, never set by hand. Point
example_config_aime24.yaml
at your backends and serve:
cre serve --config example_config_aime24.yaml
TeleQnA and TeleMath configs,
example_config_teleqna.yaml
and example_config_telemath.yaml,
are also provided, and
example_config_telemath_tpot.yaml
serves the TeleMath routing at a TPOT budget of 25 ms. A model's params in
the config are laid over each request sent to it, which is how a config sets
the paper's sampling and turns Gemma 4's reasoning on or off.
Send a standard chat-completions request to the router on port 4000, not to
a vLLM backend. The endpoint is OpenAI-compatible, so an existing client
only needs its base URL changed. No model field is needed: choosing the
model per query is what the router does.
curl http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages": [{"role": "user", "content": "..."}]}'
Routing decisions are returned as x-cre-cluster, x-cre-stage1-model,
x-cre-final-model, x-cre-escalated, and x-cre-path response headers.
Streaming is not supported: Stage 2 must inspect the complete response before
deciding whether to escalate it.
Monitor the deployment. GET /health and GET /v1/models are also served.
GET /stats returns live tallies (total requests,
escalation rate, and counts per cluster, per final model, and per
escalation path). Setting decision_log: <path> in the config also appends
one JSON record per request to that file. Both are designed to add no
latency to requests: /stats is in-memory counters, and the decision log
is written by a background task fed through a non-blocking queue.
Models. Any model you can serve behind a chat-completions HTTP endpoint (the paper uses vLLM). The pool is arbitrary and can mix sizes and families; routing, Pareto pruning, and the Stage 2 plan adapt to whatever pool you measure. The paper's pools are Qwen 3 / Qwen 3.5, Gemma 4, and VibeThinker (see REPRODUCE.md).
Datasets. cre evaluate ships tasks for three benchmarks: aime and
telemath (numeric answers) and teleqna (multiple choice), each defining an
answer parser and sampling preset. A benchmark may also ship variants for a
model's thinking and non-thinking modes, differing in sampling and in whether the
prompt is pre-rendered.
max output tokens is fixed by the task, not by the caller, so every capture of
a benchmark is capped identically however it was launched. What the caller does
choose is the served context (--max-model-len), which must leave room for the
prompt on top of this cap or requests are rejected on length.
One model can defeat that: if its whole context is smaller than the task cap plus
the prompt, it cannot be served at all. Qwen3-8B on aime is exactly this, a
40,960-token context against a 40,960-token cap, leaving nothing for the question.
--max-output-tokens lowers the cap for such a run, and only lowers it. The stats
entry then records max_output_tokens, because a capture taken at a lower cap is
not comparable with the rest of the pool on either cost or accuracy.
Sampling follows Qwen's recommended settings for thinking mode (temperature 0.6, top-p 0.95) and non-thinking mode (0.7, 0.8), with top-k 20 and min-p 0 in every task. They apply to every model a task serves for consistency, including Gemma 4, whose model card recommends temperature 1.0 and top-p 0.95.
| task | mode | temperature | top-p | max output tokens | prompt in the dataset |
|---|---|---|---|---|---|
aime, telemath, teleqna_think |
thinking | 0.6 | 0.95 | 40,960 | raw, templated at serve time |
telemath_gemma4 |
thinking | 0.6 | 0.95 | 40,960 | pre-rendered, served verbatim |
telemath_nothink |
non-thinking | 0.7 | 0.8 | 16,384 | raw, templated at serve time |
teleqna |
non-thinking | 0.7 | 0.8 | 1,024 | raw, templated at serve time |
Gemma 4's thinking switch is a chat-template keyword argument, and vLLM's
benchmark loader does not forward it, so those prompts are rendered ahead of
time by prep_gemma4_thinking.py and served with
the template skipped. Mixing the two prompt kinds does not raise an error.
Serving a raw prompt under telemath_gemma4 loses the thinking switch, and
serving a pre-rendered prompt under any other task applies the template twice.
A new dataset with a different answer format needs one new
Task entry (a parser + sampling) in evaluate.py;
clustering, routing, and the QE cascade are domain-agnostic and need no
changes.
Inputs are JSONL, one object per line. Cluster ids are strings ("0", "1", ...) throughout.
cre cluster): prompt (text to embed). Any other
fields are ignored.cre evaluate): prompt (sent to the model),
answer (ground truth), and optionally cluster. Without cluster, pass
--artifacts and each row is assigned to its nearest fitted centroid.cre qe-train, cre qe-eval): question, full_output
(the model's completion), num_tokens (output length), and decision_label
(1 = accept, 0/2 = escalate). Matches the released ymoslem/*-router datasets.cre fit): JSON with cluster_sizes, an optional pool-level
error_tol, and per-model errors
and cluster_tpot_ms; see
configs/aime_stats_1xA100_Sep2026.json.
Fitting with --cost-metric e2el also needs cluster_e2el_ms.cre stats): JSON mapping each model name to its capture tag
under models, plus fields copied into the stats file (error_tol,
vllm_version); see configs/pools/.cre compose): JSON with tiers (model name to test
capture), assign (cluster to model), gated (cluster to escalation batch
stem, one batch <stem>_r<run> per run), strong, tau (default 0.5) and
runs; see configs/routings/. basis is
"per_cluster" (the default: every cluster and every run's escalations served
as their own batches) or "per_model" (every model served its routed traffic
as one batch). A per_model spec adds served (the one-batch capture of each
model routed several clusters) and strong_batch (the stem of the strong
model's batches, run r holding its own clusters and run r's escalations);
cre batches writes it.cre serve): YAML; see
example_config_aime24.yaml.An artifacts directory (written by cre cluster / cre fit) holds
centroids.npy, router.json, and train_assignments.jsonl. router.json
carries the embedding model, the routing table, $\lambda^{\ast}$, the budget, the cost
metric it was fitted under, the Stage 2 plan (escalation target and gated
clusters), and the pool stats. cre serve reads the plan back, so the served
system escalates exactly as cre fit planned.
cre-router/
├── src/cre_router/
│ ├── clustering.py Stage 1: embed queries, fit k-means centroids
│ ├── routing.py Stage 1: cost-aware routing, Pareto, lambda*
│ ├── evaluate.py Stage 1: measure per-cluster accuracy and TPOT via vLLM
│ ├── artifacts.py load/save centroids + routing table
│ ├── captures.py read saved captures: regrade answers, join cost per request
│ ├── stats.py per-cluster stats from captures (`cre stats`)
│ ├── compose.py Stage 1 and Stage 1 + 2 from served batches (`cre compose`)
│ ├── batches.py one batch per model for a fitted routing (`cre batches`)
│ ├── cascade.py per-request Stage 1 + 2 cost, used by `cre compose`
│ ├── bootstrap.py confidence intervals over questions and runs
│ ├── throughput.py whole-run and steady-state output throughput
│ ├── baselines/ FrugalGPT and HybridLLM re-implementations
│ ├── cli.py the `cre` entry point
│ ├── textutils.py shared text helpers (strip reasoning blocks)
│ ├── qe/ Stage 2: QE classifier
│ │ ├── train.py fine-tune the accept/escalate classifier
│ │ ├── classifier.py inference wrapper used by the router
│ │ ├── cascade.py replay the QE over saved generations (`cre qe-cascade`)
│ │ └── evaluate.py standalone QE metrics (`cre qe-eval`)
│ └── server/ live cascade router
│ ├── cascade_router.py Stage 1 routing + Stage 2 escalation
│ ├── app.py HTTP endpoint, /stats, decision log
│ ├── example_config_aime24.yaml serving config (AIME)
│ ├── example_config_teleqna.yaml serving config (TeleQnA)
│ └── example_config_telemath.yaml serving config (TeleMath, E2EL routing)
├── configs/ per-cluster stats for `cre fit`
│ ├── pools/ pool specs for `cre stats`
│ └── routings/ routing specs for `cre compose`
├── data/
│ ├── download.py fetch released datasets from the Hugging Face Hub
│ ├── prep_qe.py build QE train/test splits from generation logs
│ ├── prep_telemath.py fetch and split TeleMath
│ └── prep_gemma4_thinking.py pre-render Gemma 4 prompts with thinking enabled
├── img/system.svg architecture figure
├── tests/ unit tests (routing math vs paper, cascade, ...)
├── requirements-paper.txt frozen environment behind the paper's numbers
├── REPRODUCE.md reproducing the paper
├── DEVELOP.md local development and running the tests
└── README.md
For local setup and running the test suite, see DEVELOP.md.
Apache-2.0. See LICENSE.
@article{moslem2026clusterrouteescalate, title={Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving}, author={Yasmin Moslem and Magdalena Kacmajor and Vasudevan Nedumpozhimana and Ammar Abbas and Solmaz Panahi and David Lynch and Zhuangzhuang Nie and Alexandros Agapitos and Aleksandar Milenovic and Hongmeng Song and Yucheng Shi and Yue Pan and Patricia Buffini and John D. Kelleher}, year={2026}, eprint={2606.27457}, archivePrefix={arXiv}, primaryClass={cs.PF}, url={https://arxiv.org/abs/2606.27457}, }
ymoslem/CRE-Router is a public repo on GitHub by ymoslem. Cascaded Framework for Cost-Aware LLM Serving.
It declares the Apache-2.0 license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2026-11-21. Take My Repo does not judge from that whether it is maintained.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

A high-throughput and memory-efficient inference and serving engine for LLMs
Open-Source Machine Translation Quality Estimation in PyTorch

Structured multi-perspective deliberation for hard decisions. Run full councils, focused triads, or duo debates across Claude Code, Codex, Gemini CLI, and OpenCode.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Neural Machine Translation (NMT) tutorial. Data preprocessing, model training, evaluation, and deployment.
Local machine translation app for Windows and macOS. Translate text offline with NLLB-200 and other CTranslate2 models, on the CPU or an NVIDIA GPU.
Fine-tuning Open-Source LLMs for Adaptive Machine Translation
Adaptive Machine Translation with Large Language Models
This page is also available as structured data, with the same sources and dates.
/api/v1/resources/gh/ymoslem/CRE-RouterTake My Repo has not tested this on your machine. Requirements come from the project's own documentation.