Make Router Evaluation Open and Standardized
RouterArena is an open evaluation platform and leaderboard for LLM routers—systems that automatically select the best model for a given query. As the LLM ecosystem diversifies with models varying in size, capability, and cost, routing has become critical for balancing performance and cost. Yet, LLM routers currently lack a standardized evaluation framework to assess how effectively they trade off accuracy, cost, and other related metrics.
RouterArena bridges this gap by providing an open evaluation platform and benchmarking framework for both open-source and commercial routers. It has the following key features:
- 🌍 Diverse Data Coverage: A principly-constructed, diverse evaluation dataset spanning 9 domains and 44 categories with easy, medium, and hard difficulty levels.
- 📊 Comprehensive Metrics: Five router-critical metrics measuring accuracy, cost, optimality, robustness, and latency.
- ⚙️ Automated Evaluation: An automated evaluation framework to simplify the evaluation process for open-source and commercial routers.
- 🏆 Live Leaderboard: A live leaderboard to track the performance of routers across multiple dimensions.
We aim for RouterArena to serve as a foundation for the community to evaluate, understand, and advance LLM routing systems.
Important
RouterArena is an evaluation-only dataset. Submissions that train, fit, or tune any router component on RouterArena data (including the label files) will be rejected, and any accepted submission found in violation will be withdrawn. This includes:
- Routing by dataset or template. Routing must depend on the query's own content, not on which RouterArena dataset it comes from: for example, not by matching the instruction text the evaluation harness adds to each dataset, and not by reading
config/eval_config/.
- Building router data from RouterArena's templates or configs. Calibration or training prompts must not be rendered with RouterArena's prompt templates or config files.
- Reusing other submissions' answers. Every row must come from the router's own model calls, or unchanged from RouterArena's official cached answers in
cached_results/. Answers copied from other submissions' prediction files are not accepted. A router that uses the official cache must still choose the model for each query from that query alone; the cached answers and their scores must not inform the choice.
Current Leaderboard
For more details, please see our website and blog.
Under review: Paix2 (#164), R2-Router (#68) and AgentForge Router (#111) are temporarily removed from the ranking pending an integrity review. See #211, #258 and #259.
🎓 Open-source 💼 Closed-source
Evaluating Your Router
1. Setup
Step 1.1: Install uv and RouterArena
curl -LsSf https://astral.sh/uv/install.sh | sh
cd RouterArena
uv sync
Step 1.2: Download Dataset
Download the dataset from HF dataset.
uv run python ./scripts/process_datasets/prep_datasets.py
Step 1.3: Set Up API Keys (Optional)
In the project root, copy .env.example as .env and update the API keys in .env. This step is required only if you use our pipeline for LLM inferences.
# Example .env file
OPENAI_API_KEY=<Your-Key>
ANTHROPIC_API_KEY=<Your-Key>
# ...
See the ModelInference class for the complete list of supported providers and required environment variables. You can extend that class to support more models, or submit a GitHub issue to request support for new providers.
2. Get Routing Decisions
Follow the steps below to obtain your router's model choices for each query. Start with the sub_10 split (a 10% subset) for local testing. Once your setup works, you can evaluate:
- on the
full dataset for full local evaluation and official leaderboard submission.
- on the
robustness dataset for robustness evaluation.
Step 2.1: Prepare Config File
Create a config file in ./router_inference/config/<router_name>.json. An example config file is included here.
{
"pipeline_params": {
"router_name": "your-router",
"router_cls_name": "your_router_class_name",
"models": [
"gpt-4o-mini",
"claude-3-haiku-20240307",
"gemini-2.0-flash-001"
]
}
}
Every model in your config needs a price. Prices live in model_cost/model_profiles.yaml: one entry per model, keyed by its OpenRouter id, with the other spellings of the same model as aliases. If your model is listed under another spelling, add yours to its aliases. For a new model, add an entry with its price per million tokens and a link to the published price:
"openai/gpt-4o-mini":
price: {input: 0.15, output: 0.6}
aliases: ["gpt-4o-mini"]
as_of: "2026-10-31"
source: "https://openrouter.ai/openai/gpt-4o-mini"
Then regenerate model_cost/model_cost.json, which the evaluator reads, with python scripts/pricing/build_model_cost.py. A model name with no entry is charged the highest price in the table.
For a model your router runs itself (on-device or self-hosted), don't enter your own cost. Add local: {base: <id of the listed model whose weights you serve>, bits: <weight precision>}, or local: {active_params_b: <active parameters in billions>, bits: <...>} if no listed model has these weights, plus sync: false. Then run python scripts/pricing/local_price.py --write, which prices it by model_cost/local_pricing.yaml: the listed model's price, or the market price of an open-weight model of that size, times a quantization factor.
Step 2.2: Create Your Router Class and Generate Prediction File
Create your own router class by inheriting from BaseRouter and implementing the _get_prediction() method. See router_inference/router/example_router.py for a complete example.
Then, modify router_inference/router/__init__.py to include your router class:
# Import your router class
from router_inference.router.my_router import MyRouter
__all__ = ["BaseRouter", "ExampleRouter", "MyRouter"]
Finally, generate the prediction file:
uv run python ./router_inference/generate_prediction_file.py your-router [sub_10|full|robustness]
Note
- The
<your-router> argument must match your config filename (without the .json extension). For example, if your config file is router_inference/config/my-router.json, use my-router as the argument.
- Your
_get_prediction() method must return a model name that exists in your config file's models list. The base class will automatically validate this.
Step 2.3: Validate Config and Prediction Files
uv run python ./router_inference/check_config_prediction_files.py your-router [sub_10|full|robustness]
This script checks: (1) all model names are valid, (2) prediction file has correct size (809 for sub_10, 8400 for full, 420 for robustness), and (3) all entries have valid global_index, prompt, and prediction fields.
3. Run LLM Inference
Run the inference script to make API calls for each query using the selected models:
uv run python ./llm_inference/run.py your-router
The script loads your prediction file, makes API calls using the models specified in the prediction field, and saves results incrementally. It uses cached results when available and saves progress after each query, so you can safely interrupt and resume. Results are saved to ./cached_results/ for reuse across routers.
Note
- For robustness evaluation, we only measure the model-selection flip ratio after adding noise to the original prompt, so no additional LLM inference is required for this stage.
4. Run Router Evaluation
As the last step, run the evaluation script:
uv run python ./llm_evaluation/run.py your-router [sub_10|full|robustness]
Tip
- Use
sub_10 or full to evaluate on those datasets.
- Use
robustness to run robustness-only evaluation (expects <router_name>-robustness.json).
Submitting to the leaderboard
To get your router on the leaderboard, you can open a Pull Request with your router's prediction file to trigger our automated evaluation workflow. Details are as follows:
-
Add your files:
router_inference/config/<router_name>.json - Your router configuration
router_inference/predictions/<router_name>.json - Your prediction file with generated_result fields populated
router_inference/predictions/<router_name>-robustness.json - Your prediction file for robustness evaluation, no generated_result fields needed
[!NOTE]
Routers that call LLMs themselves. If your router calls an LLM to make its decision (a routing model, a classifier, voters, a critic, or the earlier steps of a cascade), record each call in the row's router_usage list as {"model": ..., "input_tokens": ..., "output_tokens": ...}, with an optional purpose. Count reasoning tokens in output_tokens. The evaluator adds these calls to the row's cost at each model's price. Check your total with uv run python scripts/pricing/compute_cost.py <router_name>.
-
Open a Pull Request to main branch and call /evaluate in the PR comment
- When the PR is ready for evaluation, call
/evaluate in the PR comment to trigger the evaluation workflow. See an example here.
- The automated workflow will:
- Validate your submission
- Run evaluation on the full dataset with the evaluation code on
main. Only the three files above are taken from your branch, plus any new models you add to the price table (model_cost/model_profiles.yaml and the generated model_cost/model_cost.json) and universal_model_names.py (models already listed keep their price on main)
- Post results as a comment on your PR
- Update the leaderboard upon approval
The Figure below shows the evaluation pipeline.
Contributing
We welcome and appreciate contributions and collaborations of any kind.
We use pre-commit to ensure a consistent coding style. You can set it up by
pip install pre-commit
pre-commit install
Before pushing your code, run the following and make sure your code passes all checks.
pre-commit run --all-files
Contacts
Feel free to contact us for contributions and collaborations.
Yifan Lu (yifan.lu@rice.edu)
Rixin Liu (rixin.liu@rice.edu)
Jiarong Xing (jxing@rice.edu)
Citation:
If you find our project helpful, please give us a star and cite us by:
@misc{lu2025routerarenaopenplatformcomprehensive,
title = {RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers},
author = {Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing},
year = {2025},
eprint = {2510.00202},
archivePrefix= {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2510.00202}
}