File size: 6,424 Bytes

73521f8

---
license: cc-by-nc-4.0
library_name: transformers
tags:
- reinforcement-learning
- llm-routing
- cost-optimization
- tool-calling
- multi-model
pipeline_tag: text-generation
---

# xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning

<div align="center">

[![arXiv](https://img.shields.io/badge/arXiv-2510.08439-b31b1b.svg)](https://arxiv.org/abs/2510.08439)
[![GitHub](https://img.shields.io/badge/GitHub-SalesforceAIResearch%2FxRouter-blue?logo=github)](https://github.com/SalesforceAIResearch/xRouter)
[![License](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)

</div>

Welcome to **xRouter**, Salesforce AI Research's intelligent LLM routing system trained with reinforcement learning to dynamically select optimal models from 20+ available LLMs while optimizing for both performance and cost.

Modern LLM deployments face a widening cost-performance spectrum: premium models deliver strong reasoning but are expensive, while lightweight models are economical yet brittle on complex tasks. **xRouter** learns end-to-end routing policies that balance quality and cost through explicit cost-aware reward shaping, eliminating the need for hand-engineered routing rules.

## ⭐ Highlights

- **Cost-Aware Optimization**: RL-trained policies minimize costs (up to 60% reduction) while maintaining quality
- **Adaptive Routing**: Dynamic model selection based on query complexity - routes simple queries to budget models, complex ones to premium models
- **Tool-Calling Architecture**: Learns to effectively invoke 20+ models (GPT-5, o3/o4, DeepSeek R1, Qwen3, Kimi K2, etc.) and select best responses
- **Multi-Model Orchestration**: Coordinates responses from multiple LLMs for complex reasoning tasks
- **Learned Prompt Engineering**: Automatically generates optimized system prompts for target models

---

## 📊 Model Details

- **Developed by**: Salesforce AI Research
- **Base Model**: Qwen/Qwen2.5-7B-Instruct
- **Model Type**: Instruction-tuned language model with tool-calling capabilities
- **Training Algorithm**: DAPO (Distributional Advantage Policy Optimization) with cost-aware reward shaping
- **Training Data**: Derived from [Reasoning360](https://github.com/LLM360/Reasoning360) - math, code, reasoning, and STEM tasks
- **License**: CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International)

## 📈 Key Results

- **Substantial cost reductions** (up to 60%) at comparable task completion rates
- Evaluated on **17 diverse benchmarks** spanning math, coding, reasoning, and OOD tasks
- **Adaptive behavior**: Learns when to use premium vs. budget models without explicit rules
- **Multi-turn reasoning**: Effectively coordinates multiple model calls for complex tasks

For detailed results, see our [paper](https://arxiv.org/abs/2510.08439).

## 🛠️ Usage

### Installation

```bash
# Clone the repository
git clone https://github.com/SalesforceAIResearch/xRouter.git
cd xRouter

# Set up environment
conda create -n xrouter python=3.12
conda activate xrouter

pip install uv
uv pip install torch==2.6.0
uv pip install flash-attn==2.7.3 --no-build-isolation
uv pip install -e .[gpu,math,vllm,test]
pip install litellm rich python-dotenv
```

### Configure API Keys

```bash
export OPENAI_API_KEY="your_openai_key"
export TOGETHER_API_KEY="your_together_key"
export GEMINI_API_KEY="your_gemini_key"  # optional
```

### 🚀 Deployment

```bash
# Host the router model
cd evaluation
bash host_router.sh  # Serves on port 8000

# Launch the router API (in another terminal)
bash serve_router.sh  # Serves on port 8800
```

### 💬 Usage Example

```python
import openai

# Initialize client
client = openai.OpenAI(
    base_url="http://localhost:8800/v1",
    api_key="dummy"
)

# Send request
response = client.chat.completions.create(
    model="router-tool-rl",
    messages=[
        {"role": "user", "content": "Solve: If x^2 + 2x + 1 = 0, what are the values of x?"}
    ],
    max_tokens=1000
)

print(response.choices[0].message.content)

# Access routing metadata
metadata = response.router_metadata
print(f"Model used: {metadata['model_used']}")
print(f"Total cost: ${metadata['total_cost']:.6f}")
```

## 🎓 Training Methodology

xRouter uses **DAPO** (Distributional Advantage Policy Optimization) with cost-aware reward shaping:

```
reward = quality - λ × normalized_cost
```

**Training Features**:
- Cost-aware rewards penalize expensive routing decisions
- Multi-turn credit assignment across conversation turns
- Tool augmentation with 20+ model tools + response selection
- Curriculum learning from simple to complex tasks

**Supported Model Tiers**:

| Tier | Models | Best For |
|------|--------|----------|
| **Premium** | GPT-5, GPT-4.1, o3, Qwen3-235B-Instruct, Kimi K2 | Mission-critical tasks |
| **Standard** | GPT-5-Mini, GPT-4.1-Mini, o4-Mini, GPT-OSS-120B | Balanced performance |
| **Budget** | GPT-5-Nano, GPT-4.1-Nano, GPT-4o-Mini, GPT-OSS-20B | High-volume tasks |
| **Specialized** | o3, DeepSeek-R1, Qwen3-235B-Thinking, Qwen3-Coder-480B | Domain-specific |

## 📚 Citation

```bibtex
@article{qian2025xrouter,
  title={xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning},
  author={Qian, Cheng and Liu, Zuxin and Kokane, Shirley and Prabhakar, Akshara and Qiu, Jielin and Chen, Haolin and Liu, Zhiwei and Ji, Heng and Yao, Weiran and Heinecke, Shelby and Savarese, Silvio and Xiong, Caiming and Wang, Huan},
  journal={arXiv preprint arXiv:2510.08439},
  year={2025}
}
```

## 🔗 Resources

- 📄 **Paper**: [arXiv:2510.08439](https://arxiv.org/abs/2510.08439)
- 💻 **Code Repository**: [github.com/SalesforceAIResearch/xRouter](https://github.com/SalesforceAIResearch/xRouter)
- 🤗 **Model Hub**: [Salesforce/xRouter](https://huggingface.co/Salesforce/xRouter)

## 🙏 Acknowledgements

This project builds upon exceptional work from the open-source community:
- **[Reasoning360](https://github.com/LLM360/Reasoning360)**: Foundational RL training framework
- **[VERL](https://github.com/volcengine/verl)**: RL infrastructure for distributed LLM training
- **[SGLang](https://github.com/sgl-project/sglang)**: High-performance LLM serving backend
- **[LiteLLM](https://github.com/BerriAI/litellm)**: Unified API interface for 20+ LLM providers

---

_🏢 Developed by Salesforce AI Research_