Cooperative Multi-LLM Reinforcement Learning (CoMLRL) is an open-source library for training multiple LLMs to collaborate using Multi-Agent Reinforcement Learning (MARL). It provides implementations of various MARL algorithms for LLM collaboration and support for different environments and benchmarks.

Installation
CoMLRL can be installed from PyPI, conda-forge, or source and supports Python 3.10 or later. Users should also install a PyTorch build compatible with their device.
git clone https://github.com/OpenMLRL/CoMLRL.git
cd CoMLRL
pip install comlrl # install from PyPI
conda install -c conda-forge comlrl # install from conda-forge
pip install -e . # install from source in editable mode
Features
Usage
Quick start by training two Qwen-2.5 agents on Reddit prompts with MAGRPO. This toy reward encourages a 3:1 output-length ratio; it does not measure summary quality:
from datasets import load_dataset
from transformers import AutoTokenizer
from comlrl.trainers.reinforce import MAGRPOConfig, MAGRPOTrainer
# Load dataset and tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
dataset = load_dataset("trl-lib/tldr", split="train").select(range(128))
# Initialize trainer and start training
trainer = MAGRPOTrainer(
agent_model="Qwen/Qwen2.5-0.5B",
num_agents=2,
tokenizer=tokenizer,
train_dataset=dataset,
reward_func=lambda a, b: [
-abs(len(second) / max(len(first), 1) - 3.0)
for first, second in zip(a, b)
],
formatters=[lambda example: example["prompt"]] * 2,
args=MAGRPOConfig(num_turns=1),
)
trainer.train()
Contributing
We thank the gracious help of all contributors:
๐ค: Foundational Ideas; ๐ง: Maintenance; ๐ป: Code; ๐: Documentation; ๐: Bug Report.
For new contributors, please see contributing guidelines on setting up a dev environment.
Sponsorship
CoMLRL was developed using substantial computational resources. Its growth has been made possible by the generous support of the following organizations and institutions.
We welcome computational sponsorship to support the continued development of CoMLRL. If you are interested in supporting this project, please contact us.

Citation
Please cite the following papers if you find this library useful in your research:
@inproceedings{liu2026llm,
title={LLM Collaboration with Multi-Agent Reinforcement Learning},
author={Liu, Shuo and Liang, Zeyu and Lyu, Xueguang and Amato, Christopher},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={40},
number={38},
pages={32150--32158},
year={2026}
}
@inproceedings{liu2026learning,
title = {Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic},
author = {Liu, Shuo and Chen, Tianle and Amiri, Ryan and Amato, Christopher},
booktitle={Proceedings of the International Conference on Machine Learning.},
year = {2026}
}
@article{liu2026improving,
title={Improving LLM Collaboration via Multi-Agent Preference Learning},
author={Liu, Shuo and Li, Xinzichen and Chen, Tianle and Amato, Christopher},
journal={arXiv preprint arXiv:2609.32827},
year={2026}
}