
The comprehensive, cutting-edge open-source framework, benchmark
and auto-research platform for next-generation Transformers
When the training budget is fixed, how should a language model spend its computation? NextTransformer trains
recurrent depth, sparse and linear attention, residual-stream variants and hybrids on the same data, for the same
49.152B tokens, at the same theoretical training FLOPs per token, and evaluates them at the same checkpoints. Every
result is an audited, file-backed record, and AI agents can research new architectures under the very same rules.
News
- 2026-11-13 · The 2K scaling study is open, starting with 16 T3 configurations across four families. Choose a configuration, find reference code and contribute.
- 2026-11-11 · The research blog is live. The first post traces how Opus found AttnRes-Canon. Blog repository
- 2026-11-04 · NextTransformer-V by @Enderfga: a community multimodal track that trains every NextTransformer method on interleaved image and text tokens under the same compute-matched contract. Its first study, pre-registered, asks where hybrid backbones get the 2D layout of an image.
- 2026-10-31 · NextTransformer is now open source: 31 finished runs, 36 model cards, the full protocol, the documentation and the observatory.
- By 2026-11-14 · Arena v2, with full support for autonomous agent research.
Highlights




The observatory

Learning curves of every run, a replayable leaderboard, all 22 CORE tasks, architecture diagrams and model cards, in
your browser.
Results

Full leaderboard · Training replay
Quick start
1. Install (Linux, NVIDIA GPUs, uv):
git clone https://github.com/Interactive-Active-AI/NextTransformer.git && cd NextTransformer
uv sync --frozen --extra gpu --extra upstream && source .venv/bin/activate
uv pip install --no-deps --require-hashes -r requirements/linear.txt # linear-attention kernels
2. Prepare the data (ClimbMix at a pinned revision, about 600 GB, plus the tokenizer and CORE bundle):
cp site.native.example.json site.local.json # set runtime_dir, resources and data_dir
hf download karpathy/climbmix-400b-shuffle --repo-type dataset \
--revision 915333b4f8b8684f39aeaafea600fea6f43fb703 --local-dir /path/to/climbmix-400b-shuffle
source launch/env.sh --site-config site.local.json
python -m nexttransformer.prepare --site-config site.local.json
3. Train on one node with 8 GPUs:
launch/train.sh --config configs/baseline/dense_full.json --site-config site.local.json \
--output runs/dense_full --micro-batch 8 --formal --resume auto
4. Multi-node training (2 nodes × 8 GPUs)
Install uv pip install accelerate==1.15.0 in the training environment. Submit one distributed job with the following per-node entry point. Map the platform's node rank, main node address and port to the three variables shown:
export FLA_DISABLE_BACKEND_DISPATCH=1 # registered FLA Triton path (linear and hybrid methods)
accelerate launch --config_file launch/accelerate.yaml \
--machine_rank "$NODE_RANK" --main_process_ip "$MASTER_ADDR" --main_process_port "$MASTER_PORT" \
-m nexttransformer.train --config configs/linear/gdn.json --site-config site.local.json \
--output runs/gdn --micro-batch 16 --formal
On Slurm, submit sbatch launch/slurm.sbatch for the same example. The batch script gets the launch parameters from the allocation. Cluster setup and other node counts
Evaluation runs on schedule during training. The quick start guide
covers smoke tests, evaluation, auditing and exporting a result.
Model zoo
The NextTransformer organization on Hugging Face is the home for model weights. Check its Models tab for available downloads. The leaderboard lists evaluated runs, including those whose weights have not yet been released.
Contributor-hosted weights: AttnRes-Canon (49B comparison) · Dense (200B-token scaling run).
Browse architectures by family
| Family |
Models |
|
Dense Baseline |
|
Standard Residual · AttnRes · HC · mHC · MSAR · AttnRes-Canon · SimpleHC |
|
UT-ACT · Fixed Recursive · Huginn · Ouro · DeepLoop · MoR · LoopFormer · Circle Loop |
|
DSA · QSA · NSA · SeerAttention-style · MoBA |
|
DeltaNet · GSA + QK-norm · GLA · KDA · RetNet · ELU-Norm · GDN |
|
GDN–MHA 3:1 · GDN–MHA 1:1 · GDN–SWA · DeltaNet–MHA · KDA–MLA-NoPE · GDN–QSA · MoBA–MHA |
|
SSSL · MQA · GQA · MLA-QK · MLA-NoPE-QK · Gated Attention · elementwise · Gated Attention · headwise · XSA · Exclusive Self Attention · Inkling REL |
* Results coming soon. Papers, adaptations and configurations are in the model zoo.
The protocol at a glance
|
|
| Data |
ClimbMix at a pinned revision · one 32,768-token tokenizer · 2,048-token context · one global sample order |
| Budget |
49.152B input tokens · 46,875 updates of 1,048,576 tokens · seed 42 |
| Compute |
4.672 GFLOPs per input token (forward, backward, auxiliary and optimizer work); ±1% at admission, ±2% over the run |
| Evaluation |
Validation BPB at 189 points · full CORE-22 at six checkpoints · an independent final BPB |
| Baseline |
Dense: full causal attention in every layer, trained under the same budget |
Parameters, memory and speed are outcomes, not controls. The protocol
explains every rule and the compute accounting
what is counted.
Add your method
New methods and Arena entries follow the same submission process.
- Open a proposal issue to discuss the method and run plan.
- Implement the plugin on your fork, calibrate it to the shared budget, and complete a
--formal run.
- Once you have audited and exported the result, open one pull request to upstream
main with the implementation and result. Link the proposal issue.
Start with the plugin guide and
CONTRIBUTING.md.
Auto-research arena

Note
The arena is in active development. Arena v2, with full support for autonomous agent research (a research gym
with MCP tools, metered compute, research traces and self-improving research loops), ships by November 14, 2026.
Any LLM, with or without a research framework, can already enter an architecture today: it is trained under the same
protocol, ranked on the same leaderboard, and disclosed with how it was found. See the
arena.
Built on NextTransformer
Community projects that extend NextTransformer.
by @Enderfga
Every NextTransformer method on frozen SigLIP2 image tokens interleaved with text at an 8,192-token context, compute-matched within 1%. Its first study, pre-registered, asks where hybrid backbones read the 2D layout of an image.
|
by AI4AI for Anything, contributed by @henry-y
An agent that profiles, optimizes and validates AI training code. It contributed NextTransformer's opt-in --speedups (#26): fused kernels for the shell every method shares. Its exact mode reproduces the default path bit for bit, formal runs accept it, and it trains dense_full 1.38× faster on one eight-GPU node.
|
Your project here
Built something on NextTransformer's code or protocol? Share it in Show and tell or open a pull request that adds a card with a wide banner image.
Listed projects are public and say in their README that they build on NextTransformer.
|
Roadmap
- By November 14, 2026: Arena v2 and agent-native research.
- 2K scaling: start with the T3 size sweep, refine the fits, check preregistered forecasts, then hold out C200 for the final test.
- Hybrid GDN + QSA: completed formal results, cumulative audit and run notes.
- Planned: checkpoints and the tokenizer on the Hugging Face Hub.
More on the roadmap.
Community
Contributions are welcome; please read CONTRIBUTING.md and the code of conduct.
Community projects are listed under Built on NextTransformer.
Citation
@software{nexttransformer2026,
title = {NextTransformer: An Open Framework and Fair Benchmark for Next-Generation Transformer Architectures},
author = {{NextTransformer contributors}},
year = {2026},
version = {0.1.0},
url = {https://github.com/Interactive-Active-AI/NextTransformer}
}
Acknowledgments
NextTransformer builds on nanochat and ideas from
modded-nanogpt, uses
flash-linear-attention kernels, and trains on
ClimbMix. We thank the authors of every method in the
zoo. Released under the MIT License; see third-party notices.