1. Introduction
Reinforcement learning is now a standard step in post-training language models. The model attempts a task, a reward scores the attempt, and the update makes high-scoring attempts more likely. When a program computes the reward, such as a math checker, a test suite or a database query, the setup is called RL with verifiable rewards (RLVR). It is how reasoning models like DeepSeek-R1 were trained (paper), and the task, its tools and its reward together are called an environment. GRPO (paper) is the most common algorithm: it samples a group of answers per prompt and pushes up the ones that score above the group’s average.
Generation is usually the slow part. A math completion can run to 16k tokens, and an agent rollout is multi-turn: the model works inside a harness, calls tools and reads their results, over a long horizon. In synchronous GRPO the same GPUs take turns generating and training, so the trainer sits idle for most of the run. Most modern RL implementations now disaggregate the two: inference runs on its own GPUs and the trainer on others, connected by a queue of finished rollouts, with a weight sync after each step. The tradeoff in this design is handling staleness and policy drift, since a sample can come from weights a few steps older than the ones being trained. There is also a trainer-inference mismatch: an inference engine like vLLM samples the tokens and a different library trains on them, and their kernels give slightly different log-probs for the same tokens.
In March we wrote about how async RL is implemented in the wild, across 16 open-source libraries (async RL training landscape). Our own async trainer, AsyncGRPOTrainer, is now stable in TRL v1.18.0 (launch post). This post is the benchmark we ran before calling it stable: we compared it against the main open-source RL libraries: prime-rl, verl, SkyRL, OpenRLHF, miles and NeMo-RL, on our internal B200 cluster.
The benchmark varies two things:
- The model size, from small to large and from dense to mixture-of-experts (MoE): Qwen3.5-2B and 9B (dense) and Qwen3.5-35B-A3B (MoE).
- The workload, from single-turn to agentic: math with one long completion, terminal tasks where an agent works in a sandbox through a coding-agent harness, and text-to-SQL with tool calls, a harder use case for the 35B.
We couldn’t run every library end to end on every model and workload. Some libraries failed to install, ran out of memory, or computed wrong numbers on our hardware (section 7), and we were limited in time and compute, so every run has one seed and the results only compare the runs that trained. All of it, failed attempts and smoke tests included, took about 2,770 B200 GPU-hours over 280 GPU jobs. At AWS’s on-demand price of $14.24 per B200 GPU-hour, that’s about $39,400 in total. Every config, launcher and log is in the benchmark repo (Appendix C). If you can make one of the failed runs work, tell us and we’ll rerun it. Tap a cell to see what happened there.
Only TRL, prime-rl and SkyRL have a training run in every column.
The results per workload, with TRL in bold:
- Scores are level. On every workload, TRL lands inside the range of the other libraries, and the differences are within each eval’s noise.
- Steps are bigger. By default TRL sizes a step by a token budget instead of a count of samples, which keeps memory per GPU tight and predictable. On math that budget made a TRL step twice the tokens of a prime-rl step. Per useful token the two are level, but prime-rl reaches the same scores in half the wall-clock (section 4.5). A later post will cover token-budget batching.
Section 2 covers the parts of the trainer the results depend on. Section 3 covers the setup. Sections 4, 5 and 6 give the results: math, terminal agents and text-to-SQL. Section 7 lists the failures.
2. AsyncGRPO in one picture
The launch post covers how to install, run and customize AsyncGRPOTrainer. This section covers only the parts of its design that the results below depend on.
In our runs a node’s 8 GPUs are split in two: 4 serve vLLM and 4 train with FSDP2, plus expert parallelism (EP) for the MoE. A rollout worker sends generation requests to vLLM, scores each group of num_generations completions, and queues the samples, each tagged with the weight version that generated it. The trainer pulls from the queue, packs, takes an optimizer step, and syncs its weights back to vLLM. Neither side waits unless the queue is empty or generation is too far ahead.
Staleness. A sample’s staleness is the number of weight updates between the weights that generated it and the ones being trained. max_staleness (4 by default) caps it in two places: generation is held back so it can’t run more than max_staleness steps ahead, and the trainer drops any sample older than that when it pulls it. The loss takes vLLM’s log-probs as the old policy, so one importance ratio covers both staleness and the vLLM-trainer numerical gap.
Token-sized steps. Each training GPU forwards one packed row of up to token_budget tokens per micro-batch, and gradient_accumulation_steps micro-batches make a step. So TRL can’t set the shared config’s 32 prompts x 8 samples exactly (section 3.6), and we set the token budget to approximate it. Steps still differ in size between libraries: the batch charts in Appendix F show samples and tokens per step for every run.
Same-reward groups. If every completion in a group gets the same reward, all its advantages are zero. The rollout worker drops these groups before the queue and fills the step with groups that carry signal, as in DAPO’s dynamic sampling (paper). It logs their share as frac_zero_std_groups. verl, OpenRLHF and SkyRL keep such groups in the loss by default, which is why section 4 compares throughput per useful token.
Tree packing. A packed row is a prefix trie: a token shared by several samples is forwarded once, and on Qwen3.5’s linear-attention layers no state crosses between samples. The docs cover how to use it; a separate post on how tree attention works is coming.
Score centering is the default objective (score_centering=True). At every position it subtracts the sampler’s expected score, estimated over vLLM’s 8 most likely tokens (num_sampler_logprobs, paper), so vLLM returns 8 log-probs per generated token. Our shared-config runs use the clipped objective instead (epsilon=0.2, epsilon_high=0.28). Appendix D gives the step time with score centering on.
Weight sync happens after every step (weight_sync_steps=1): the trainer pauses vLLM and sends the new weights over NCCL. On 2B math with score centering off, that took 0.22 s per step. Requests in flight keep their KV cache through the pause and finish on the new weights, and every request carries the weight version as a cache_salt, so a new version never reuses prefix-cache blocks computed by older weights (#8300).
The terminal-agent runs use HarnessAsyncGRPOTrainer, which stays in trl.experimental and may change (section 5.2).
3. Benchmark setup
We ran seven libraries on the same hardware, the same model family and the same three environments. This section explains how we ran them, then goes through the libraries, the model, the environments, the training configs and how we scored the results.
3.1 How the benchmark was run
We set the benchmark up the way many people now start an RL run: they point a coding agent at a library and an environment, and ask it to get training working. That is a realistic test of a library (Appendix C has one of the briefs we gave the agents). The agent reads the same docs, examples and error messages a user would, and runs into the same problems. We wanted to measure that, while keeping control of the parts that make a comparison fair: the rules on what a run may change, the configs, the environments and the evaluation. So the agents did the hands-on work, and we set the frame and checked every step that mattered:
- We picked the models (Qwen3.5 2B, 9B and 35B-A3B) and the three environments, and wrote one shared training config (section 3.6).
- We set what a run may change: only the options each library ships, with no source patches (section 3.2).
- One Claude Code session, the coordinator, started one subagent per library: TRL, prime-rl, verl, SkyRL, miles and OpenRLHF. A NeMo-RL subagent joined later, first for the 35B, then for math and terminal agents.
- Each subagent installed its library from the docs, with
uvin its own locked environment. miles and NeMo-RL came from their own container images. - Each subagent checked on a B200 that the library runs Qwen3.5 correctly: the fast kernels are used, and packed samples stay separate.
- Each subagent mapped the shared config onto the library’s options and wrote down every setting it couldn’t express. We read each mapping before the run started.
- Each subagent launched its runs and logged every problem and its fix.
- The coordinator ran the job queue (at most 4 nodes at once) and kept a dated log of every decision. We decided when to stop a run.
- One eval script per workload scored every library’s checkpoints.
- At the end, a separate subagent checked this post against the raw files.
The problem log has 159 rows: 39 library failures, 38 mistakes of ours, and 82 cluster problems, generic Qwen3.5-on-B200 problems and notes. The subagents logged every problem, however small, so the counts in section 7 are high for every library.
3.2 Rules
- Built-in options only. Each library runs with the options it ships: no custom losses, no plugin overrides, no source patches. Where it can’t express part of the shared config, it uses its closest built-in option and we note the gap.
- A failure is a result. When a library’s own option breaks, that’s its result. We don’t patch it and rerun.
3.3 Libraries
| library | version | trainer / inference | max staleness |
|---|---|---|---|
TRL AsyncGRPOTrainer | v1.18.0 | FSDP2 + vLLM 0.33.0 | max_staleness=4 (8 on text-to-SQL), weight sync every step |
| prime-rl | main @cbd3702 | FSDP2 + vLLM 0.32.0 | max staleness 8, stale samples masked |
| verl | v0.9.1 | FSDP2 + vLLM 0.24.0, separate_async | at most 8 versions behind, older dropped |
| SkyRL | main @d8700468 | FSDP2 + vLLM 0.33.0, fully async; Megatron + vLLM, colocated, on the 35B | staleness up to 4 (0 on the 35B) |
| miles | v0.1.3 (v0.2.0 for text-to-SQL) | Megatron + SGLang 0.5.23 | no cap (measured up to 4) |
| OpenRLHF | v0.11.2 | DeepSpeed ZeRO-2 + vLLM 0.29.0 | queue of 1, staleness exactly 1 |
| NeMo-RL | v0.8.0 | Megatron (math) or Automodel (35B) + vLLM 0.29.0 | async on math, at most 1 step behind; colocated on the 35B |
Each library makes its own choices: training backend, inference engine, how far generation may run ahead, and which workloads it ships recipes for. Each one runs its own recommended inference stack: four vLLM versions and SGLang.
3.4 Model: Qwen3.5
Every run uses Qwen3.5, a recent open model family that goes from a 2B dense model to a 35B mixture-of-experts. It is a hybrid: three out of four layers are gated DeltaNet layers, a form of linear attention, and the fourth is full attention.
Linear attention keeps a recurrent state, which makes packing harder: a trainer that packs several samples into one row has to reset that state at each sample boundary (section 7.2). TRL’s tree packing does this per trie segment.
The checkpoints are vision-language models. We train the text part and keep vision frozen. We use three sizes:
- Qwen3.5-2B for math. Cheap enough to run every library twice, with room to improve: the base scores 0.212 on AIME24, 0.171 on AIME25 and 0.776 on MATH-500.
- Qwen3.5-9B for terminal tasks, and as a second math size. On math the base already scores 0.700 on AIME24, so there is little left to learn; those runs are in Appendix G. On terminal tasks it’s the smallest size where the base model succeeds often enough to train: 0.386 of episodes on the candidate tasks.
- Qwen3.5-35B-A3B, a mixture-of-experts, with LoRA. A mixture-of-experts routes each token through a few of many expert MLPs (3B of the 35B parameters are active per token). Fitting it on one node takes expert parallelism: each GPU holds a share of the experts. Full fine-tuning would need an estimated 140 GB per trainer GPU before activations, so we train rank-32 LoRA adapters on a bf16 base.
3.5 Environments
We picked three environments to cover two things: the type of workload (one long completion, a long agent episode, short tool calls) and a difficulty that fits the model size. An environment is useful for RL only when the base model succeeds some of the time and has room to improve.
Why these three:
- Math is the standard GRPO workload, and every library ships a recipe for it. It tests long completions and raw trainer throughput. The 2B base solves about one AIME problem in five, so it has a lot to learn.
- Terminal agents test what RL libraries are now used for: long multi-turn episodes with a real agent harness and sandboxes. The 9B is the smallest size that solves some tasks. We kept the tasks the base 9B solves 20 to 80% of the time.
- Text-to-SQL is there because the 35B saturates the terminal tasks: the base model already passes them as often as any trained run (Appendix E). Text-to-SQL is miles’
examples/text_to_sqlrecipe, which has a published reward climb at this model size.
Details per environment:
- Math. The prompt is the problem plus “Please reason step by step, and put your final answer within
\boxed{}.”math_verifychecks the last boxed answer. Completions are capped at 16,384 tokens with thinking off: with thinking on, 98.9% of the base 2B’s completions hit the cap. - Terminal agents. Each task is a container, an instruction and a test script. The agent, terminus-2, works in the container’s terminal for up to 30 model calls and 600 seconds (section 5.1). From 128 candidate tasks we removed 15 that pass when the agent does nothing, then kept 42 in the 20 to 80% band: 28 for training, 14 held out.
- Text-to-SQL. The model gets a question about a database, can run up to 5 read-only
execute_sql_querycalls, then answers with a query in a<solution>block. The training reward also has small penalties for skipped evidence rules and invalid tool calls. Evals score correctness only. A third split, hard (658 questions), overlaps train, so validation and arcwise are the held-out numbers.
3.6 Configs
On math and terminal tasks, every library maps one shared config onto its own options:
| setting | shared value |
|---|---|
| data, prompt, reward | the same for every library |
| sampling | temperature 1.0, no top-p or top-k |
| batch per step | 32 prompts x 8 samples on math, 8 tasks x 8 episodes on terminal tasks |
| optimizer | AdamW, constant 1e-6, gradient clipping 1.0, one step per batch |
| loss | GRPO, group-mean advantage, DAPO’s Clip-Higher (0.2 / 0.28, paper) against the inference engine’s log-probs, token-level, no KL |
| groups where every sample gets the same reward | kept in the batch. They give no gradient, and no extra groups are generated to replace them |
| length | 200 steps on math, 60 on terminal tasks |
Each library keeps its own async setting and inference engine. Not every setting can be matched. This is what each library ran on 2B math:
| library | loss | batch per step | same-reward groups | packing | max staleness |
|---|---|---|---|---|---|
| TRL | PPO clip 0.2 / 0.28 | token budget, about 2.7M tokens | dropped, step filled from groups with signal | tree packing | 4 |
| prime-rl | its default IPO mask (no built-in PPO clip) | 32 x 8 | kept, excluded from the loss | packed | 8 |
| verl | PPO clip 0.2 / 0.28 | 32 x 8 | kept | packed | 8 |
| OpenRLHF | PPO clip 0.2 / 0.28 with its token-level IS mask | 32 x 8 | kept | off (leaks on Qwen3.5, section 7.2) | 1 |
| SkyRL | its default dppo (fully async mode doesn’t accept the PPO clip loss) | 32 x 8 | kept | off, padded | 4 |
miles has no finished math run (section 7.2).
The shared config pulls every library off its own defaults, and some can’t express all of it: prime-rl has no built-in PPO clip, and SkyRL’s fully async mode doesn’t accept the PPO clip loss. So on 2B math we also ran each library the way its authors recommend, a recommended config:
- It keeps everything that defines the task: data, prompt, reward, hardware, eval and number of steps.
- Every training setting the shared config overrode goes back to the library’s own default: loss, objective, KL, and what happens to same-reward groups.
- We didn’t tune anything: a default is whatever the library does when you don’t set the option.
Compared with its shared run:
- TRL turns on score centering, its default objective.
- prime-rl turns on refill: when every sample in a group gets the same reward, it generates new groups to replace it, so every step trains on 256 rollouts that carry a gradient.
- OpenRLHF adds a KL loss against a frozen reference model.
- SkyRL changes nothing: its shared run already uses its default loss.
The 35B text-to-SQL has no shared config. Each 35B run takes 8 to 12 node-hours, so we could afford one run per library, and we used it to put each library on its best footing: its own recipe, on the same task data, reward and eval (section 6.2).
3.7 Hardware and evaluation
Every run gets one whole 8xB200 node (180 GB per GPU): 4 GPUs for inference and 4 for training, so seconds and memory compare directly. On the 35B, SkyRL’s recipe shares all 8 GPUs between training and inference (section 6.2).
We score every library’s saved checkpoints ourselves, with one eval script per workload:
- Math: AIME24 and AIME25 (avg@16) and MATH-500, sampled with vLLM, up to 16k tokens, thinking off.
- Terminal tasks: 14 held-out tasks, 12 episodes each per checkpoint, on local containers.
- Text-to-SQL: 398 validation and 498 arcwise questions, greedy decoding, one try per task.
So every library had to save a complete Hugging Face checkpoint that vLLM can load, and some needed the frozen vision weights copied back in.
3.8 What we measure
Training reward per step shows whether a library learns on the environment at all: a broken loss, a collapse or a reward bug shows up there first. It’s measured on training data, so it can rise without the model getting better on new problems. The held-out score checks that: it measures whether training generalizes to new problems, in domain or out of domain. It’s the number a user cares about, but in some setups our eval sets are small, so differences under a few points are noise (Appendix B).
Step time is how long an optimizer step takes. We also split each step into forward and backward, waiting for rollouts, weight sync and other, which shows whether a library is limited by its trainer or by generation. Steps hold different amounts of data in each library, so we also divide by the tokens a step trains on and report seconds per million tokens.
Time to reward is the wall-clock hours until the training reward first reaches a level. It combines speed and how much each step learns, so it’s a good measure of how fast a run converges. Take it with a grain of salt: it’s training reward, so read it next to the held-out scores.
Peak memory per GPU decides the largest model, context and batch a library can fit on one node, and whether a long agent episode runs out of memory (section 7.3). Finally, we log every failure we hit while getting a run to work, as a proxy for what a user will run into (section 7).
4. Math: DAPO-Math-17k
Every library trains Qwen3.5-2B on DAPO-Math-17k for 200 steps of 32 prompts x 8 samples. One eval script scores every library’s checkpoints at steps 50, 100, 150 and 200.
4.1 Shared and recommended configs
We ran 2B math twice per library (section 3.6). The shared config puts every library on the same settings, so differences come from the trainer. The recommended config runs each library with its own defaults, to check that the shared settings didn’t hold one back. Each chart has a Recommended config tab first and a Shared config tab second. They open on the shared config, except time to reward (section 4.5), which opens on the recommended one.
Held-out scores in the recommended config:
| library, recommended config | AIME24 | AIME25 | MATH-500 |
|---|---|---|---|
| TRL | 0.494 | 0.363 | 0.912 |
| OpenRLHF | 0.460 | 0.373 | 0.922 |
| prime-rl | 0.452 | 0.329 | 0.910 |
| SkyRL (same run as its shared config) | 0.410 | 0.329 | 0.916 |
- The table shows TRL’s first recommended run. A second run, with its batching aligned to the shared config’s step size, finished at 0.402 / 0.354 / 0.928. That 0.09 gap on AIME24 between two TRL runs is larger than the gaps between libraries in this table, so single runs here are within noise of each other.
- TRL’s recommended step takes 65.6 s, of which 19.4 s is the score-centering weight-sync stall (Appendix D). With centering off it’s 0.22 s.
- verl’s recommended run needed a library patch, miles has no as-shipped run, and NeMo-RL ran only the shared config, so none of them appears.
4.2 Training reward
Mean training reward per step on the shared config, Qwen3.5-2B. For every library this is the mean over all scored rollouts, same-reward groups included: for TRL that’s its reward metric, not reward_trained, which covers only the groups it trains on.
Every library learns, at about the same rate: mean reward goes from 0.40-0.47 over the first 10 steps to 0.56-0.62 over the last 10.
4.3 Held-out scores
Held-out scores on the shared config (section 3.6).
| library | loss | AIME24 | AIME25 | MATH-500 | hours, first to last step |
|---|---|---|---|---|---|
| OpenRLHF | shared | 0.404 | 0.365 | 0.908 | 3.93 |
| TRL | shared | 0.388 | 0.304 | 0.904 | 2.80 |
| prime-rl | shared | 0.352 | 0.302 | 0.900 | 1.41 |
| verl | shared | 0.350 | 0.300 | 0.892 | 2.67 |
| NeMo-RL | shared | 0.317 | 0.288 | 0.890 | 6.25 |
| SkyRL | its default (dppo) | 0.410 | 0.329 | 0.916 | 3.13 |
| base Qwen3.5-2B | 0.212 | 0.171 | 0.776 |
- Each library mapped the shared loss onto its closest built-in option: PPO clip 0.2 / 0.28 for TRL, verl, OpenRLHF and NeMo-RL, the IPO mask for prime-rl, dppo for SkyRL.
- We skipped miles’ results here: its flash-attention path gives wrong log-probs for Qwen3.5 on B200 (section 7.2), and the correct unfused backend was projected at 14-16 hours.
4.4 Step time
A TRL math step trains on about 2.7M tokens, a prime-rl step on 1.37M, so seconds per step don’t compare directly. Per useful token the two are level: 53.5k useful tokens per second for TRL, 54.2k for prime-rl. verl, SkyRL, OpenRLHF and NeMo-RL train 12.2k to 31.9k. How each library fills a step, and the full throughput table, are in Appendix F.
- Mean step time: prime-rl 25.2 s, verl 48.0 s, TRL 51.0 s, SkyRL 56.2 s, OpenRLHF 71.0 s, NeMo-RL 112.9 s.
- NeMo-RL barely waits for rollouts (0.26 s per step): its step is 79 s of training plus 26 s recomputing log-probs, on its Qwen3.5 recipe’s Megatron setup with micro-batch 1 and no packing.
- OpenRLHF doesn’t log rollout wait or weight sync, so 40 s of its step shows as “other”. We didn’t look into SkyRL’s 14.3 s of weight sync.
- TRL’s 15.1 s of “other” is rank 0 packing micro-batches in Python on CPU. A numpy rewrite cuts it to 1.35-1.63 s; it isn’t in TRL v1.18.0.
4.5 Time to reward
Best training reward so far against wall-clock hours.
- Recommended config, at 0.60: SkyRL 0.80 h, prime-rl 0.90 h, TRL 0.94 h (step 54), OpenRLHF 5.26 h. On its own defaults TRL is level with the fastest libraries.
- Shared config, at 0.60: prime-rl 0.45 h, SkyRL 0.80 h, OpenRLHF 1.30 h, TRL 1.50 h, verl 1.73 h, NeMo-RL 3.43 h. prime-rl reaches every reward level first.
- prime-rl reaches the same step-200 evals in half the wall-clock (1.41 h vs TRL’s 2.80 h). TRL’s extra tokens per step didn’t improve the held-out scores.
- TRL needs more steps to reach 0.60 (110, vs 60 for prime-rl). This gap is in training reward only: the held-out scores at step 200 don’t show it.
4.6 Memory
Peak memory decides how much model, context and batch a library can fit on a GPU. Memory a trainer doesn’t need is room for a bigger model or longer episodes on the same node (section 3.8).
The charts show two peaks per trainer GPU:
- nvidia-smi peak (bar): everything the process holds on the GPU. That includes memory PyTorch’s caching allocator has reserved but isn’t using, plus CUDA and NCCL buffers. It’s what’s left for anything else on that GPU.
- torch peak (diamond): the most memory live tensors used at once, as the library logs it. It’s closer to what the trainer actually needs.
The gap between the two is mostly allocator cache.
- Inference GPUs sit at 167-180 GB, because vLLM preallocates its KV cache. NeMo-RL’s sit at 157 GB: it reserves 85% of the GPU.
- Trainer peaks: prime-rl 26.3 GB, TRL 41.4 GB, SkyRL 42.1 GB, NeMo-RL 43.8 GB, OpenRLHF 70.4 GB.
- verl shows 191.5 GB in nvidia-smi against a torch peak of 31.6 GB. We didn’t look into it; compare the torch peak.
We also ran the math workload on Qwen3.5-9B; those results are in Appendix G.
On math, TRL scores in the same range as the other libraries, and its useful tokens per second are level with prime-rl’s. prime-rl reaches the same 2B scores in half the wall-clock.
5. Terminal agents: TermGrade
This workload is closer to what agentic coding benchmarks measure. The model doesn’t answer in one completion: it drives an agent that works in a terminal inside a sandbox, reads files, runs commands and edits code, for up to 30 model calls and 10 minutes. At the end, the task’s hidden tests decide the reward: 1 if they pass, 0 if not.
We replicated the setup you would use to train a model with RL inside an existing agent harness. The tasks are in the Harbor format, the agent is terminus-2, Harbor’s default agent, and every episode gets its own container. The trainer only sees the model calls and the final reward. This is the hard case for an RL library: episodes are long and uneven, most of the time goes to waiting on sandboxes and tools, and one episode can produce two dozen model calls that share most of their context. We trained Qwen3.5-9B; five libraries made it through.
A TRL step takes 199 s, between prime-rl’s 185 s and verl’s 206 s. Every library that trained ends above the base model on held-out tasks, too close to rank. TRL ran on its experimental harness trainer with 4 Harbor servers instead of one.
5.1 The task
We used the Harbor task format: each task is an instruction.md, a container image and a pytest verifier the agent never sees. The tasks come from TermGrade, 1,004 terminal tasks in that format. The agent is terminus-2, Harbor’s default.
- Episode: each turn sends the whole chat to the model and types its keystrokes into tmux, until the model says it’s done, 30 calls or 600 s. A typical episode makes 23 to 24 calls and ends with a 30k-token context.
- Limits: 8,192 tokens per call, 65,536-token context, thinking and summarization off.
- Sandboxes: one per episode on the training node, Apptainer for TRL, verl and SkyRL, rootless podman for prime-rl.
- Tasks: 28 training and 14 held-out tasks the base 9B solves 20 to 80% of the time (section 3.5).
- Run shape: 60 steps of full fine-tuning, at most 64 episodes in flight, one node with 4 GPUs for vLLM and 4 for the trainer.
5.2 How each library runs an episode
The harness is the multi-turn infrastructure between the agent and the trainer. Each library answers three questions differently: where terminus-2 runs, how its model calls reach vLLM, and what becomes a training sample. TRL and verl use OpenEnv’s openenv harbor serve, a server that runs the agents and puts a token-capture proxy between them and vLLM, so the trainer gets the engine’s exact token ids, log-probs and loss mask.
- TRL: terminus-2 runs on the host inside 4 OpenEnv Harbor servers of 16 episodes each. The proxy sends chat requests to vLLM, and each model call becomes its own sample. The trainer is
HarnessAsyncGRPOTrainer(harness path), still undertrl.experimentalin TRL v1.18.0. - prime-rl: terminus-2 runs inside every podman box, spread over 64 processes. An interception server turns each chat request into token ids for vLLM and appends sampled tokens as they come, so an episode stays one sequence.
- verl: no Harbor integration, but an agent-loop API. Our 300-line
AgentLoop, with no change to verl, drives 8 OpenEnv Harbor servers and routes calls through verl’s token-in, token-out generation, so an episode is also one sequence. - SkyRL: its own Harbor generator runs every episode as an asyncio task inside one Ray actor with one CPU and calls vLLM through LiteLLM. One sample per model call.
- NeMo-RL: NeMo Gym’s
harbor_agentruns terminus-2 in local Apptainer boxes and calls NeMo-RL’s vLLM through its OpenAI server. NeMo-RL’s on-policy token correction keeps each episode one sequence. The diagram above doesn’t show it: NeMo-RL joined after it was drawn. - What we added: for TRL and verl, a launcher with several servers and the 30-call cap passed to terminus-2 (TRL v1.18.0’s example starts one server). For prime-rl, a two-line podman wrapper.
5.3 Training reward
Mean training reward per step, Qwen3.5-9B, 8 tasks x 8 episodes per step.
5.4 Held-out scores
Every checkpoint was scored with TRL’s eval script: 14 held-out tasks, 12 episodes each, on local boxes behind 4 Harbor servers with the call cap.
| checkpoint | TRL | prime-rl | verl | SkyRL | NeMo-RL |
|---|---|---|---|---|---|
| base Qwen3.5-9B | 0.301 | 0.301 | 0.301 | 0.301 | 0.301 |
| step 20 | 0.386 | 0.337 | 0.369 | 0.417 | 0.395 |
| step 40 | 0.417 | 0.405 | 0.370 | 0.417 | 0.353 |
| step 60 | 0.443 | 0.411 | 0.429 | 0.491 | 0.416 |
- Every trained run ends above the base. Paired with the base task by task, step-60 gains are +0.11 to +0.19 for TRL, prime-rl, verl and SkyRL; only prime-rl’s is under 3 standard errors. NeMo-RL ends at +0.115 (not paired).
- The libraries can’t be ranked on 14 tasks. The standard error over per-task pass rates is 0.063 to 0.087, and the step-60 spread (0.411 to 0.491) is about one of them.
- This is the harness TRL and verl trained with; we didn’t measure what it changes for prime-rl and SkyRL.
5.5 Step time
TRL, prime-rl and verl land within 11% of each other per step. SkyRL’s steps took about 25 minutes (section 5.8), so it is left out of the breakdown below.
-
prime-rl and verl wait on rollouts for two thirds of the step. TRL waits 51 s and spends half its step in forward and backward.
-
TRL’s 50 s of “other” was not broken down; rank 0 tree-packing 1,280 samples per step on CPU is the likely cause.
-
NeMo-RL’s steps take twice as long: 223 s waiting on rollouts and 186 s training (151 s of training plus 35 s recomputing log-probs). Its training hours sum its step times, because the run was resumed three times (section 5.8).
-
TRL trains each model call as its own sample and tree-packs them, so a shared prefix is forwarded once: 27.4M sample tokens per step become 3.6M forwarded tokens. Appendix H shows how.
5.6 Time to reward
Best training reward so far against wall-clock hours.
- At 0.60: prime-rl 1 h 26 min, TRL 2 h 11 min, verl 3 h 0 min, NeMo-RL 5 h 2 min, SkyRL 19 h 12 min.
- As on math, this is training reward only: the held-out scores (section 5.4) don’t separate the libraries.
5.7 Memory
The chart shows the same two peaks as on math (section 4.6): the bar is the nvidia-smi peak, everything the process holds on the GPU including PyTorch’s allocator cache, and the diamond is the torch peak, the most memory live tensors used at once.
- Trainer GPUs, nvidia-smi peak: prime-rl 67.2 GB (it keeps AdamW state on CPU by default), TRL 127.2 GB, NeMo-RL 131.6 GB, verl 191.4 GB.
- Trainer GPUs, torch peak where logged: TRL 124 GB, verl 88.7 GB. TRL’s tensors use nearly all it holds; most of verl’s nvidia-smi number is allocator cache.
- Inference GPUs: 137.8 GB for NeMo-RL, 167 to 191 GB for the others. vLLM reserves its KV cache up front, so these are set by its memory setting, not by the workload.
5.8 Libraries that did not train or needed changes
- SkyRL: its recommended recipe collapsed in five steps: timeouts from its single one-CPU Harbor actor rose to 106 of 128 episodes, and a timeout masks its whole group, so the gradient went to zero. Its default loss at the shared run shape trained all 60 steps and reached 0.491, at about 25 minutes per step (24.8 hours).
- OpenRLHF: its agent executor rejects Qwen3.5’s re-rendered chats; with the token-level hook, the first backward ran out of memory at the 65k-token context (section 7.3).
- NeMo-RL: trained all 60 steps, but it needed three resumes from checkpoints. One episode that returns no result aborts the whole run, and after about 34 steps new boxes failed because pseudo-terminals ran out, which triggered it. It also needed workarounds to start its Harbor boxes on our nodes (a missing Python module in its container, a glibc mismatch, a
/procmount flag). - miles: completed episodes, then asked for 102 GiB at the first optimizer step, with the unfused attention backend it needs on B200 (section 7.3).
We also ran these tasks on the 35B MoE. Its base model already passes them about as often as any trained run, so they don’t separate the libraries at that size. The results are in Appendix E.
6. Text-to-SQL: BIRD on the 35B MoE
The last workload is the hardest for the libraries: multi-turn tool use on Qwen3.5-35B-A3B, a mixture-of-experts model with 35B parameters and about 3B active per token. TRL, prime-rl and SkyRL trained it on one node; verl, NeMo-RL and miles did not (section 6.8).
6.1 The task
The task is miles’ examples/text_to_sql recipe, on databases from the BIRD benchmark:
- Episode: the model gets a question about a database and its schema. It can run up to 5 read-only
execute_sql_querycalls and read their results, then answers with a final query in a<solution>block. - Reward: 1 if the final query returns the right rows, 0 if not, -1 for no answer or an unsafe query, with small penalties for skipped evidence rules and invalid tool calls.
- Data: 2,033 training questions. We score 398 validation and 498 arcwise questions, greedy, one try each; a third split, hard (658 questions), overlaps train.
- Why it’s harder: the model is a 35B MoE that only fits on one node with LoRA and expert parallelism, and every episode is several tool calls long.
Below are full episodes from the three runs: 30 validation tasks, played by the base model and by each checkpoint scored in section 6.4. Click a cell to read an episode: the model’s reasoning, each SQL call and its result, and its final answer next to the gold one. The viewer also opens on its own page.
6.2 Setup
The model runs on one 8-GPU node. TRL and prime-rl put the trainer on 4 GPUs, each holding a quarter of the experts (expert parallelism, EP) and an FSDP shard of the rest, with vLLM on the other 4. SkyRL follows its own recipe for this model: its Megatron trainer (TP2, EP8) and one vLLM engine (TP8) take turns on all 8 GPUs. SkyRL supports Qwen3.5 MoE only on its Megatron backend.
SkyRL’s colocated step runs in three phases, one after the other. vLLM, set to 70% of each GPU’s memory, generates all 1,280 episodes (374 s on average). Megatron then trains on them (295 s). Last, the new weights go to vLLM over NCCL (13 s). Nothing overlaps, so every sample comes from the current weights.
from transformers.distributed import DistributedConfig
from peft import LoraConfig
from trl import AsyncGRPOConfig
config = AsyncGRPOConfig(
model_init_kwargs={
"dtype": "bfloat16",
"distributed_config": DistributedConfig(fsdp_size=4, ep_size=4),
},
save_only_model=True,
...
)
peft_config = LoraConfig(r=32, lora_alpha=64, target_modules=LORA_TARGETS) # attention, GDN, shared expert
- LoRA r=32 on attention, gated DeltaNet (GDN) and shared-expert projections, bf16 base. The routed experts stay frozen: PEFT can’t put LoRA on fused 3D expert tensors under EP yet.
- TRL merges LoRA into the weights and syncs the full model, since with vLLM in data-parallel mode an adapter would reach only one replica.
- Every library uses the same data, reward code and greedy eval (section 6.1). In TRL the episode is a subclass of its rollout loop (
loop_cls), shown below.
The text-to-SQL rollout loop in TRL (trimmed)
async def run_episode(sample, tokenizer, task):
tokens = opening_prompt_ids(tokenizer, task) # system prompt, schema and question
turns, episode = [], {"turns": [], "final_text": "", "unsafe": False}
for turn in range(1, MAX_TOOL_TURNS + 2):
output_ids, logprobs, text, finish_reason = await sample(tokens, MAX_TOKENS_PER_TURN)
turns.append(TurnRecord(list(tokens), output_ids, logprobs))
tokens = tokens + output_ids
episode["turns"].append(text)
message = {"role": "assistant", "content": visible_text(text)}
if turn > MAX_TOOL_TURNS: # out of tool calls: this turn is the answer
episode["final_text"] = message["content"]
break
call = next((c for c in parse_tool_calls(message) if c.name == "execute_sql_query"), None)
if call is None: # no tool call: this turn is the answer
episode["final_text"] = message["content"]
break
observation = await sql_observation(task, call.arguments) # runs the query read-only
if observation is None: # an unsafe query ends the episode
episode["unsafe"] = True
break
tokens = tokens + observation_ids(tokenizer, observation, last_token_id=tokens[-1])
reward = await reward_func(task, episode) # miles' reward: run the final query, compare rows
return turns, reward, "".join(episode["turns"])
class Text2SQLLoop(AsyncRolloutLoop):
async def setup(self):
await super().setup()
self._tasks = {row["prompt"][-1]["content"]: json.loads(row["task"]) for row in self.dataset}
async def generate_one(self, prompt, tools, group_id=0):
task = self._tasks[prompt[-1]["content"]]
async def sample(prompt_ids, max_tokens):
payload = completion_payload(self._request_model, prompt_ids, max_tokens, self.config.temperature)
return parse_choice(await self._post("/v1/completions", payload, self.config.vllm_server_timeout))
turns, reward, text = await run_episode(sample, self.tokenizer, task)
completion_ids = [i for t in turns for i in t.output_ids]
return Rollout([{"role": "assistant", "content": text}], completion_ids, turns, reward=reward)
def compute_advantages(self, group):
# miles' recipe: reward minus the group mean, no std normalization
return group.rewards - group.rewards.mean()
class Text2SQLTrainer(AsyncGRPOTrainer):
loop_cls = Text2SQLLoop
Everything else is each library’s own recipe, so this compares three recommended setups, one run each.
| TRL | prime-rl | SkyRL | |
|---|---|---|---|
| GPUs | 8 (trainer EP4 x FSDP4, vLLM DP4 + EP) | 8 (same trainer layout, vLLM DP4) | 8, shared by Megatron (TP2 EP8) and one vLLM TP8 engine |
| loss | clipped GRPO (0.2 / 0.28) | its IPO loss | dppo |
| learning rate | 5e-5, cosine | 5e-5, cosine | 3e-5, constant |
| samples per question, temperature | 16, T = 1.0 | 16, T = 1.0 | 5, T = 0.6 |
| batch | token budget (32,768 tokens x 10 micro-batches per rank) | 912 with refill, 960 without | 256 prompts x 5 |
| mode | async, max staleness 8 | async, max staleness 8 | synchronous |
Limits of this setup:
- LoRA can’t target the routed experts under EP until PEFT’s 3D-parameter support is EP-aware.
- Full fine-tuning and a two-node layout weren’t run.
- One run per library and per arm.
- SkyRL ships no single-node LoRA recipe for this model, so we combined two of its recipes: the layout of its 1-node 35B recipe (Megatron TP2 EP8, one vLLM TP8 engine) and the LoRA, learning rate and batch of its text-to-SQL LoRA recipe. We left out three settings those recipes use: token-level importance-sampling correction (TIS, capped at 2.0), vLLM’s
mpexecutor with full-decode CUDA graphs, and optimizer offload to CPU. The last one only changes memory.
6.3 Training reward
Mean training reward per step on text-to-SQL, Qwen3.5-35B-A3B with LoRA. Each library runs its own recipe (section 6.2).
- SkyRL’s run stopped at step 79 of 100: the job hit our 16-hour time limit, and the resume from its step-77 checkpoint hung in training at step 80. Its reward had flattened by then, so we report it as is; the charts show steps 1-75. It replaces a first run that used 4 GPUs because our launcher overrode the recipe’s layout.
6.4 Held-out scores
Greedy pass rate per checkpoint for the three runs.
At step 100, TRL and prime-rl both reach 0.877 on validation, from the base model’s 0.827. On arcwise TRL reaches 0.871 and prime-rl 0.857, from 0.751. SkyRL’s step 20 scores 0.874 on validation after 4.0 hours, level with TRL’s step 60 (0.872 after 4.5 hours). Its step 60 scores 0.892 on validation, 0.855 on arcwise and 0.757 on hard after 12.2 hours; TRL’s step 100 scores 0.704 on hard.
- TRL and prime-rl track each other from step 80 on: identical on validation, and within one standard error on arcwise and hard. One run per library with different recipes: parity, not a ranking.
- prime-rl dipped below the base on validation at step 40 (0.779) and recovered by step 60.
- TRL reached step 100 in 8.3 hours, prime-rl in 11.0.
- One binomial standard error is 0.015 to 0.019 per checkpoint. Hours count from job start, model load included.
- A SkyRL step holds 1,280 episodes, about twice TRL’s, so compare it in hours (section 6.6), not steps.
The episode viewer in section 6.1 shows how these checkpoints solve 30 of the validation tasks.
6.5 Step time
Batch per update. Libraries handle same-reward groups differently, so the batch size moves.
- TRL sizes a step in tokens (about 0.8M sampled tokens per update) and fills it from groups that carry signal. As episodes got longer, episodes per update fell from about 940 to about 500.
- prime-rl’s default “refill” tops up to 912 episodes. With about 73% of rollouts in all-correct groups, steps grew from 147 s to 2,255 s by step 40, and we stopped it at 11.9 h. The no-refill run shown in the charts drops those groups without replacing them, like TRL: its updates fell to about 210 to 300 episodes after step 20, and a step took 380 s on average, against 289 s for TRL.
- SkyRL keeps every episode, same-reward groups included, at a constant 1,280.
- TRL and prime-rl spend most of a step waiting on rollouts: generating several tool calls per episode takes longer than training on them.
- SkyRL is synchronous and colocated: generation and training take turns on the same 8 GPUs, so its “waiting on rollouts” is generation time. Over steps 2-75 a step takes 705 s: 295 s training, 374 s generating and 13 s of weight sync.
6.6 Time to reward
Best training reward so far against wall-clock hours.
- At 0.85: TRL 1.73 h, prime-rl 2.03 h, SkyRL 2.18 h. At 0.90: TRL 3.11 h, prime-rl 4.17 h, SkyRL 6.69 h. SkyRL’s best 5-step mean so far is 0.921, after 12.3 h.
- Training reward only: TRL’s and prime-rl’s held-out scores (section 6.4) end in the same band.
6.7 Memory
- Trainer GPUs: TRL 56.2 GB (torch 42.5 GB), prime-rl 66.1 GB (torch 63.0 GB). With LoRA on a frozen bf16 base and the experts split 4 ways, the trainer has room left.
- Inference GPUs: TRL 170.4 GB, prime-rl 178.4 GB, as vLLM preallocates its KV cache.
- SkyRL runs training and its vLLM engine on the same 8 GPUs: they peak at 172.0 GB, with a torch peak of 57.2 GB (steps 1-75).
6.8 Libraries that did not train
None of these reached a training step on the 35B (details in section 7.3).
- verl: LoRA weight sync crashed on B200 (verl #7978, fixed on main after v0.9.1).
- NeMo-RL: LoRA setup broke under expert parallelism.
- miles: wrote this recipe on 8xH200. On B200 we ran it with the unfused attention backend, and its first train step ran out of memory. Its v0.2.0 image looks like it has the fix for the flash-attention bug (section 7.2), and we didn’t try that path.
- OpenRLHF wasn’t tried at this size.
7. Failures
Every library failed somewhere during setup, TRL included. We asked each agent to log every failure it hit, as a proxy for what a user would run into. They come from our environments, hardware (B200) and cluster, so you may not see them on your setup. The matrix below has all of them; click a cell to read one.
7.1 Failure counts
These are the failures: a library broke, hung, crashed or gave wrong numbers, or its docs, examples, defaults, lock file or image didn’t work as written. We count what is still open in the version we pinned. Our own mistakes, cluster problems and problems any Qwen3.5-on-B200 setup would hit are left out.
Each failure has a severity: ✕ blocker (no valid run until fixed or worked around), ▲ major (cost a run or forced a config change), ● minor.
That’s 42 failures, 14 of them blockers and 4 silent. verl, SkyRL and NeMo-RL hold 10 of the 14 blockers. None of prime-rl’s six stopped a training run.
Open failures per library: TRL 2, OpenRLHF 3, NeMo-RL 5, prime-rl 6, verl 6, miles 8, SkyRL 12. The benchmark found 4 more in TRL that we fixed before the release. Counted like the others, TRL is at 6, level with prime-rl and verl. TRL is also the only library we could fix during the benchmark, and the one we read most closely (Appendix B).
7.2 Silent failures
A crash or a hang is easy to notice. We also looked for silent failures: runs that finish without an error but compute wrong numbers, where nothing tells you the training is off.
Packing leaks state on hybrid models (OpenRLHF, SkyRL). Qwen3.5’s gated DeltaNet layers carry a recurrent state and a short convolution. When a trainer packs several samples into one row, that state has to reset at each sample boundary. If it doesn’t, each sample starts from the end of the previous one.
We compared the log-probs of a sample packed after another one with the same sample unpacked. The chart shows the largest per-token difference. Hollow marks are a control: the same check with the boundary removed on purpose.
OpenRLHF’s packing shifts log-probs by up to 7.94 nats per token, and SkyRL’s default FSDP packing by up to 2.38. prime-rl, verl and TRL stay within bf16 rounding (0.25 to 0.41 nats). TRL resets the state at each boundary in its own tree-attention kernel for gated DeltaNet, adapted from flash-linear-attention (tree_gated_delta_rule.py in trl/kernels/).
Neither library warns (OpenRLHF #1421, SkyRL #1525). Both turn packing off in their own Qwen3.5 examples, and so did we, which costs them throughput (Appendix F).
miles computes wrong log-probs on B200. On the same tokens, miles’ trainer gave -1.87 per token and SGLang gave -0.39. The bug is in TransformerEngine, not in miles: it passes FlashAttention 4 a window argument that FlashAttention 4 treats as local attention. miles’ CI runs on H200 with FlashAttention 3, which doesn’t have the problem. TransformerEngine’s unfused attention gives the right numbers but is 7 to 8x slower, so 200 math steps would take 14 to 16 hours. We left miles out of the math results. The newer v0.2.0 image, which we used only for text-to-SQL, includes TransformerEngine’s fix. We checked the code in the image but didn’t run it. We added our B200 numbers to miles #4735.
SkyRL has one more: it unties tied embeddings on its FSDP backend (small drift here, 5.5e-4 at step 200; #2360). It also masks a whole group when one episode times out (#2745). On its recommended agentic recipe that zeroed most of the gradient (section 5). The timeout rate there depended on the run shape and on SkyRL running every Harbor trial in one process, and we didn’t separate the two.
We’re reporting these and the other library bugs upstream (section 7.4).
7.3 Out of memory and the 35B
Long agent episodes. A terminal-agent episode can run to the shared 65,536-token context. The trainer’s memory for logits and attention grows with the length of each sequence, so one long episode, not the batch size, sets the peak. Two libraries ran out of memory on a 180 GB B200:
- OpenRLHF casts the logits of the whole sequence to fp32. At Qwen3.5’s 248k vocabulary that’s about 1 GB per thousand tokens, and its first backward ran out of memory (#1422).
- miles’ v0.1.3 image has to use the unfused attention backend on B200 (section 7.2), which needs about 170 GB for one 65,536-token episode.
The 35B MoE. TRL, prime-rl and SkyRL trained Qwen3.5-35B-A3B on one node. Three libraries didn’t, as shipped:
- verl’s Megatron route doesn’t install as shipped. Its FSDP2 route with LoRA crashed at the first weight sync on B200: vLLM’s FlashInfer MoE backend keeps the experts in a layout verl’s weight loader can’t fill (verl #7978, fixed on main after v0.9.1).
- NeMo-RL’s built-in LoRA failed at model setup under expert parallelism (Automodel #4842). We also tried its full fine-tuning recipes, and those failed on our side, so they aren’t counted. The 4-node recipe ran out of memory at the first train step on our one node with our 29,696-token sequences, 3x the length it was validated at. The 2-node recipe first failed on our launch settings, and we stopped it to give the GPUs to another run.
- miles ships the text-to-SQL example, tested on 8xH200. We ran it on B200 with the unfused attention backend, and its first train step ran out of memory. Its v0.2.0 image looks like it has the flash-attention fix, which we didn’t try.
We didn’t try OpenRLHF at 35B.
7.4 Upstream issues
These are the library and dependency bugs we’re reporting. Where an issue or fix already exists, we add our data to it instead of opening a new one. We posted the blockers first; the others are still to come.
- OpenRLHF:
- Sample packing leaks gated DeltaNet state across samples on Qwen3.5 (#1421).
- Its OpenAI agent executor asserts “prefix mismatch” on the second turn with Qwen3.5’s chat template (#1423).
- The actor builds fp32 logits over the whole vocabulary for the whole sequence, so 32k-65k-token agent episodes run out of memory (#1422).
- SkyRL:
- FSDP packing leaks gated DeltaNet state (comment on #1525).
- FSDP unties tied embeddings, still in v0.5.0 (comment on #2360).
- One agent timeout masks its whole group (comment on #2745).
- The router answered 503 for about 35 s after a weight sync, which killed a fully async run (comment on #2694, to come).
- verl:
- TransferQueue drops samples after a 30 s acknowledgement timeout, then the step fails with
KeyError: 'loss_mask'(#8768). - v0.9.1 can’t be installed as tagged: its lock file points to wheel URLs that now return 404 (#8769).
- The stable Docker image can’t import Megatron-Bridge without
modelopt(#8770). - LoRA weight sync without merging only syncs the base weights (comment on #7495, to come).
- TransferQueue drops samples after a 30 s acknowledgement timeout, then the step fails with
- miles:
--fully-asyncturns on a radix-cache option that rejects the session server its own agentic example uses (#4967).--save-hfwrites 320 of Qwen3.5’s 632 tensors (#4966).run_qwen3_4b.pypasses the converted checkpoint as--load, so a fresh run dies at startup (to come).- TransformerEngine calls FlashAttention 4 with the wrong window on B200 (comment on #4735).
- prime-rl (its
verifiersdependency): Harbor test staging untars without--no-same-ownerand fails on rootless containers (to come). - NeMo-RL (NeMo Gym and Automodel):
- A Harbor trial that fails before the first turn comes back as reward 0 with no mask, which aborts training (Gym #6522).
- Singularity boxes leak PTYs until new boxes can’t start (comment on #6347, to come).
--fakerootis hard-coded, so boxes die with older-libc task images (Gym #6521).- LoRA on Qwen3.5-35B-A3B with expert parallelism fails at setup in
fully_shard_by_dtype(Automodel #4842).
- Harbor: the HF sandbox upload untars without
--no-same-owner, so every upload fails (#4399). - TermGrade: 15 of 128 tasks pass with an agent that does nothing, because
setup.shships the answer (to be sent to the dataset maintainers).
TRL’s own open failures and the bugs fixed in v1.18.0 are in Appendix D.
8. Conclusion
We spent months building AsyncGRPOTrainer into a trainer that is stable, fast, and ready for hybrid and MoE models and agent workloads. This benchmark was the last check before calling it stable. Here is where it stands in TRL v1.18.0.
- Fast. It’s level with prime-rl per useful token on math (53.5k tokens per second against 54.2k), and in the same range per step on terminal agents (199 s, next to 185 s for prime-rl and 206 s for verl). prime-rl still reaches the same math scores in half the wall-clock, because a TRL step holds twice the tokens (section 4.5).
- Learns correctly. Reward climbs on every workload, and held-out scores land inside the other libraries’ range. On the shared math config it reaches 0.388 on AIME24 from a 0.212 base; the other libraries land between 0.317 and 0.410.
- Stable. It ran every workload in the benchmark. The bugs the benchmark found are fixed in v1.18.0, and the two that remain have a one-line workaround (Appendix D). It keeps the trainer’s log-probs matched to what vLLM sampled: with top-p or top-k it renormalizes them over vLLM’s candidate tokens (#7627), and score centering, its default objective, subtracts the sampler’s expected score at every position (section 2).
- Hybrid and MoE models, agent workloads. It trains hybrid linear-attention models with correct packing, a 35B mixture-of-experts on one node with LoRA and expert parallelism, single-turn math, multi-turn tool use and terminal agents.
- Simple to install.
pip install trland a vLLM server. No custom inference fork, no container image and no extra infrastructure such as a Ray cluster. - Simple to customize. A new environment is a subclass of the rollout loop: the text-to-SQL task in section 6 was ported that way, with no change to the library.
Stable doesn’t mean done. It means the API is one we’ll support, and the trainer is tested well enough to hand to the community. Next on the list:
- Shorter math steps. prime-rl reaches the same reward in half the wall-clock, because a TRL step holds twice the tokens.
- Moving the harness trainer for agent frameworks out of
trl.experimental, with several Harbor servers by default.
Thanks to the authors of prime-rl, verl, SkyRL, OpenRLHF, miles and NeMo-RL for publishing their code and recipes. If you maintain one of them and think we got something wrong, tell us and we’ll correct the post.
To try it, start from the AsyncGRPO quick start, and open an issue on huggingface/trl if something breaks.
Appendix A. Every run
The table lists the 29 reported runs, including failed and stopped ones. Click a row for the run’s stack and config files.
Steps are optimizer steps out of those planned. SkyRL’s math runs have an extra eval at step 201. A step holds a different amount of data in each library, so read the column as progress, not work done.
Appendix B. Caveats
- Small eval sets.
- AIME24 and AIME25 have 30 problems each, scored as avg@16. Differences under about 0.05 are noise.
- The terminal-agent held-out set is 14 tasks at 12 episodes each. Counted over tasks, one standard error is 0.06-0.09 on the 9B runs and up to 0.11 on the 35B. Differences under about 0.1 are noise.
- Text-to-SQL is one greedy try per task on 398 validation, 498 arcwise and 658 hard tasks, with one standard error of 0.015-0.019. The hard split overlaps the train data.
- One seed per run. TRL’s text-to-SQL rerun is a bug fix, not a second seed.
- Which run is charted. The charts show one run per library and workload (on 2B math, one per arm). For TRL on 9B terminal agents that’s the run at
max_staleness=4; its 12-episode held-out numbers (0.386 / 0.417 / 0.443 at steps 20 / 40 / 60) come from a second run at staleness 8. Raising the cap changed almost nothing: the cap-4 run sat at its cap on 17 of 60 steps, the cap-8 run reached 5 once, and mean staleness was about 2.2 in both. The 35B terminal-agent charts work the same way. - Configs can’t be identical.
- TRL’s step is token-sized and filled from the groups it keeps after dropping same-reward ones. On 2B math that’s 2.73M tokens per step against prime-rl’s 1.37M, so by step 200 TRL has trained on about twice the data.
- prime-rl has no built-in PPO clip and runs its default IPO loss.
- SkyRL’s async mode refuses a PPO clip. Our mapping of the shared clip onto its
rollout_ismask collapsed math at both sizes, so its default loss (dppo) stands in. - verl, OpenRLHF and SkyRL keep same-reward groups in the loss. OpenRLHF and SkyRL reshuffle prompts.
- On text-to-SQL each library ran its own recipe. SkyRL’s colocated recipe uses all 8 GPUs for both training and inference.
- The shared setup leans toward TRL. The shared loss (a 0.2 / 0.28 clip against the inference engine’s log-probs, token-level averaging) is TRL’s clipped objective, and prime-rl and SkyRL can’t express it. The terminal-agent evals use TRL’s eval script, and TRL got more runs than anyone on terminal agents and text-to-SQL.
- TRL version. TRL runs used pre-release builds of TRL v1.18.0; Appendix D lists the fixes they included.
- Failure counts. We wrote the shared config, the eval and the ledger rules, and TRL’s code and logs got far more attention than anyone else’s, so a problem in another library is more likely to have gone unnoticed. Counts don’t weigh severity beyond the blocker / major / minor tags, and several ledger rows can share one root cause (section 7.1).
- The recommended arm is incomplete. On 2B math each library also ran its own defaults. miles’ runs and verl’s recommended run aren’t shown because they needed library patches. TRL’s recommended run includes the score-centering weight-sync stall (Appendix D).
- Different inference stacks. vLLM 0.24 (verl), 0.29 (OpenRLHF, NeMo-RL), 0.32 (prime-rl) and 0.33 (TRL, SkyRL), and SGLang for miles.
- Throughput definitions differ. We rebuilt one definition from each library’s logs. Useful-token counts are exact for TRL and prime-rl, and estimates for verl, OpenRLHF and SkyRL, which don’t log the share of same-reward groups. SkyRL pads every sequence to the longest in the step, as its Qwen3.5 example does, which is about a third of forwarded positions on 2B math (section 4).
- Terminal-agent harness placement differs. The 12-episode held-out evals all ran through one eval script on local Apptainer boxes, base models included. Training differed:
- prime-rl runs terminus-2 inside each podman box. It can’t fully turn summarization off (triggered in 4 of 202 smoke episodes).
- TRL and verl run terminus-2 on the host with Apptainer boxes. verl’s agent loop is about 300 lines of our glue. TRL’s launcher swaps OpenEnv’s terminus-2 entry point at startup to pass the call cap; we count it as glue, but it’s close to a patch.
- SkyRL runs Harbor in-process inside one single-CPU Ray actor, with the shared run shape instead of its recipe’s.
- SkyRL’s 35B startup hang. SkyRL’s first 8-GPU text-to-SQL attempt hung at startup: one of its 8 vLLM workers crashed while compiling, and the other seven waited for it with no error for over two hours until our watchdog stopped the job. The second attempt trained. We didn’t confirm the cause; giving each job its own compile cache is the change we made.
- Untested. miles’ packing numerics; the fla 0.5.2 + Triton 3.7.1 gated DeltaNet corruption on B200 that SkyRL’s lock warns about, in the other environments; TRL’s packed-vs-unpacked check on B200; adapter-only weight sync (vLLM
--enable-lora) on the 35B MoE.
Appendix C. Reproduce
The benchmark code is in experiments/asyncgrpo_stable_bench/ on the asyncgrpo-benchmark branch of TRL, and the raw results of every run are in the dataset trl-lib/asyncgrpo-stable-bench-runs. The layout:
envs/<lib>/: per-library install scripts, the pinned source commit and the resolved packages, plus setup notes.configs/<lib>/: configs and equivalence tables against the shared config (configs/SHARED_W1.md,configs/SHARED_W2.md), with every gap written down.launch/<lib>/: the launchers.results/raw/<run_id>/: raw results in one schema for every run, downloaded from the dataset.results/robustness/: the failure ledger, one row per distinct problem with its cause, fix and time lost.results/tables/throughput_fair.md: the common throughput definition.figures/build.sh: rebuilds the charts from raw data only.
Each library’s subagent started from a brief like the one below, prime-rl’s, sent on 2026-11-09. The other libraries got the same structure with their own links and recipes; TRL’s also had it build the shared math data and reward. We replaced our cluster paths with placeholders such as $BENCH and $RUNS, and removed our job-queue tool’s name.
The prime-rl subagent's brief
You own the **prime-rl** arm (https://github.com/PrimeIntellect-ai/prime-rl) of a cross-library RL benchmark for the TRL AsyncGRPOTrainer stable release. Other agents do TRL, SkyRL, miles, verl and OpenRLHF in parallel. The comparison must be fair: we run each library the way its authors recommend.
Read these first, they are the contract: `$BENCH/{README.md,PLAN.md,configs/SHARED_W1.md,collect/SCHEMA.md}` ($BENCH is the benchmark directory) and our cluster rules.
Hard rules (the user is strict about these):
- No CPU work on the login node beyond editing files, git clone, `sbatch`, `squeue`, short `tail -n 20`/`grep -c`. Installs, imports, tests, data conversion, analysis: CPU-partition jobs (<=28 CPUs). Wait for a cpu job with `sbatch --wait ...` in a background shell.
- GPU jobs: only through our job-queue wrapper, run in a background shell (it blocks until the job ends). Never submit to the GPU partition directly, never more than 1 node for your jobs. Smoke tests `--gres=gpu:2..4`; timed runs `--nodes=1 --gres=gpu:8 --exclusive`. Logs under `$RUNS/prime-rl/` (create dirs first; Slurm won't). Total recipe-reproduction budget: about 3 node-hours.
- `uv` only, never pip. Source checkout in `$SRC/prime-rl` (pin latest release tag or main, record SHA in `$BENCH/envs/prime-rl/SOURCE`), venv `$ENVS/bench-prime-rl` (never inside a repo; if the lib insists on its own `.venv` via `uv sync`, use `UV_PROJECT_ENVIRONMENT` to point it there). a shared `HF_HUB_CACHE`.
- Only write in `$BENCH/{envs,configs,launch}/prime-rl/` and `$BENCH/collect/prime-rl.py` (+ `$BENCH/results/raw/<run_id>/` for your runs). Don't edit report.md, PLAN.md, SHARED_W1.md, other libs' dirs. Don't git commit. Never hard-wrap markdown prose.
- Log every problem and fix, in order, in `$BENCH/envs/prime-rl/NOTES.md`. Proposed changes to the shared config go under a "Proposed changes to SHARED_W1" heading there. Don't patch the library unless you must; any patch is recorded in NOTES.md and setup.json.
Tasks, in order:
1. Research (read the repo/docs, don't guess): recommended install, how async/off-policy works and its default (`max_off_policy_steps` / async level), training backend, inference engine and version, its math recipes and any published numbers/curves (e.g. `examples/hendrycks_sanity`, `examples/qwen30b_math`, reverse_text), and multi-turn / agentic support relevant to W2: does verifiers / prime-environments have a Harbor or terminal-bench environment, and could it run tasks through OpenEnv's Harbor server or HF Sandboxes? Write this at the top of NOTES.md.
2. `envs/prime-rl/install.sh` (from-scratch, runs in a CPU job), `SOURCE`, `setup.json` per SCHEMA (time the install).
3. GPU smoke: their smallest example (e.g. reverse_text) runs end to end on a B200.
4. Qwen3.5 check: `Qwen/Qwen3.5-2B` is a hybrid gated-DeltaNet model shipped as a VLM checkpoint (`Qwen3_5ForConditionalGeneration`). Check the library can load it in both trainer and inference, train text-only, and that the fast kernels are bound: `flash-linear-attention` installed WITH deps (fla-core) and `causal-conv1d`; without them transformers silently uses a ~20x slower torch path. Verify by importing `fla.ops.gated_delta_rule.chunk_gated_delta_rule` in the venv and checking the training step time / profile, not just a flag. Also check sequence packing doesn't cross sample boundaries in the linear-attention layers (packed hybrids need `cu_seqlens`/`seq_idx`-aware kernels); write down what the lib does. If it can't run Qwen3.5 correctly, say so plainly and use `google/gemma-4-E4B-it` as the fallback for this lib.
5. Recipe reproduction (scaled down, within budget): run their closest math recipe as published for enough steps to compare against any curve or number they publish. Record what you compared and whether it matched.
6. W1 shared config: wait for `$BENCH/workloads/math/data/train.parquet` and `$BENCH/workloads/math/reward.py` (being built by the TRL agent; if they're not there yet, do the other tasks first). Write `configs/prime-rl/w1-S.*`, `configs/prime-rl/EQUIVALENCE.md` (one row per SHARED_W1 knob: lib knob, value, semantic notes; settle by reading the code what "batch size" counts, loss aggregation, advantage normalization, clipping/IS correction, max length semantics), `launch/prime-rl/w1.sbatch` (1 node exclusive, 4 inference + 4 train GPUs, gpu.csv sampler per SCHEMA, port from job id, HF checkpoints at steps 50/100/150/200 under `$RUNS/prime-rl/<run_id>/`), `collect/prime-rl.py` (native logs -> results/raw/<run_id>/ per SCHEMA; `null` for what the lib doesn't expose).
7. Submit the full W1-S shared run (200 steps) through the queue. Once it is training healthily (reward logged, no errors for a few steps), you may return without waiting for the end.
Final message (concise, plain English): what works, what broke and how you fixed it, measured numbers (install time, first-step time, step time, reward at last logged step), recipe-repro verdict, Qwen3.5 verdict, W2/Harbor feasibility, job ids still running, open problems, proposed SHARED_W1 changes.
Appendix D. TRL issues
Open in v1.18.0. Two failures are open. Both are in TRL’s defaults, and both are major.
Score centering stalls weight sync on long completions. Score centering, TRL’s default objective, asks vLLM for the top 8 log-probs of every sampled token. At 16k-token completions that backlog stalls vLLM’s API servers, and the pause TRL sends before each weight sync waits behind it. On 2B math with TRL’s defaults, weight sync took 19.4 s per step, against 0.22 s with score centering off, out of a 65.6 s step. prime-rl pauses vLLM in 0.048 s on the same topology.
Workarounds: set score_centering=False (our shared runs do), or start vLLM with --api-server-count 16, which brought the pause to 0.06 s in a standalone test.
accuracy_reward can freeze the rollout worker. math_verify enforces its timeouts with a Unix signal, which only works on the main thread. AsyncGRPO runs synchronous rewards in a worker thread, so an answer like \boxed{10^{10^{10}}} can hang it. It’s fixed on main by #8330, which grades in a separate process; it merged after v1.18.0 was tagged, so it ships in the next release. On v1.18.0, writing the reward as an async def avoids it.
Fixed in v1.18.0. The benchmark found four TRL bugs, all on the 35B (section 6):
- a FLOPs estimate that crashed MoE training at step 1 (#8236)
- tree patching that missed models sharded at load time (#8294)
- a merged-LoRA weight sync that crashed on expert-parallel weights (#8295)
- vLLM’s prefix cache surviving weight syncs, a silent one (#8212, fixed in #8300)
Appendix E. Terminal agents at 35B
We also ran section 5’s terminal tasks on Qwen3.5-35B-A3B, with the setup of section 6.2 (LoRA r=32, expert parallelism, one node). TRL and prime-rl both finished 60 steps. The base model already passes most of these tasks.
- Base 0.577, prime-rl 0.583, TRL 0.607 at step 60, inside a task-clustered standard error of 0.08 to 0.11. These tasks are too easy at 35B to show what RL adds.
- The TRL bar is the staleness-8 run, the only one with 12-episode evals. Step times below come from the staleness-4 run, whose 4-episode evals (0.554 / 0.643 / 0.661) are also inside the noise.
- A harder 44-task TermGrade slice gave the same picture: flat train reward for both, no held-out gain over base.
- SkyRL trained with its default loss; we stopped it at step 9, so it has no eval.
- 348 s per step for TRL, 339 s for prime-rl.
- TRL: 141 s forward/backward, 159 s rollout wait. prime-rl: 41 s and 295 s.
- TRL forwards 3.5M tokens per step after tree packing, prime-rl 0.95M (one sequence per episode, Appendix H). We haven’t broken down that gap or TRL’s remaining 44 s.
Appendix F. Math throughput
Steps aren’t the same size. About half the groups are same-reward (TRL logged 46%, prime-rl 51%), and the libraries handle them differently.
- prime-rl drops same-reward groups without replacing them: 126 completions, 1.37M tokens per update.
- TRL sizes a step in tokens and fills that budget from groups with spread: 234 completions, 2.7M tokens, twice the tokens of a prime-rl step.
- verl, OpenRLHF and SkyRL keep all 256 completions (2.48M to 2.91M tokens); same-reward groups carry no gradient.
So we compare useful tokens: tokens from groups with some reward spread (estimated from mean reward for the last three).
| library | seconds per step | useful tokens per step | useful tokens per second | seconds per million useful tokens |
|---|---|---|---|---|
| prime-rl | 25.2 | 1.37M | 54.2k | 18.5 |
| TRL | 51.0 | 2.73M | 53.5k | 18.7 |
| verl | 48.0 | 1.53M (est.) | 31.9k | 31.3 |
| SkyRL (dppo) | 56.2 | 1.39M (est.) | 24.8k | 40.4 |
| OpenRLHF | 71.0 | 1.24M (est.) | 17.4k | 57.3 |
| NeMo-RL | 112.9 | 1.37M (est.) | 12.2k | 82.3 |
- TRL and prime-rl are level: 53.5k vs 54.2k useful tokens per second.
- verl trains the most raw tokens (60.7k per second), but half belong to same-reward groups.
- OpenRLHF runs unpacked because its packing leaks recurrent state on Qwen3.5 (section 7.2).
- SkyRL pads to the longest sequence per its Qwen3.5 example (4.26M positions for 2.87M real tokens), which understates its math throughput.
Appendix G. Math on Qwen3.5-9B
The 9B base is already strong (0.700 on AIME24), and three quarters of groups are same-reward (75% prime-rl, 74% TRL).
| library | step | AIME24 | AIME25 | MATH-500 | hours since first step |
|---|---|---|---|---|---|
| prime-rl | 200 | 0.829 | 0.692 | 0.980 | 1.50 |
| TRL | 50 | 0.827 | 0.690 | 0.978 | 1.39 (2.31 to step 75) |
| SkyRL (dppo) | 200 | 0.802 | 0.677 | 0.990 | 5.74 |
| OpenRLHF | 200 | 0.802 | 0.683 | 0.980 | 6.19 |
| base Qwen3.5-9B | 0.700 | 0.600 | 0.956 |
- TRL has one checkpoint: we stopped it at step 75 to free its node for the agentic runs.
- A TRL step took 114 s, a prime-rl step 27 s. prime-rl’s updates shrank to about 60 completions (0.6M tokens); TRL’s held about 316 (2.7M). Per sampled token they’re level, 22.1k vs 23.7k per second.
- In data trained they line up: TRL’s step 50 at 1.39 h with 135M tokens, prime-rl’s step 200 at 1.50 h with 119M, both at 0.83 on AIME24.
- SkyRL and OpenRLHF take 101 s and 112 s per step (reasons in Appendix F, plus fp32 checkpoint exports for SkyRL). We didn’t schedule verl or miles at this size.
Appendix H. Tree packing on terminal agents
On terminal agents, TRL trains each model call as its own sample and tree-packs them, so a shared prefix is forwarded once.
Tree packing turns 27.4M sample tokens per step into 3.6M forwarded tokens, a 7.7x cut.