AsyncGRPOTrainer: a stable async
RL trainer in TRL 1.18

Part 1 of a series on building a stable, fast RL trainer. We ran it on dense and MoE models from 2B to 35B, in three kinds of environment: single-turn math, terminal agents in an agent harness, and multi-turn tool calling for text-to-SQL.

Affiliation

  1. Hugging Face

Published

 

Resources

1. Introduction

Reinforcement learning is now a standard step in post-training language models. The model attempts a task, a reward scores the attempt, and the update makes high-scoring attempts more likely. When a program computes the reward, such as a math checker, a test suite or a database query, the setup is called RL with verifiable rewards (RLVR). It is how reasoning models like DeepSeek-R1 were trained (paper), and the task, its tools and its reward together are called an environment. GRPO (paper) is the most common algorithm: it samples a group of answers per prompt and pushes up the ones that score above the group’s average.

Generation is usually the slow part. A math completion can run to 16k tokens, and an agent rollout is multi-turn: the model works inside a harness, calls tools and reads their results, over a long horizon. In synchronous GRPO the same GPUs take turns generating and training, so the trainer sits idle for most of the run. Most modern RL implementations now disaggregate the two: inference runs on its own GPUs and the trainer on others, connected by a queue of finished rollouts, with a weight sync after each step. The tradeoff in this design is handling staleness and policy drift, since a sample can come from weights a few steps older than the ones being trained. There is also a trainer-inference mismatch: an inference engine like vLLM samples the tokens and a different library trains on them, and their kernels give slightly different log-probs for the same tokens.

In March we wrote about how async RL is implemented in the wild, across 16 open-source libraries (async RL training landscape). Our own async trainer, AsyncGRPOTrainer, is now stable in TRL v1.18.0 (launch post). This post is the benchmark we ran before calling it stable: we compared it against the main open-source RL libraries: prime-rl, verl, SkyRL, OpenRLHF, miles and NeMo-RL, on our internal B200 cluster.

The benchmark varies two things:

We couldn’t run every library end to end on every model and workload. Some libraries failed to install, ran out of memory, or computed wrong numbers on our hardware (section 7), and we were limited in time and compute, so every run has one seed and the results only compare the runs that trained. All of it, failed attempts and smoke tests included, took about 2,770 B200 GPU-hours over 280 GPU jobs. At AWS’s on-demand price of $14.24 per B200 GPU-hour, that’s about $39,400 in total. Every config, launcher and log is in the benchmark repo (Appendix C). If you can make one of the failed runs work, tell us and we’ll rerun it. Tap a cell to see what happened there.

The benchmark at a glance
Seven libraries by five workloads: whether each library trained on the options it ships, trained after a fix or glue code, had no valid run, or was not tried.

Only TRL, prime-rl and SkyRL have a training run in every column.

The results per workload, with TRL in bold:

Results at a glance
Held-out score (top, base model dashed) and seconds per step (bottom) for every library that trained, per workload. On math a TRL step holds twice the tokens of a prime-rl step, so per useful token the two are level (section 4.4). SkyRL's terminal-agent bar is cut at 450 s.

Section 2 covers the parts of the trainer the results depend on. Section 3 covers the setup. Sections 4, 5 and 6 give the results: math, terminal agents and text-to-SQL. Section 7 lists the failures.

2. AsyncGRPO in one picture

The launch post covers how to install, run and customize AsyncGRPOTrainer. This section covers only the parts of its design that the results below depend on.

In our runs a node’s 8 GPUs are split in two: 4 serve vLLM and 4 train with FSDP2, plus expert parallelism (EP) for the MoE. A rollout worker sends generation requests to vLLM, scores each group of num_generations completions, and queues the samples, each tagged with the weight version that generated it. The trainer pulls from the queue, packs, takes an optimizer step, and syncs its weights back to vLLM. Neither side waits unless the queue is empty or generation is too far ahead.

The AsyncGRPOTrainer loop
The rollout worker sends requests to the vLLM server, scores the rollouts and queues them. The trainer packs them, computes the loss, steps, and syncs its weights back to vLLM. Outlined boxes are the methods you can override.

Staleness. A sample’s staleness is the number of weight updates between the weights that generated it and the ones being trained. max_staleness (4 by default) caps it in two places: generation is held back so it can’t run more than max_staleness steps ahead, and the trainer drops any sample older than that when it pulls it. The loss takes vLLM’s log-probs as the old policy, so one importance ratio covers both staleness and the vLLM-trainer numerical gap.

Token-sized steps. Each training GPU forwards one packed row of up to token_budget tokens per micro-batch, and gradient_accumulation_steps micro-batches make a step. So TRL can’t set the shared config’s 32 prompts x 8 samples exactly (section 3.6), and we set the token budget to approximate it. Steps still differ in size between libraries: the batch charts in Appendix F show samples and tokens per step for every run.

Same-reward groups. If every completion in a group gets the same reward, all its advantages are zero. The rollout worker drops these groups before the queue and fills the step with groups that carry signal, as in DAPO’s dynamic sampling (paper). It logs their share as frac_zero_std_groups. verl, OpenRLHF and SkyRL keep such groups in the loss by default, which is why section 4 compares throughput per useful token.

Tree packing. A packed row is a prefix trie: a token shared by several samples is forwarded once, and on Qwen3.5’s linear-attention layers no state crosses between samples. The docs cover how to use it; a separate post on how tree attention works is coming.

Score centering is the default objective (score_centering=True). At every position it subtracts the sampler’s expected score, estimated over vLLM’s 8 most likely tokens (num_sampler_logprobs, paper), so vLLM returns 8 log-probs per generated token. Our shared-config runs use the clipped objective instead (epsilon=0.2, epsilon_high=0.28). Appendix D gives the step time with score centering on.

Weight sync happens after every step (weight_sync_steps=1): the trainer pauses vLLM and sends the new weights over NCCL. On 2B math with score centering off, that took 0.22 s per step. Requests in flight keep their KV cache through the pause and finish on the new weights, and every request carries the weight version as a cache_salt, so a new version never reuses prefix-cache blocks computed by older weights (#8300).

The terminal-agent runs use HarnessAsyncGRPOTrainer, which stays in trl.experimental and may change (section 5.2).

3. Benchmark setup

We ran seven libraries on the same hardware, the same model family and the same three environments. This section explains how we ran them, then goes through the libraries, the model, the environments, the training configs and how we scored the results.

3.1 How the benchmark was run

We set the benchmark up the way many people now start an RL run: they point a coding agent at a library and an environment, and ask it to get training working. That is a realistic test of a library (Appendix C has one of the briefs we gave the agents). The agent reads the same docs, examples and error messages a user would, and runs into the same problems. We wanted to measure that, while keeping control of the parts that make a comparison fair: the rules on what a run may change, the configs, the environments and the evaluation. So the agents did the hands-on work, and we set the frame and checked every step that mattered:

The problem log has 159 rows: 39 library failures, 38 mistakes of ours, and 82 cluster problems, generic Qwen3.5-on-B200 problems and notes. The subagents logged every problem, however small, so the counts in section 7 are high for every library.

3.2 Rules

3.3 Libraries

libraryversiontrainer / inferencemax staleness
TRL AsyncGRPOTrainerv1.18.0FSDP2 + vLLM 0.33.0max_staleness=4 (8 on text-to-SQL), weight sync every step
prime-rlmain @cbd3702FSDP2 + vLLM 0.32.0max staleness 8, stale samples masked
verlv0.9.1FSDP2 + vLLM 0.24.0, separate_asyncat most 8 versions behind, older dropped
SkyRLmain @d8700468FSDP2 + vLLM 0.33.0, fully async; Megatron + vLLM, colocated, on the 35Bstaleness up to 4 (0 on the 35B)
milesv0.1.3 (v0.2.0 for text-to-SQL)Megatron + SGLang 0.5.23no cap (measured up to 4)
OpenRLHFv0.11.2DeepSpeed ZeRO-2 + vLLM 0.29.0queue of 1, staleness exactly 1
NeMo-RLv0.8.0Megatron (math) or Automodel (35B) + vLLM 0.29.0async on math, at most 1 step behind; colocated on the 35B

Each library makes its own choices: training backend, inference engine, how far generation may run ahead, and which workloads it ships recipes for. Each one runs its own recommended inference stack: four vLLM versions and SGLang.

3.4 Model: Qwen3.5

Every run uses Qwen3.5, a recent open model family that goes from a 2B dense model to a 35B mixture-of-experts. It is a hybrid: three out of four layers are gated DeltaNet layers, a form of linear attention, and the fourth is full attention.

Linear attention keeps a recurrent state, which makes packing harder: a trainer that packs several samples into one row has to reset that state at each sample boundary (section 7.2). TRL’s tree packing does this per trie segment.

The checkpoints are vision-language models. We train the text part and keep vision frozen. We use three sizes:

3.5 Environments

We picked three environments to cover two things: the type of workload (one long completion, a long agent episode, short tool calls) and a difficulty that fits the model size. An environment is useful for RL only when the base model succeeds some of the time and has room to improve.

The three environments
Episode shape, data, models, reward and held-out eval for each environment. The tabs switch between environments.

Why these three:

Details per environment:

3.6 Configs

On math and terminal tasks, every library maps one shared config onto its own options:

settingshared value
data, prompt, rewardthe same for every library
samplingtemperature 1.0, no top-p or top-k
batch per step32 prompts x 8 samples on math, 8 tasks x 8 episodes on terminal tasks
optimizerAdamW, constant 1e-6, gradient clipping 1.0, one step per batch
lossGRPO, group-mean advantage, DAPO’s Clip-Higher (0.2 / 0.28, paper) against the inference engine’s log-probs, token-level, no KL
groups where every sample gets the same rewardkept in the batch. They give no gradient, and no extra groups are generated to replace them
length200 steps on math, 60 on terminal tasks

Each library keeps its own async setting and inference engine. Not every setting can be matched. This is what each library ran on 2B math:

librarylossbatch per stepsame-reward groupspackingmax staleness
TRLPPO clip 0.2 / 0.28token budget, about 2.7M tokensdropped, step filled from groups with signaltree packing4
prime-rlits default IPO mask (no built-in PPO clip)32 x 8kept, excluded from the losspacked8
verlPPO clip 0.2 / 0.2832 x 8keptpacked8
OpenRLHFPPO clip 0.2 / 0.28 with its token-level IS mask32 x 8keptoff (leaks on Qwen3.5, section 7.2)1
SkyRLits default dppo (fully async mode doesn’t accept the PPO clip loss)32 x 8keptoff, padded4

miles has no finished math run (section 7.2).

The shared config pulls every library off its own defaults, and some can’t express all of it: prime-rl has no built-in PPO clip, and SkyRL’s fully async mode doesn’t accept the PPO clip loss. So on 2B math we also ran each library the way its authors recommend, a recommended config:

Compared with its shared run:

The 35B text-to-SQL has no shared config. Each 35B run takes 8 to 12 node-hours, so we could afford one run per library, and we used it to put each library on its best footing: its own recipe, on the same task data, reward and eval (section 6.2).

3.7 Hardware and evaluation

Every run gets one whole 8xB200 node (180 GB per GPU): 4 GPUs for inference and 4 for training, so seconds and memory compare directly. On the 35B, SkyRL’s recipe shares all 8 GPUs between training and inference (section 6.2).

We score every library’s saved checkpoints ourselves, with one eval script per workload:

So every library had to save a complete Hugging Face checkpoint that vLLM can load, and some needed the frozen vision weights copied back in.

3.8 What we measure

Training reward per step shows whether a library learns on the environment at all: a broken loss, a collapse or a reward bug shows up there first. It’s measured on training data, so it can rise without the model getting better on new problems. The held-out score checks that: it measures whether training generalizes to new problems, in domain or out of domain. It’s the number a user cares about, but in some setups our eval sets are small, so differences under a few points are noise (Appendix B).

Step time is how long an optimizer step takes. We also split each step into forward and backward, waiting for rollouts, weight sync and other, which shows whether a library is limited by its trainer or by generation. Steps hold different amounts of data in each library, so we also divide by the tokens a step trains on and report seconds per million tokens.

Time to reward is the wall-clock hours until the training reward first reaches a level. It combines speed and how much each step learns, so it’s a good measure of how fast a run converges. Take it with a grain of salt: it’s training reward, so read it next to the held-out scores.

Peak memory per GPU decides the largest model, context and batch a library can fit on one node, and whether a long agent episode runs out of memory (section 7.3). Finally, we log every failure we hit while getting a run to work, as a proxy for what a user will run into (section 7).

4. Math: DAPO-Math-17k

Every library trains Qwen3.5-2B on DAPO-Math-17k for 200 steps of 32 prompts x 8 samples. One eval script scores every library’s checkpoints at steps 50, 100, 150 and 200.

We ran 2B math twice per library (section 3.6). The shared config puts every library on the same settings, so differences come from the trainer. The recommended config runs each library with its own defaults, to check that the shared settings didn’t hold one back. Each chart has a Recommended config tab first and a Shared config tab second. They open on the shared config, except time to reward (section 4.5), which opens on the recommended one.

Held-out scores in the recommended config:

library, recommended configAIME24AIME25MATH-500
TRL0.4940.3630.912
OpenRLHF0.4600.3730.922
prime-rl0.4520.3290.910
SkyRL (same run as its shared config)0.4100.3290.916

4.2 Training reward

Mean training reward per step on the shared config, Qwen3.5-2B. For every library this is the mean over all scored rollouts, same-reward groups included: for TRL that’s its reward metric, not reward_trained, which covers only the groups it trains on.

Training reward, Qwen3.5-2B
Training reward per optimizer step for each library on the shared config.

Every library learns, at about the same rate: mean reward goes from 0.40-0.47 over the first 10 steps to 0.56-0.62 over the last 10.

4.3 Held-out scores

Held-out scores on the shared config (section 3.6).

Held-out score by checkpoint, Qwen3.5-2B
AIME24 (avg@16) at steps 50 to 200 for each library on the shared config, from the base model's 0.212. The menu switches to AIME25 or MATH-500, the tabs switch to the recommended config, and clicking a library in the legend hides it.
librarylossAIME24AIME25MATH-500hours, first to last step
OpenRLHFshared0.4040.3650.9083.93
TRLshared0.3880.3040.9042.80
prime-rlshared0.3520.3020.9001.41
verlshared0.3500.3000.8922.67
NeMo-RLshared0.3170.2880.8906.25
SkyRLits default (dppo)0.4100.3290.9163.13
base Qwen3.5-2B0.2120.1710.776

4.4 Step time

Step time and cost per token, Qwen3.5-2B
Left: seconds per optimizer step over the run. Right: mean seconds per 1M useful tokens (tokens from groups whose rewards differ), lower is better.
TRL's steps hold twice the tokens of prime-rl's

A TRL math step trains on about 2.7M tokens, a prime-rl step on 1.37M, so seconds per step don’t compare directly. Per useful token the two are level: 53.5k useful tokens per second for TRL, 54.2k for prime-rl. verl, SkyRL, OpenRLHF and NeMo-RL train 12.2k to 31.9k. How each library fills a step, and the full throughput table, are in Appendix F.

Step time breakdown, Qwen3.5-2B
Top: mean seconds per step over steps 6 to 200, split into forward + backward, waiting for rollouts, weight sync and other. Bottom: tokens per step, trained (light) and useful (solid). The tabs switch between the shared and recommended configs.

4.5 Time to reward

Best training reward so far against wall-clock hours.

Time to reach a training reward, Qwen3.5-2B
Each run's best training reward so far (5-step mean) against hours since its first step, on a log time scale. The chart opens on the recommended config; the tabs switch to the shared one. The slider picks a reward level, and the bars below give each run's hours to reach it.

4.6 Memory

Peak memory decides how much model, context and batch a library can fit on a GPU. Memory a trainer doesn’t need is room for a bigger model or longer episodes on the same node (section 3.8).

The charts show two peaks per trainer GPU:

The gap between the two is mostly allocator cache.

Peak GPU memory, Qwen3.5-2B
Peak memory per GPU against the B200's 180 GB, trainer GPUs on the left and inference GPUs on the right. From nvidia-smi, allocator cache included. The diamond is the trainer's own torch peak where it is logged.

We also ran the math workload on Qwen3.5-9B; those results are in Appendix G.

Key takeaway

On math, TRL scores in the same range as the other libraries, and its useful tokens per second are level with prime-rl’s. prime-rl reaches the same 2B scores in half the wall-clock.

5. Terminal agents: TermGrade

This workload is closer to what agentic coding benchmarks measure. The model doesn’t answer in one completion: it drives an agent that works in a terminal inside a sandbox, reads files, runs commands and edits code, for up to 30 model calls and 10 minutes. At the end, the task’s hidden tests decide the reward: 1 if they pass, 0 if not.

We replicated the setup you would use to train a model with RL inside an existing agent harness. The tasks are in the Harbor format, the agent is terminus-2, Harbor’s default agent, and every episode gets its own container. The trainer only sees the model calls and the final reward. This is the hard case for an RL library: episodes are long and uneven, most of the time goes to waiting on sandboxes and tools, and one episode can produce two dozen model calls that share most of their context. We trained Qwen3.5-9B; five libraries made it through.

Key takeaway

A TRL step takes 199 s, between prime-rl’s 185 s and verl’s 206 s. Every library that trained ends above the base model on held-out tasks, too close to rank. TRL ran on its experimental harness trainer with 4 Harbor servers instead of one.

5.1 The task

We used the Harbor task format: each task is an instruction.md, a container image and a pytest verifier the agent never sees. The tasks come from TermGrade, 1,004 terminal tasks in that format. The agent is terminus-2, Harbor’s default.

One agentic episode
A task starts a sandbox, terminus-2 loops model call, keystrokes and terminal output for at most 30 calls and 600 s, and the tests give reward 0 or 1.

5.2 How each library runs an episode

The harness is the multi-turn infrastructure between the agent and the trainer. Each library answers three questions differently: where terminus-2 runs, how its model calls reach vLLM, and what becomes a training sample. TRL and verl use OpenEnv’s openenv harbor serve, a server that runs the agents and puts a token-capture proxy between them and vLLM, so the trainer gets the engine’s exact token ids, log-probs and loss mask.

How each library runs the same episode

where this library differs from the others where the terminus-2 agent runs

Six boxes per library (sandboxes, episode driver, model-call path, vLLM, training sample, trainer), with the boxes where TRL, prime-rl, verl and SkyRL differ highlighted.

5.3 Training reward

Mean training reward per step, Qwen3.5-9B, 8 tasks x 8 episodes per step.

Training reward, Qwen3.5-9B on terminal tasks
Training reward per optimizer step for each library that trained.

5.4 Held-out scores

Every checkpoint was scored with TRL’s eval script: 14 held-out tasks, 12 episodes each, on local boxes behind 4 Harbor servers with the call cap.

Held-out pass rate, 12 episodes per task
Pass rate on 14 unseen tasks at steps 20, 40 and 60 for TRL, prime-rl, verl, SkyRL and NeMo-RL, with one binomial standard error and the base model as a dashed line. Every trained run ends above the base, and the libraries overlap.
checkpointTRLprime-rlverlSkyRLNeMo-RL
base Qwen3.5-9B0.3010.3010.3010.3010.301
step 200.3860.3370.3690.4170.395
step 400.4170.4050.3700.4170.353
step 600.4430.4110.4290.4910.416

5.5 Step time

Step time, Qwen3.5-9B on terminal tasks
Seconds per optimizer step over the run.

TRL, prime-rl and verl land within 11% of each other per step. SkyRL’s steps took about 25 minutes (section 5.8), so it is left out of the breakdown below.

Step time breakdown, Qwen3.5-9B on terminal tasks
Top: mean seconds per step, split into forward + backward, waiting for rollouts, weight sync and other. Bottom: tokens per step; for TRL, before and after tree packing.

5.6 Time to reward

Best training reward so far against wall-clock hours.

Time to reach a training reward, Qwen3.5-9B on terminal tasks
Each run's best training reward so far (5-step mean) against hours since its first step, on a log time scale. The slider picks a reward level, and the bars below give each run's hours to reach it.

5.7 Memory

Peak GPU memory, Qwen3.5-9B on terminal tasks
Peak memory per GPU against the B200's 180 GB, trainer GPUs on the left and inference GPUs on the right. From nvidia-smi, allocator cache included. The diamond is the trainer's own torch peak where it is logged.

The chart shows the same two peaks as on math (section 4.6): the bar is the nvidia-smi peak, everything the process holds on the GPU including PyTorch’s allocator cache, and the diamond is the torch peak, the most memory live tensors used at once.

5.8 Libraries that did not train or needed changes

We also ran these tasks on the 35B MoE. Its base model already passes them about as often as any trained run, so they don’t separate the libraries at that size. The results are in Appendix E.

6. Text-to-SQL: BIRD on the 35B MoE

The last workload is the hardest for the libraries: multi-turn tool use on Qwen3.5-35B-A3B, a mixture-of-experts model with 35B parameters and about 3B active per token. TRL, prime-rl and SkyRL trained it on one node; verl, NeMo-RL and miles did not (section 6.8).

6.1 The task

The task is miles’ examples/text_to_sql recipe, on databases from the BIRD benchmark:

Below are full episodes from the three runs: 30 validation tasks, played by the base model and by each checkpoint scored in section 6.4. Click a cell to read an episode: the model’s reasoning, each SQL call and its result, and its final answer next to the gold one. The viewer also opens on its own page.

6.2 Setup

The model runs on one 8-GPU node. TRL and prime-rl put the trainer on 4 GPUs, each holding a quarter of the experts (expert parallelism, EP) and an FSDP shard of the rest, with vLLM on the other 4. SkyRL follows its own recipe for this model: its Megatron trainer (TP2, EP8) and one vLLM engine (TP8) take turns on all 8 GPUs. SkyRL supports Qwen3.5 MoE only on its Megatron backend.

SkyRL’s colocated step runs in three phases, one after the other. vLLM, set to 70% of each GPU’s memory, generates all 1,280 episodes (374 s on average). Megatron then trains on them (295 s). Last, the new weights go to vLLM over NCCL (13 s). Nothing overlaps, so every sample comes from the current weights.

from transformers.distributed import DistributedConfig
from peft import LoraConfig
from trl import AsyncGRPOConfig

config = AsyncGRPOConfig(
    model_init_kwargs={
        "dtype": "bfloat16",
        "distributed_config": DistributedConfig(fsdp_size=4, ep_size=4),
    },
    save_only_model=True,
    ...
)
peft_config = LoraConfig(r=32, lora_alpha=64, target_modules=LORA_TARGETS)  # attention, GDN, shared expert
The text-to-SQL rollout loop in TRL (trimmed)
async def run_episode(sample, tokenizer, task):
    tokens = opening_prompt_ids(tokenizer, task)  # system prompt, schema and question
    turns, episode = [], {"turns": [], "final_text": "", "unsafe": False}
    for turn in range(1, MAX_TOOL_TURNS + 2):
        output_ids, logprobs, text, finish_reason = await sample(tokens, MAX_TOKENS_PER_TURN)
        turns.append(TurnRecord(list(tokens), output_ids, logprobs))
        tokens = tokens + output_ids
        episode["turns"].append(text)
        message = {"role": "assistant", "content": visible_text(text)}
        if turn > MAX_TOOL_TURNS:  # out of tool calls: this turn is the answer
            episode["final_text"] = message["content"]
            break
        call = next((c for c in parse_tool_calls(message) if c.name == "execute_sql_query"), None)
        if call is None:  # no tool call: this turn is the answer
            episode["final_text"] = message["content"]
            break
        observation = await sql_observation(task, call.arguments)  # runs the query read-only
        if observation is None:  # an unsafe query ends the episode
            episode["unsafe"] = True
            break
        tokens = tokens + observation_ids(tokenizer, observation, last_token_id=tokens[-1])
    reward = await reward_func(task, episode)  # miles' reward: run the final query, compare rows
    return turns, reward, "".join(episode["turns"])


class Text2SQLLoop(AsyncRolloutLoop):
    async def setup(self):
        await super().setup()
        self._tasks = {row["prompt"][-1]["content"]: json.loads(row["task"]) for row in self.dataset}

    async def generate_one(self, prompt, tools, group_id=0):
        task = self._tasks[prompt[-1]["content"]]

        async def sample(prompt_ids, max_tokens):
            payload = completion_payload(self._request_model, prompt_ids, max_tokens, self.config.temperature)
            return parse_choice(await self._post("/v1/completions", payload, self.config.vllm_server_timeout))

        turns, reward, text = await run_episode(sample, self.tokenizer, task)
        completion_ids = [i for t in turns for i in t.output_ids]
        return Rollout([{"role": "assistant", "content": text}], completion_ids, turns, reward=reward)

    def compute_advantages(self, group):
        # miles' recipe: reward minus the group mean, no std normalization
        return group.rewards - group.rewards.mean()


class Text2SQLTrainer(AsyncGRPOTrainer):
    loop_cls = Text2SQLLoop

Everything else is each library’s own recipe, so this compares three recommended setups, one run each.

TRLprime-rlSkyRL
GPUs8 (trainer EP4 x FSDP4, vLLM DP4 + EP)8 (same trainer layout, vLLM DP4)8, shared by Megatron (TP2 EP8) and one vLLM TP8 engine
lossclipped GRPO (0.2 / 0.28)its IPO lossdppo
learning rate5e-5, cosine5e-5, cosine3e-5, constant
samples per question, temperature16, T = 1.016, T = 1.05, T = 0.6
batchtoken budget (32,768 tokens x 10 micro-batches per rank)912 with refill, 960 without256 prompts x 5
modeasync, max staleness 8async, max staleness 8synchronous

Limits of this setup:

6.3 Training reward

Mean training reward per step on text-to-SQL, Qwen3.5-35B-A3B with LoRA. Each library runs its own recipe (section 6.2).

Training reward, text-to-SQL at 35B
Training reward per optimizer step for TRL, prime-rl and SkyRL.

6.4 Held-out scores

Greedy pass rate per checkpoint for the three runs.

Text-to-SQL pass rate by checkpoint, Qwen3.5-35B-A3B
Greedy, one try per task, bars are one binomial standard error, dashed line is the base model. TRL and prime-rl end in the same band. SkyRL has its step-20 and step-60 checkpoints: its step-40 eval failed at vLLM startup in our eval job, and its run stopped before step 80.
Key result

At step 100, TRL and prime-rl both reach 0.877 on validation, from the base model’s 0.827. On arcwise TRL reaches 0.871 and prime-rl 0.857, from 0.751. SkyRL’s step 20 scores 0.874 on validation after 4.0 hours, level with TRL’s step 60 (0.872 after 4.5 hours). Its step 60 scores 0.892 on validation, 0.855 on arcwise and 0.757 on hard after 12.2 hours; TRL’s step 100 scores 0.704 on hard.

The episode viewer in section 6.1 shows how these checkpoints solve 30 of the validation tasks.

6.5 Step time

Step time, text-to-SQL at 35B
Seconds per optimizer step over the run.

Batch per update. Libraries handle same-reward groups differently, so the batch size moves.

Batch per optimizer update on text-to-SQL
Episodes per update, one dot per update and a rolling mean of 5. TRL and prime-rl without refill shrink as episodes get longer or more groups score the same; SkyRL keeps every episode.
Step time breakdown, text-to-SQL at 35B
Mean seconds per step over steps 2 and later, split into training, waiting for rollouts, weight sync and other.

6.6 Time to reward

Best training reward so far against wall-clock hours.

Time to reach a training reward, text-to-SQL at 35B
Each run's best training reward so far (5-step mean) against hours since its first step, on a log time scale. The slider picks a reward level, and the bars below give each run's hours to reach it.

6.7 Memory

Peak GPU memory, text-to-SQL at 35B
Peak memory per GPU against the B200's 180 GB, trainer GPUs on the left and inference GPUs on the right. From nvidia-smi, allocator cache included. The diamond is the trainer's own torch peak. SkyRL shares its 8 GPUs between training and vLLM.

6.8 Libraries that did not train

None of these reached a training step on the 35B (details in section 7.3).

7. Failures

Every library failed somewhere during setup, TRL included. We asked each agent to log every failure it hit, as a proxy for what a user would run into. They come from our environments, hardware (B200) and cluster, so you may not see them on your setup. The matrix below has all of them; click a cell to read one.

7.1 Failure counts

These are the failures: a library broke, hung, crashed or gave wrong numbers, or its docs, examples, defaults, lock file or image didn’t work as written. We count what is still open in the version we pinned. Our own mistakes, cluster problems and problems any Qwen3.5-on-B200 setup would hit are left out.

Each failure has a severity: ✕ blocker (no valid run until fixed or worked around), ▲ major (cost a run or forced a config change), ● minor.

Failures by library and type
Rows are libraries, columns are failure types. A cell counts failures and takes the color of its worst one. Click a cell for the details.

That’s 42 failures, 14 of them blockers and 4 silent. verl, SkyRL and NeMo-RL hold 10 of the 14 blockers. None of prime-rl’s six stopped a training run.

Open failures per library: TRL 2, OpenRLHF 3, NeMo-RL 5, prime-rl 6, verl 6, miles 8, SkyRL 12. The benchmark found 4 more in TRL that we fixed before the release. Counted like the others, TRL is at 6, level with prime-rl and verl. TRL is also the only library we could fix during the benchmark, and the one we read most closely (Appendix B).

7.2 Silent failures

A crash or a hang is easy to notice. We also looked for silent failures: runs that finish without an error but compute wrong numbers, where nothing tells you the training is off.

Packing leaks state on hybrid models (OpenRLHF, SkyRL). Qwen3.5’s gated DeltaNet layers carry a recurrent state and a short convolution. When a trainer packs several samples into one row, that state has to reset at each sample boundary. If it doesn’t, each sample starts from the end of the previous one.

Gated DeltaNet state across packed samples
With no reset at the boundary, the second sample starts from the first one's recurrent and conv state.

We compared the log-probs of a sample packed after another one with the same sample unpacked. The chart shows the largest per-token difference. Hollow marks are a control: the same check with the boundary removed on purpose.

Packed vs unpacked, per library
Largest per-token log-prob difference between a packed sample and the same sample unpacked. OpenRLHF and SkyRL leak. prime-rl, TRL and verl stay at bf16 rounding.

OpenRLHF’s packing shifts log-probs by up to 7.94 nats per token, and SkyRL’s default FSDP packing by up to 2.38. prime-rl, verl and TRL stay within bf16 rounding (0.25 to 0.41 nats). TRL resets the state at each boundary in its own tree-attention kernel for gated DeltaNet, adapted from flash-linear-attention (tree_gated_delta_rule.py in trl/kernels/).

Neither library warns (OpenRLHF #1421, SkyRL #1525). Both turn packing off in their own Qwen3.5 examples, and so did we, which costs them throughput (Appendix F).

miles computes wrong log-probs on B200. On the same tokens, miles’ trainer gave -1.87 per token and SGLang gave -0.39. The bug is in TransformerEngine, not in miles: it passes FlashAttention 4 a window argument that FlashAttention 4 treats as local attention. miles’ CI runs on H200 with FlashAttention 3, which doesn’t have the problem. TransformerEngine’s unfused attention gives the right numbers but is 7 to 8x slower, so 200 math steps would take 14 to 16 hours. We left miles out of the math results. The newer v0.2.0 image, which we used only for text-to-SQL, includes TransformerEngine’s fix. We checked the code in the image but didn’t run it. We added our B200 numbers to miles #4735.

SkyRL has one more: it unties tied embeddings on its FSDP backend (small drift here, 5.5e-4 at step 200; #2360). It also masks a whole group when one episode times out (#2745). On its recommended agentic recipe that zeroed most of the gradient (section 5). The timeout rate there depended on the run shape and on SkyRL running every Harbor trial in one process, and we didn’t separate the two.

We’re reporting these and the other library bugs upstream (section 7.4).

7.3 Out of memory and the 35B

Long agent episodes. A terminal-agent episode can run to the shared 65,536-token context. The trainer’s memory for logits and attention grows with the length of each sequence, so one long episode, not the batch size, sets the peak. Two libraries ran out of memory on a 180 GB B200:

The 35B MoE. TRL, prime-rl and SkyRL trained Qwen3.5-35B-A3B on one node. Three libraries didn’t, as shipped:

We didn’t try OpenRLHF at 35B.

7.4 Upstream issues

These are the library and dependency bugs we’re reporting. Where an issue or fix already exists, we add our data to it instead of opening a new one. We posted the blockers first; the others are still to come.

TRL’s own open failures and the bugs fixed in v1.18.0 are in Appendix D.

8. Conclusion

We spent months building AsyncGRPOTrainer into a trainer that is stable, fast, and ready for hybrid and MoE models and agent workloads. This benchmark was the last check before calling it stable. Here is where it stands in TRL v1.18.0.

Stable doesn’t mean done. It means the API is one we’ll support, and the trainer is tested well enough to hand to the community. Next on the list:

Thanks to the authors of prime-rl, verl, SkyRL, OpenRLHF, miles and NeMo-RL for publishing their code and recipes. If you maintain one of them and think we got something wrong, tell us and we’ll correct the post.

To try it, start from the AsyncGRPO quick start, and open an issue on huggingface/trl if something breaks.

Appendix A. Every run

The table lists the 29 reported runs, including failed and stopped ones. Click a row for the run’s stack and config files.

Every reported run
One row per reported run, with job id, steps reached out of planned, eval count and status; click a row for its stack and config files.

Steps are optimizer steps out of those planned. SkyRL’s math runs have an extra eval at step 201. A step holds a different amount of data in each library, so read the column as progress, not work done.

Appendix B. Caveats

Appendix C. Reproduce

The benchmark code is in experiments/asyncgrpo_stable_bench/ on the asyncgrpo-benchmark branch of TRL, and the raw results of every run are in the dataset trl-lib/asyncgrpo-stable-bench-runs. The layout:

Each library’s subagent started from a brief like the one below, prime-rl’s, sent on 2026-11-09. The other libraries got the same structure with their own links and recipes; TRL’s also had it build the shared math data and reward. We replaced our cluster paths with placeholders such as $BENCH and $RUNS, and removed our job-queue tool’s name.

The prime-rl subagent's brief
You own the **prime-rl** arm (https://github.com/PrimeIntellect-ai/prime-rl) of a cross-library RL benchmark for the TRL AsyncGRPOTrainer stable release. Other agents do TRL, SkyRL, miles, verl and OpenRLHF in parallel. The comparison must be fair: we run each library the way its authors recommend.

Read these first, they are the contract: `$BENCH/{README.md,PLAN.md,configs/SHARED_W1.md,collect/SCHEMA.md}` ($BENCH is the benchmark directory) and our cluster rules.

Hard rules (the user is strict about these):
- No CPU work on the login node beyond editing files, git clone, `sbatch`, `squeue`, short `tail -n 20`/`grep -c`. Installs, imports, tests, data conversion, analysis: CPU-partition jobs (<=28 CPUs). Wait for a cpu job with `sbatch --wait ...` in a background shell.
- GPU jobs: only through our job-queue wrapper, run in a background shell (it blocks until the job ends). Never submit to the GPU partition directly, never more than 1 node for your jobs. Smoke tests `--gres=gpu:2..4`; timed runs `--nodes=1 --gres=gpu:8 --exclusive`. Logs under `$RUNS/prime-rl/` (create dirs first; Slurm won't). Total recipe-reproduction budget: about 3 node-hours.
- `uv` only, never pip. Source checkout in `$SRC/prime-rl` (pin latest release tag or main, record SHA in `$BENCH/envs/prime-rl/SOURCE`), venv `$ENVS/bench-prime-rl` (never inside a repo; if the lib insists on its own `.venv` via `uv sync`, use `UV_PROJECT_ENVIRONMENT` to point it there). a shared `HF_HUB_CACHE`.
- Only write in `$BENCH/{envs,configs,launch}/prime-rl/` and `$BENCH/collect/prime-rl.py` (+ `$BENCH/results/raw/<run_id>/` for your runs). Don't edit report.md, PLAN.md, SHARED_W1.md, other libs' dirs. Don't git commit. Never hard-wrap markdown prose.
- Log every problem and fix, in order, in `$BENCH/envs/prime-rl/NOTES.md`. Proposed changes to the shared config go under a "Proposed changes to SHARED_W1" heading there. Don't patch the library unless you must; any patch is recorded in NOTES.md and setup.json.

Tasks, in order:
1. Research (read the repo/docs, don't guess): recommended install, how async/off-policy works and its default (`max_off_policy_steps` / async level), training backend, inference engine and version, its math recipes and any published numbers/curves (e.g. `examples/hendrycks_sanity`, `examples/qwen30b_math`, reverse_text), and multi-turn / agentic support relevant to W2: does verifiers / prime-environments have a Harbor or terminal-bench environment, and could it run tasks through OpenEnv's Harbor server or HF Sandboxes? Write this at the top of NOTES.md.
2. `envs/prime-rl/install.sh` (from-scratch, runs in a CPU job), `SOURCE`, `setup.json` per SCHEMA (time the install).
3. GPU smoke: their smallest example (e.g. reverse_text) runs end to end on a B200.
4. Qwen3.5 check: `Qwen/Qwen3.5-2B` is a hybrid gated-DeltaNet model shipped as a VLM checkpoint (`Qwen3_5ForConditionalGeneration`). Check the library can load it in both trainer and inference, train text-only, and that the fast kernels are bound: `flash-linear-attention` installed WITH deps (fla-core) and `causal-conv1d`; without them transformers silently uses a ~20x slower torch path. Verify by importing `fla.ops.gated_delta_rule.chunk_gated_delta_rule` in the venv and checking the training step time / profile, not just a flag. Also check sequence packing doesn't cross sample boundaries in the linear-attention layers (packed hybrids need `cu_seqlens`/`seq_idx`-aware kernels); write down what the lib does. If it can't run Qwen3.5 correctly, say so plainly and use `google/gemma-4-E4B-it` as the fallback for this lib.
5. Recipe reproduction (scaled down, within budget): run their closest math recipe as published for enough steps to compare against any curve or number they publish. Record what you compared and whether it matched.
6. W1 shared config: wait for `$BENCH/workloads/math/data/train.parquet` and `$BENCH/workloads/math/reward.py` (being built by the TRL agent; if they're not there yet, do the other tasks first). Write `configs/prime-rl/w1-S.*`, `configs/prime-rl/EQUIVALENCE.md` (one row per SHARED_W1 knob: lib knob, value, semantic notes; settle by reading the code what "batch size" counts, loss aggregation, advantage normalization, clipping/IS correction, max length semantics), `launch/prime-rl/w1.sbatch` (1 node exclusive, 4 inference + 4 train GPUs, gpu.csv sampler per SCHEMA, port from job id, HF checkpoints at steps 50/100/150/200 under `$RUNS/prime-rl/<run_id>/`), `collect/prime-rl.py` (native logs -> results/raw/<run_id>/ per SCHEMA; `null` for what the lib doesn't expose).
7. Submit the full W1-S shared run (200 steps) through the queue. Once it is training healthily (reward logged, no errors for a few steps), you may return without waiting for the end.

Final message (concise, plain English): what works, what broke and how you fixed it, measured numbers (install time, first-step time, step time, reward at last logged step), recipe-repro verdict, Qwen3.5 verdict, W2/Harbor feasibility, job ids still running, open problems, proposed SHARED_W1 changes.

Appendix D. TRL issues

Open in v1.18.0. Two failures are open. Both are in TRL’s defaults, and both are major.

Score centering stalls weight sync on long completions. Score centering, TRL’s default objective, asks vLLM for the top 8 log-probs of every sampled token. At 16k-token completions that backlog stalls vLLM’s API servers, and the pause TRL sends before each weight sync waits behind it. On 2B math with TRL’s defaults, weight sync took 19.4 s per step, against 0.22 s with score centering off, out of a 65.6 s step. prime-rl pauses vLLM in 0.048 s on the same topology.

Workarounds: set score_centering=False (our shared runs do), or start vLLM with --api-server-count 16, which brought the pause to 0.06 s in a standalone test.

accuracy_reward can freeze the rollout worker. math_verify enforces its timeouts with a Unix signal, which only works on the main thread. AsyncGRPO runs synchronous rewards in a worker thread, so an answer like \boxed{10^{10^{10}}} can hang it. It’s fixed on main by #8330, which grades in a separate process; it merged after v1.18.0 was tagged, so it ships in the next release. On v1.18.0, writing the reward as an async def avoids it.

Fixed in v1.18.0. The benchmark found four TRL bugs, all on the 35B (section 6):

Appendix E. Terminal agents at 35B

We also ran section 5’s terminal tasks on Qwen3.5-35B-A3B, with the setup of section 6.2 (LoRA r=32, expert parallelism, one node). TRL and prime-rl both finished 60 steps. The base model already passes most of these tasks.

Held-out pass rate, terminal agents, Qwen3.5-35B-A3B
14 unseen tasks x 12 episodes per checkpoint, bars are one binomial standard error, dashed line is the base model. Neither run moves beyond noise.
Step time breakdown, terminal agents at 35B
Top: mean seconds per step, split into forward + backward, waiting for rollouts, weight sync and other, Bottom: tokens per step; for TRL, before and after tree packing.

Appendix F. Math throughput

Steps aren’t the same size. About half the groups are same-reward (TRL logged 46%, prime-rl 51%), and the libraries handle them differently.

What goes into one optimizer step
Three ways to fill a step from groups of 8 samples: prime-rl takes the next 32 groups and drops the same-reward ones, TRL keeps pulling groups with spread until its token budget is full, and verl, OpenRLHF and SkyRL keep all 32 groups.
Batch per optimizer update, Qwen3.5-2B
Samples in each optimizer update over the run, with a switch to tokens, and a table of means and of how each library treats same-reward groups.

So we compare useful tokens: tokens from groups with some reward spread (estimated from mean reward for the last three).

libraryseconds per stepuseful tokens per stepuseful tokens per secondseconds per million useful tokens
prime-rl25.21.37M54.2k18.5
TRL51.02.73M53.5k18.7
verl48.01.53M (est.)31.9k31.3
SkyRL (dppo)56.21.39M (est.)24.8k40.4
OpenRLHF71.01.24M (est.)17.4k57.3
NeMo-RL112.91.37M (est.)12.2k82.3

Appendix G. Math on Qwen3.5-9B

The 9B base is already strong (0.700 on AIME24), and three quarters of groups are same-reward (75% prime-rl, 74% TRL).

librarystepAIME24AIME25MATH-500hours since first step
prime-rl2000.8290.6920.9801.50
TRL500.8270.6900.9781.39 (2.31 to step 75)
SkyRL (dppo)2000.8020.6770.9905.74
OpenRLHF2000.8020.6830.9806.19
base Qwen3.5-9B0.7000.6000.956
Batch per optimizer update, Qwen3.5-9B
Samples in each optimizer update on Qwen3.5-9B math, with a switch to tokens. TRL ran 75 steps.

Appendix H. Tree packing on terminal agents

On terminal agents, TRL trains each model call as its own sample and tree-packs them, so a shared prefix is forwarded once.

Per-call samples, tree packing and one sequence per episode
context: system prompt, instruction, terminal output tokens the model sampled (trained) the same reply, re-rendered by the chat template as history
What the harness hands over: one sample per model call27.4M tokens per step

terminus-2 sends the whole chat again on every call, and Qwen3.5's template writes a past reply differently from the tokens that were sampled. So call 2 does not start with call 1's tokens, and every call is its own sample that repeats the history.

TRL: tree packing forwards each shared prefix once3.6M forwarded · 7.7× less

The repeated history becomes one trunk. Each call's sampled tokens hang off it as a branch, because the trunk continues with the re-rendered version of that reply. Every call forks the tree, so the sampled tokens are forwarded on top of a full episode.

prime-rl and verl: one token sequence per episode1.8M and 2.0M forwarded · 16.2× less

They bypass the re-rendering: each call appends only the new terminal output to the exact tokens already sampled (token in, token out). The episode is one sequence with nothing to pack.

Drawn for an episode of 4 model calls. Real episodes average about 24 calls and 30k tokens of final context, which is why the first form is so much larger. Token counts are per training step on Qwen3.5-9B.

Per-call samples repeat the history (27.4M tokens per step); tree packing forwards each shared prefix once but forks at every re-rendered turn (3.6M); appending sampled tokens keeps one sequence per episode (1.8M and 2.0M).

Tree packing turns 27.4M sample tokens per step into 3.6M forwarded tokens, a 7.7x cut.