In AI agent training, you can pay for powerful GPUs and still spend much of your time waiting. Agents need to attempt tasks and produce results before the model can learn from them. A few slow attempts can hold up the next training update, making the systems that run those tasks a major factor in training speed and cost.
Di Jin, VP of AI Training at Amsterdam-headquartered Nebius, says teams coming from pretraining or supervised fine-tuning can underestimate the work these systems do.
“If you staff and design it as a training project, inference becomes the bottleneck nobody owns.”
Rollout generation often takes most of a training step’s wall-clock time, making batching, cache management and scheduling central to how fast the model can learn. “If you staff and design it as a training project, inference becomes the bottleneck nobody owns,” Jin says.
Time to a trainable batch
He adds that engineers should measure time to a trainable batch alongside throughput. Average tokens per second can hide the long tail: in a synchronous batch, the step cannot finish until the final trajectory is completed.
Agent trajectories do not all have the same length. Many finish quickly, while others take longer, leaving training GPUs waiting on a handful of stragglers. For this reason, Jin advises engineers to treat the inference engine as part of the post-training reinforcement learning optimizer.
Data misunderstood
Learning from data generated by an older model version involves a tradeoff. Some policy lag can improve GPU efficiency by letting generation and training overlap, Jin says, but its safety depends on how much the policy has changed and how the training algorithm handles the difference.
Jin tells The New Stack, “In fully synchronous RL, inference GPUs wait while training runs, and training GPUs wait while generation runs. Letting generation use weights that are one or a few updates old allows both to run in parallel. That overlap is how most large-scale systems reach reasonable utilization.” When success on a hard task is rare, he adds, an already successful trajectory can be valuable to reuse even if it is slightly stale.
Staleness becomes harmful when the policy that generated a trajectory has drifted too far from the policy being trained. The useful measure is divergence, not a universal limit on how many training steps old the data can be.
“Staleness isn’t the enemy. Unmeasured staleness is.”
Rather than seeking to eradicate old data, engineers and organizations should measure and manage staleness. They can apply importance-sampling corrections, usually truncated to limit variance; drop or down-weight samples beyond a staleness threshold; and track divergence between the generating policy and the current policy. Jin says: “Staleness isn’t the enemy. Unmeasured staleness is.”
Environments are the new data
In Jin’s experience, data and environments take up most of the work in successful RL post-training. Algorithm design is the smallest slice, while infrastructure enables fast iteration. “Infrastructure determines how fast you can iterate. But what the model actually learns is decided by the tasks, environments, and rewards you give it,” he says.
For agents, that means focusing on the environments that generate training experience. Supervised fine-tuning uses a collected dataset; in agentic RL, the model generates trajectories by interacting with an environment.
Reproducible containers, codebases, websites, and APIs with realistic states help the model’s learning transfer to real use. Engineers also need reliable verifiers and tasks at a useful difficulty level. As the model improves, the task set needs refreshing so tasks don’t saturate and stop producing a learning signal. Jin says: “Building environments is the new data curation.”
As with previous technologies, problems occur where systems hand over to one another. The trainer, inference engine, sandbox, and verifier may each work on their own, yet failures emerge at the interfaces. A model can exploit verifier loopholes and receive rewards for behavior that doesn’t accomplish the intended task. Sandboxes can also fail or misbehave, corrupting the training data.
Engineers should record whether an episode ended in model success, model failure, infrastructure failure, or an unknown outcome. Exclude failures attributable to infrastructure from policy-gradient updates. Failures caused by the model’s own actions should still be scored as model failures, so triggering a timeout or another error doesn’t become a way to avoid a penalty.
Long-horizon agentic RL combines trajectories, tool calls, and external latencies, creating bursts of GPU demand and periods of waiting. For GRPO-style training, where sibling attempts at a task are scored against one another, engineers should schedule around group completion rather than launch order. A group enters a ready bank once all sibling attempts finish; the trainer then takes eligible complete groups regardless of when they were launched. Time to a trainable batch is the governing measure, while tokens per second remains useful for diagnosing inference. Routing matters because keeping a conversation close to its cached history can avoid repeated prefill work.
In a Nebius experiment on a Terminal-Bench-based workload, changes to routing, sandbox execution, and batch assembly reduced fixed-weight collection time for a 256-trajectory batch by 66.9%. That measured batch collection, excluding the trainer’s forward, backward, and optimizer work; it did not measure complete training speed.
For agentic AI rollouts, an infrastructure goal is to reduce the time expensive GPUs spend waiting on sandboxes, browsers, code execution, and other non-GPU dependencies.
“Your most expensive GPU is often waiting on your cheapest container.”
Jin recommends scaling CPU and sandbox capacity independently of GPUs and provisioning it generously, because idle containers are comparatively inexpensive. “Your most expensive GPU is often waiting on your cheapest container,” he says.
Keeping sandboxes ready, using snapshot and fork methods, can take environment startup off the critical path. It doesn’t remove the time needed to run tools. Jin recommends putting sandboxes on the same network fabric as the GPUs where possible to minimize the cumulative latency of the hundreds of tool calls an agent may make. Per-step timeouts and budgets can limit the disruption from a hung browser or process. Engineers should track GPU-hours spent waiting on environments, alongside utilization and time to a trainable batch, to identify where additional investment would help.
When the maths does not match
During RL post-training, the rollout engine and the trainer can assign different probabilities to the same tokens, even when they start from the same weights. Quantization, kernels, batching, and, in mixture-of-experts models, expert routing can all contribute. Those differences can introduce systematic bias into training updates.
Jin recommends aligning the numerics first, then measuring and correcting the remaining mismatch. That can mean matching the quantization scheme and computation used in the trainer’s forward pass to the rollout engine, while retaining higher-precision weights for optimization. Importance-sampling corrections can help manage residual differences, but clipping alone should not be assumed to solve every form of mismatch.
Evaluation should use the precision, inference engine, and version that will serve the model. Changing quantization or kernels calls for another evaluation.
Keeping the weights in step
As training updates the policy, rollout workers need a reliable way to receive new weights while keeping policy lag within acceptable limits. Jin describes three broad patterns: resharding weights in memory when training and inference share GPUs; transferring weights directly to separate inference workers; and publishing checkpoints that inference engines hot-load through storage. The best choice depends on the model, hardware topology, and freshness requirements.
A full service restart can introduce avoidable downtime, but storage-based publication does not have to require one. Asynchronous transfer and compact updates can move much of the work off the critical path. Teams should measure end-to-end publication latency separately from the pause needed to apply the weights.
Weight publication also needs an explicit plan for active requests and cached state. Depending on the engine, teams can finish, abort, or pause requests before applying an update, and clear or recompute cached context as needed. If a trajectory spans policy versions, record which version generated its tokens so the training system can handle that mixture deliberately. Coordinate transfers with rollout traffic where they share resources, and monitor publication delays alongside policy age.
Designing these components together helps teams improve iteration speed while keeping training data and policy updates reliable. Jin advises smaller teams to start with a simple setup and add asynchronous complexity when measured bottlenecks justify it.
Before scaling, check:
- Batch readiness: How long does it take to assemble a complete, usable batch, including the straggler tail?
- Policy freshness: Can every rollout be traced to the policy version that generated it, and what drift makes it too stale to use?
- Numerical alignment: How different are the trainer’s and rollout engine’s probabilities, and what correction handles the remaining gap?
- Environment reliability: Can you distinguish infrastructure failures from model failures and reproduce an affected episode?
- Weight publication: What happens to active requests and KV caches during an update, and how long does publication actually take?
- Evaluation: Are you testing the model in the precision and serving stack users will encounter?
The post Nebius cut rollout collection time by 66.9% — here’s what changed appeared first on The New Stack.