Neeraj Bisht

RL post-training for agents (2 of 2) 1. Rollouts and training a. Training and Inference infra can be colocated and same GPUs alternate between generating and training, and weights are resharded between the training layout and the inference layout. b. Colocated GPUs wait on the longest trajectory. Separate pools let both sides run at the same time 2. Managing model artifacts a. For training weights, optimizer state, scheduler state, RNG state are needed. These is pretty large size for e.g. 1 TB for a 70B model with Adam b. Inference weights don't need optimizer states and so much smaller e.g. 140 GB for a 70B model in bf16 c. Syncing inference weights over NCCL and syncing shards rather than entire copy can cut down sync time. d. In verl (RL library) Delta weight sync goes further. RL updates are sparse and over 99% of bf16 weight bytes do not change between steps. verl can send only the changed values and verify them with a checksum, so the result is bit-exact. 3. Annotating trajectories is important a. policy_version that generated it b. Per-token log probabilities from the rollout engine Both are needed by the trainer. 4. Async rollouts and staleness a. Rollout and Training can work synchronously but rollout GPU's can stay idle during training which is bound by longest trajectory. b. Fully async trainers can group updates and once enough samples have accumulated from rollout update weights. c. It is important to manage delta/staleness between trainer and rollout to avoid complete divergence of data from policy. 5. Weight updates without stopping the rollout fleet There are multiple stratgies a. Update replicas in groups by draining them and loading new weights. After that mark replicas that are updated as active. b. Loading weights in the background needs spare GPU memory for a second copy of the weights. Many setups pause a replica briefly instead c. Pause in-flight rollout generation (sleep) and resume it after the sync. d. Save state of a multi-turn tool loop mid-generation and continues it after the sync, so the agent does not restart the task. 6. Tuning a. The efficiency is goal is to keep rollout time close to training time b. If rollout GPU's idle time is high and trainer idle time is low: move GPUs to the trainer. The reverse: move GPUs to rollout. c. Manage the tradeoff - The more often you sync weights the more closer to on-policy you are but at the cost of frequent updates. To summarize the system production traces → tasks → synthetic tasks → rollouts → rewards → training → new weights → weight sync → new rollouts

14

Most AI initiatives don't fail on accuracy. They fail because nobody agreed on what decision the output was supposed to change. That idea came up again and again over the last few weeks, and it's the one I'm carrying forward. I've just completed the Leadership with AI programme at the Indian School of Business (ISB). I went in expecting a course about tools. I came out with a much sharper view of the harder part — how AI changes the way decisions get made, who owns them, and what leaders are accountable for once a model is in the loop. Three things that stuck: 1. The bottleneck is rarely the model. It's the clarity of the business question feeding it. 2. Adoption is a change-management problem long before it's a technical one. 3. Governance isn't a tax on speed — it's what makes speed defensible. Why this mattered to me: I've spent my career in Data and Analytics — turning messy data into decisions senior stakeholders can act on. The skill I want to keep building is the one this programme kept pushing on: deciding which questions are worth answering at all, and making sure someone actually acts on the answer. That's also where I'd like to take my next step. I'm open to conversations around Decision Science, Product Analytics and AI-adjacent analytics roles — the kind of work that sits closer to strategy. If you're hiring, or you know a team building in this space, I'd genuinely welcome the connection. And if you're working at this intersection already, I'd love to trade notes regardless. A special thank you to Milind Naik, who mentored our cohort through weekly live sessions. Those conversations were where the frameworks stopped being theory and started being useful. #ISB #ArtificialIntelligence #DataScience #ProductAnalytics #Leadership #LeadershipwithAI

40

3 commenti