Lakhvinder Gaba

RL post-training for agents (2 of 2) 1. Rollouts and training a. Training and Inference infra can be colocated and same GPUs alternate between generating and training, and weights are resharded between the training layout and the inference layout. b. Colocated GPUs wait on the longest trajectory. Separate pools let both sides run at the same time 2. Managing model artifacts a. For training weights, optimizer state, scheduler state, RNG state are needed. These is pretty large size for e.g. 1 TB for a 70B model with Adam b. Inference weights don't need optimizer states and so much smaller e.g. 140 GB for a 70B model in bf16 c. Syncing inference weights over NCCL and syncing shards rather than entire copy can cut down sync time. d. In verl (RL library) Delta weight sync goes further. RL updates are sparse and over 99% of bf16 weight bytes do not change between steps. verl can send only the changed values and verify them with a checksum, so the result is bit-exact. 3. Annotating trajectories is important a. policy_version that generated it b. Per-token log probabilities from the rollout engine Both are needed by the trainer. 4. Async rollouts and staleness a. Rollout and Training can work synchronously but rollout GPU's can stay idle during training which is bound by longest trajectory. b. Fully async trainers can group updates and once enough samples have accumulated from rollout update weights. c. It is important to manage delta/staleness between trainer and rollout to avoid complete divergence of data from policy. 5. Weight updates without stopping the rollout fleet There are multiple stratgies a. Update replicas in groups by draining them and loading new weights. After that mark replicas that are updated as active. b. Loading weights in the background needs spare GPU memory for a second copy of the weights. Many setups pause a replica briefly instead c. Pause in-flight rollout generation (sleep) and resume it after the sync. d. Save state of a multi-turn tool loop mid-generation and continues it after the sync, so the agent does not restart the task. 6. Tuning a. The efficiency is goal is to keep rollout time close to training time b. If rollout GPU's idle time is high and trainer idle time is low: move GPUs to the trainer. The reverse: move GPUs to rollout. c. Manage the tradeoff - The more often you sync weights the more closer to on-policy you are but at the cost of frequent updates. To summarize the system production traces → tasks → synthetic tasks → rollouts → rewards → training → new weights → weight sync → new rollouts

14

Almost anything can be built now. 🚀 That’s no longer the hard part. AI has collapsed the cost of execution. Prototypes in hours. Features in days. Shipping feels easy — sometimes deceptively so. What hasn’t changed: • Customers still ignore features 👀 • Attention is still scarce ⏳ • Adoption is still hard 📉 At scale, the risk isn’t slow delivery — it’s mistaking output for impact. I see teams add AI features because they’re easy to build, not because they change a meaningful user decision. Roadmaps expand. Velocity looks good. Outcomes don’t. When buildability explodes, judgment becomes the bottleneck ⚖️ This is where the TPM leverage lives: • Shaping bets, not just plans 🎯 • Forcing clarity on outcomes before teams commit • Applying restraint when marginal value is low ✋ Execution still matters. But in a world where anything can be built, deciding what not to build is the leadership skill. The hardest question isn’t “Can we ship this?” It’s “What happens if this ships?” 🤔

7

While scaling distributed systems at Amazon, I found that the most damaging production failures rarely came from bad technology. They came from tradeoffs the team had never named. What separates experienced architects from everyone else is the willingness to name tradeoffs out loud, and own them. Patterns and frameworks help. They don't decide for you. Take stateless vs. stateful design. Stateless services scale cleanly. You can spin up instances fast and distribute load evenly. The catch is that the state still has to live somewhere. Now every request reaches out to a cache or database to rebuild the context it needs, which adds latency, one more dependency, and a fresh set of consistency questions. Stateful services skip that round trip and keep context close, but they are far harder to scale and to recover when a node dies. You are not picking the easy path. You are picking which problem you would rather debug. Or microservices vs. a monolith. Microservices give teams autonomy and let you scale individual pieces. The cost is real: more moving parts, harder observability, and cascading failures that are genuinely difficult to trace. You don't get agility for free. You trade one set of problems for another. 🔁 I have seen the failure mode up close. Early in scaling one of our platforms, the team chose an event-driven architecture to handle volume. Smart call. But we never wrote down what we were giving up: tighter operational complexity, harder failure tracing, service dependencies that weren't obvious at first. When things broke under load, the debugging took far longer than it should have. The tradeoff had been made. It just hadn't been named. That's the part that costs you. The best teams I've built or been part of do one thing consistently: they talk about tradeoffs before they become incidents. They document the choice, the upside, and what they accepted as the cost. Then they revisit it when context shifts, because it always does. That discipline is the real engineering practice. 💡 Robust distributed systems aren't built from correct answers. They're built from the courage to own hard choices, communicate them clearly, and keep revisiting them as the system grows. #DistributedSystems #EngineeringLeadership #HVSays

26

6 Comments

It seems that distributed inference observability has some gaps. In terms of framing this, I am referring to inference deployments at the edge (or so called near edge).. pops close to end users. Let's say you are using ollama for some early testing and/or scaling but are using vllm in production. Traditional monitoring platforms will report on GPU/CPU load, memory usage, network status, etc, etc. However, other stuff is also happening: —GPU throttled - 100% utilization but clock speed dropped 33% —KV cache saturated causing some queue backlog —Time to first token spiked 200% from CPU contention —Another tenant's PCIe traffic impacted inference maybe some contextual drift - some hardware stresses that degrade inference performance but it is happening in ways that is generally invisible to system metrics. Most of the monitoring in the market is built for servers and takes a peek at intervals that may not make sense for inference —token generation:20-100 per second —cache saturation: spikes in seconds Thermal throttling happens instantly but traditional monitoring might see this as smooth if only glancing at the server every 30 seconds. But, you also can't grab data every 2 seconds or you might contribute to some cpu scheduling pressure. So, if you are going to run both ollama (for dev/test or smaller loads) and vLLM for production they have completely different failure modes but traditional monitoring would treat them the same. We also have a blind spot with regard to time to first token (ttft) and time per output token (tpot). We might show request latency spiking, but we need to know whether ttft spiked or tpot spiked.. The Prometheus endpoint for vLLM helps a bit here but has some blind spots relative to cpu and PCI performance. Also, increasing the Prometheus scrape frequency could contribute to some performance drift. so, I am thinking about an open source project that would be a lightweight observability agent.. large companies will likely solve this by building a giant observability layer on top of their distributed inference solution -- but I think having a more bottoms up approach that can be deployed giving teams more control of their data might make sense.. this observability agent would strive to: —have limited cpu impact/overhead — 2 second sampling with some intelligent backoff —built in ttft/tpot splitting —contextual drift detection —works with vLLM/Prometheus and Ollama API stats —embedded DB storage (duckDB?) - no external dependencies — runs at edge.. maybe federates.. Curious to get feedback on where people are hitting observability gaps.. What are you doing to monitor vLLM and/or other inference engines? What metrics do you wish you had? Drop the war stories here.. thanks..

25

1 Comment

IF YOU muted out AI LAY-OFFS look at the AI CODING OUTPUT Many engineers close their eyes/ears on news about AI's impact on job market And, yes, it might be hard to disentangle from over-hiring (in COVID) before our simply bad managerial decisions leading to companies ending up in corner and needing pruning of their organization. ⚡ But what is undeniably visible and we can all easily agree on is that we DID NOT GET +50% engineer count in last year. There is no way you can 1.5x the SW engineering in 12M. Yet, ... ... visible outputs of the SEW engineering work got 1.3 - 1.5x in last year. You can look at new Apps being launched, Code repositories growth, New websites, all around. Everywhere we see Cambrian explosion of the IT output. 👀 So let's get back to original question: If we did not get 1.5x more FTEs working in IT development (we suspect we most likely shrank it a little) and we got 1.5x of output, how does that square with AI not having real impact on engineering work yet? 2️⃣✖️ These charts are mapping just crossing from 2025 to 2026. The end of 2026 will be much more dramatic. Because when IT outputs boost beyond 1.8x or 2.0x we will get to point that Ai will be NOT ABLE but ALREADY INDEED PRODUCING equivalent of all existing engineering effort. And then some things will get nasty in some places. 🍥 Not all companies will go for greedy variant. As Jensen Huang said, those having some master dreams or ideas will try to use this production boost to create more. The SW development "creme-de-la-creme". But then there will be companies (loads of them) who did not know what exactly to build (if anything) first place. And these ... 🎛️ ... will simply turn the dial down on human costs. Because if you can use AI to build more for same, you are left with building same for (significantly) less. ⁉️ And, yes, there will be companies, who will (be not able to) deploy AI on grand scale and will use human coding to keep delivering same with same. But in era where there are other doing much more or yet another doing same cheaper, what is your "life expectancy" of company that kept doing same for same? And would you like to still work for such a company?

12

4 Comments

Imagine you're in an E4 system design interview at Meta (80LPA+ role) and the interviewer asks: If apps like Instagram are free, how do they store billions of photos and videos without charging users anything? This is a classic large-scale storage + serving problem. Btw, if you’re preparing for system design/coding interviews, check out our mock interview tool. You can use it for free here: https://lnkd.in/gpCn7t2T [1] Clarify what we actually need to solve This is not just “save files somewhere.” It is: - store billions of user-generated photos and videos durably - serve them globally with low latency - handle huge spikes in uploads and views - keep storage cost low enough that the free product still works as a business - support metadata, privacy settings, deletes, thumbnails, and multiple video qualities So the real problem is cheap durable blob storage + fast global delivery, not just databases. [2] High level approach Use an object storage architecture, very similar to S3-style thinking, plus a CDN. - user uploads media through an upload service - media gets stored as objects in distributed blob storage - metadata goes into a separate database - background workers generate thumbnails, compressed variants, and multiple resolutions - users fetch media through CDN edges, not directly from storage every time That separation is the key: - blobs in object storage - metadata in databases - delivery through CDN [3] Upload and storage flow When a user uploads a photo or video: - client asks backend for an upload URL - backend authenticates user and creates a media record - client uploads file to object storage - storage returns object key like `media/user123/abc.jpg` - metadata DB stores:   - owner_id   - object_key   - media_type   - visibility   - created_at   - processing_status Why not store media in a database? - media files are huge - databases are expensive for large blobs - object stores are cheaper, simpler, and built for this exact use case [4] Processing pipeline After upload, run async workers: - image resize - thumbnail generation - video transcoding into multiple bitrates - format conversion - moderation / safety checks This is queue-based because video processing is expensive and should not block the upload request. [5] How they keep it free The short answer: ads fund the storage bill. The technical answer: - object storage is much cheaper per GB than traditional databases - cold media can move to lower-cost storage tiers - CDN reduces repeated origin reads - media is compressed aggressively - old videos/photos are rarely accessed compared to new ones, so caching and tiering cut costs a lot At this scale, storage is not free. It is just optimized so well that ad revenue more than covers it.

790

48 Comments

System Design isn't about knowing the "right" answer—it's about knowing the trade-offs. Moving from SDE1 to SDE2 (or L3 to L4) requires a shift in perspective. You stop asking "how do I code this?" and start asking "how does this scale, fail, and evolve?" Here is a deep dive into 3 core architectural concepts that often trip up candidates in System Design interviews: 1️⃣ Monolith vs. Microservices We often rush to Microservices, but they introduce massive operational complexity (network latency, distributed transactions/sagas, eventual consistency). 💡 The SDE2 Insight: "Start with a Monolith." Microservices solve organizational scaling problems (Conway’s Law) more than technical ones. Unless you have multiple teams tripping over merge conflicts, a modular monolith is usually faster to build and easier to test. 2️⃣ Web Server vs. Application Server These terms are often used interchangeably, but they serve distinct roles in a tiered architecture. 🔹 Web Server (Nginx, Apache): The "Mail Carrier." Delivers static content, handles SSL termination, and acts as a reverse proxy. 🔹 App Server (Tomcat, Gunicorn): The "Personal Assistant." Executes business logic, talks to the DB, and generates dynamic content. Why separate them? Security (DMZ isolation) and specialized performance (Nginx handles connections; Tomcat handles CPU tasks). 3️⃣ Service Discovery In the world of Kubernetes and auto-scaling, hardcoded IPs are dead. Services need a dynamic phonebook (Service Registry). 🔸 Client-Side Discovery: The client queries the registry (e.g., Netflix Eureka) and picks an instance. Pro: Client-side load balancing. Con: High coupling. 🔸 Server-Side Discovery: The client calls a Load Balancer (e.g., AWS ELB), which handles the lookup. Pro: Simple client. Con: The LB becomes a critical component. System design is rarely black and white. It’s about choosing the complexity you are willing to manage. Which architectural pattern do you see misused the most in production? Let’s discuss in the comments. 👇 #SystemDesign #SoftwareArchitecture #Microservices #SDE2 #Engineering #Scalability

23

2 Comments