Xiaoning Wang

Graphmend Fixes PyTorch 2 Graph Breaks, Enabling Compilation of 75% More Models and Reducing Fallbacks by 8% GraphMend, a new compiler, automatically eliminates interruptions in PyTorch 2 programs by transforming source code, achieving up to 75% reductions in processing time and boosting performance across several machine learning models. #quantum #quantumcomputing #technology https://lnkd.in/e5kSm8vt

1

What if the next optimization for LLMs isn’t about computing faster, but looking smarter? I recently explored Declarative Attention an approach where an LLM can decide which part of its context it actually needs to attend to, instead of reading the entire KV cache. The idea is simple: LLM decides where to look → Runtime fetches that context → Model reasons over it. This could be particularly interesting for long-context and agentic workloads. I wrote a short breakdown of the idea and its trade-offs: 🔗 https://lnkd.in/dS8HF3xU Can LLMs learn to decide where they should look before deciding what to think? #AI #LLM #GenAI #Inference #MachineLearning

7

2 Comments

Agent Memory - is it a critical bottleneck? A new survey from researchers across NUS, Renmin, Fudan, and other institutions offers the most comprehensive taxonomy of agent memory to date. The core insight: existing frameworks like "short-term vs. long-term memory" are insufficient. The field has become fragmented, with researchers using the same terms to describe fundamentally different systems. The authors propose a unified lens of "forms, functions, and dynamics." Memory forms include token-level (explicit, editable text), parametric (encoded in model weights), and latent (hidden states and KV caches). Functions are divided into factual memory (what the agent knows), experiential memory (how it improves), and working memory (what it's thinking about now). Dynamics covers how memories are formed, evolve, and retrieved over time. They identify the following convergence: reinforcement learning is increasingly taking over memory management, moving from hand-crafted rules toward fully learnable systems. The paper argues we may see agents that autonomously design their own memory architectures through RL optimization. All in all, this survey provides a conceptual foundation for building the next generation of agents that can genuinely learn and adapt over time. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

138

9 Comments

GPU idle time is often a symptom of data architecture, not compute capacity. Coupang's ML platform faced a strict scheduling constraint: training jobs were physically tied to the specific clusters where the datasets resided. Migrating data to overflow GPU clusters during peak times required a manual, time-consuming data preparation step before jobs could even enter the queue. To solve this, they introduced a distributed cache to decouple compute and storage. ▪️ Decoupled Scheduling: Workloads can now run on any available GPU across multiple regions without requiring manual data migration first. ▪️ Continuous Pipelining: The manual data-copying step was eliminated. Jobs start immediately and pull data directly from the central data lake as needed. ▪️ Improved I/O: Utilizing local NVMe for the caching layer delivered ~40% better I/O performance compared to their standard cloud parallel file system. Read more: https://lnkd.in/gr26EDvs #MLOps #AIInfrastructure #GPU #DistributedSystems #PlatformEngineering

2

We are very excited to publish our research on Contextual Sparsity in the PyTorch community. Explore our in-depth white paper https://lnkd.in/gcFN4abk for exploring what’s state of the art in Sparsity and how NimbleEdge is pushing the boundaries of on-device LLM inference optimization.

2

Announcing torchcomms – a new PyTorch API for distributed programming designed to support scalability, fault tolerance, heterogeneity, and extensibility. Torchcomms makes it easier for developers to leverage innovations in existing and new collective communications backends like NCCL, RCCL, and NCCLX. 🖇️ Read the full blog from Team torchcomms at Meta here: https://lnkd.in/gdwbnMSV 👥 Contributors include: 🍃 Tristan Rice, Subodh Iyengar, Junjie W., Feng Tian, James Hongyi Zeng, Sudharssun Subramanian, Pavan Balaji, Qiye Tan, Rodrigo De Castro, Saif Hasan, Min Si, Yifan Mao, Dingming Wu, Zhaoyang Han, Blake Matheny, Art Zhu, Denis Boyda, Regina Ren, Jingyi Yang, Bingzhe Liu, Shuqiang Zhang, Mingran Yang, Cen Zhao, Adi Gangidi, Ashmitha Jeevaraj Shetty, Bruce Wu, Ching-Hsiang Chu, Yulun Wang, Srinivas Vaidyanathan, Chris Gottbrath, Davide Italiano, Shashi Gandham, Omar Baldonado #Torchcomms #PyTorch #API #OpenSourceAI #PyTorchCon

169

3 Comments

Given a single model, how do we improve an #LLM’s reasoning performance with limited resources 💻 and inference time ⌛️? Can a smaller 1.5B model outperform a 7B model without incurring long inference time from sequential queries?    In the work of Wenyang Hu, Gregory Lau, See-Kiong Ng, Bryan Kian Hsiang Low et al., we introduce the framework called Dipper to create #LLMs ensembles from an optimized set of diverse reasoning prompts to improve performance. Dipper runs queries in parallel with a prompt optimization method inspired by Determinantal Point Processes (DPP), making it super fast ⏩️ and effective. Furthermore, Dipper can work with LLM APIs without model access 📦!   With Dipper, we demonstrated how a small ensemble of just three 1.5B models can outperform a 7B model on a range of math and non-math reasoning tasks, while taking almost the same inference time and just < 3x compute for a normal query thanks to accelerated batch inference methods 😱 !   Find out more at #EMNLP2025 (Poster 4492) at Hall C, Session 11, Nov 6 at 16:30! Paper: https://lnkd.in/gsDzJxPb

55

2 Comments