Repository navigation
Description
TL;DR
Annual holidays truly offer a chance to stop and think things through. Like a processor receiving far fewer “interrupts,” one becomes much more efficient. It seems there are only two days left before work begins again, so I might as well write down a few thoughts...
Shortly after the Apsara Conference ended, I started two projects at the same time. One analyzes off-policy RL post-training algorithms; returning to a very fundamental starting point reveals many details worth exploring. The other explores a new XPU architecture, somewhat like OpenAI Janapeno but incorporating many new insights into model architecture. Of course, the latter project is confidential, so I will not say much about it.
The intersection of the two has led to a new question: what does the AI infrastructure for a 10T-parameter model require? I want to write about this topic because there once again seem to be many misconceptions in the industry, including blind attempts to design things that are locally suboptimal. We will first discuss what a 10T model should look like from the perspective of model architecture, and then analyze what its AI infrastructure requires.
1. Model Architecture
1.1 Why Do We Still Need to Scale Models to 10T Parameters?
The first question to answer is why we need to scale to 10T parameters. In the era of pure language models, the training data generally consisted of common Internet corpora, totaling roughly 35T tokens. In practice, model parameter counts were constrained by the amount of training data. In the Agentic era—or the RSI era that may soon arrive—frontier models have access to enough training data, especially vast numbers of long-horizon Agent workflows and a great deal of new, high-value proof-oriented analysis, such as proofs of many mathematical theorems. AI4S is also likely to see major breakthroughs in the next few months. These developments are driving exponential growth in the amount of data available for model training. This is also the prerequisite for further scaling model parameter counts.
Another factor is the evolution of infrastructure: from the 96 GB HBM of early Hopper systems to the 288 GB HBM of today's Blackwell / Rubin systems, together with the development of relatively large supernode systems such as NVL72. As a result, a single supernode can theoretically host a 5T–10T model for inference.
1.2 What Should the Architecture of a 10T Model Be?
Once infrastructure and training data can both support continued scaling to 10T parameters, the next question is: what should the architecture of a 10T model look like?
First, from the perspective of compute constraints, meeting the high-speed inference requirements and cost constraints of Agent workflows means that the desired model should be sparser. Today's open-source 3T models have roughly 100B active parameters, so can a 10T model still keep its active parameter count at around 100B–150B? Second, from the perspective of memory constraints, one particularly noteworthy scaling opportunity is context scaling: can we push context length to 5M without significantly increasing compute or KV cache usage? This constraint comes not only from inference cost and speed, but more importantly from scaling extensive post-training workloads: Rollout must be sufficiently fast and concurrency sufficiently high.
A simple derivation follows. For the first problem, we may need some form of MoE with 2,048–4,096 Experts. For the second problem, the Attention Block needs better algorithmic support. This is the fundamental logic behind the new changes and experiments in model architecture by various foundation-model teams this year.
What deserves even more attention is the intricate connection hidden between these two problems. We often want Attention to be sufficiently efficient, and some approximate mathematical derivations appear to lead to algorithms such as Linear Attention. But when such an algorithm actually has to support a model with num_experts = 2048~4096, it could have disastrous consequences—for example, a large number of dead experts... On the other hand, we also want a larger num_experts so that local parameter updates can be performed more easily during post-training, avoiding a significant decline in other capabilities after one task has been enhanced. Of course, this involves some post-training RL algorithms, which I will write about later when I have more time.
This creates an algorithmic dilemma. On one hand, we need to improve the quality of Attention outputs as much as possible, enabling the MoE Routing Gate to more accurately activate the small number of Experts that are needed. On the other hand, we need to reduce the computational complexity of Attention and use a series of techniques to reduce KV cache usage. Given these two constraints, one can perhaps understand why several open-source models in China have recently begun moving toward YOCO-like approaches, such as “A Detailed Analysis of the DeepSeek-V4.1 Flash Technical Report (1): Model Architecture”.
There are also some so-called Loop Transformer approaches, but I do not believe they are correct. For details, see “Discussing the Future of Transformers: Loops Are Not What You Need”. As the article states:
Why Is the Loop Transformer Not a Correct Approach?
Let the post-Loop input be
At the same time, the perturbation to the value is Q, K, and V in the new algorithm.
What we would prefer is some recursive approach that can separate the processing of Q from that of K,V during training and inference. Could Q gain greater expressive capacity through some form of memory injection, thereby further improving selection accuracy in the subsequent MoE Routing Gate? And could K,V be reused more effectively, further easing memory capacity constraints?
This is a core question surrounding Attention algorithms. Or, viewed mathematically, could the changes to Q, K, and V in every original layer become a form of processing that exploits local K,V invariance, further fusing the original cross-layer mappings over the KV manifold into Q?
Another issue is the algorithmic deficiencies in current context management that I discovered in some highly complex agentic workflows recently. This is also a problem I discussed in “Discussing Kimi K3's KDA (2): A Comparison with Sparse Attention”: what I need is an Attention algorithm whose KV computation supports block-based composability.
These are the problems that a 10T-parameter model may have to address first.
2. What Does the AI Infrastructure for a 10T-Parameter Model Require?
2.1 How Large Should a Supernode Be?
There has recently been an interesting discussion in China. Perhaps some people have acquired a general understanding of top models in North America while also possessing strong networking and interconnect capabilities themselves, so they have begun discussing supernodes spanning 1,000 to 10,000 GPUs. This is the first issue I primarily want to discuss. There is no denying that large-scale supernodes do solve many problems, and memory semantics offer fairly clear benefits for EP-style dispatch/combine operations. But does that mean the so-called “one GPU per expert” claim also holds? Behind EP lie a requirement for “synchronization” and reliability constraints on the entire ScaleUP domain.
As the scale grows, so does the variance in “compute” and “transport” latency. In other words, long-tail latency across the system causes utilization to decline significantly as the node count increases. Poorly designed ScaleUP protocols can also have a major impact. When I began working on NetDAM at Cisco in 2020, it was essentially one of the earliest attempts at Ethernet-based ScaleUP. I later had many in-depth discussions with BRCM, but BRCM made a mistake when designing SUE. At the time, I sent them NVIDIA's FinePack paper and asked them to learn from its mechanism to avoid Ethernet header overhead. However, they did not ask me to review their subsequent SUE design, leaving the packing scheme vulnerable to severe tail latency. This would further constr 6B74 ain the size of the entire ScaleUP domain.
Another problem is reliability. The MTBF of the overall system also falls sharply as supernodes grow and the forced use of optical interconnects reduces reliability. I am therefore strongly opposed to supernodes with thousands of GPUs. One can also note that the Ascend 950 infrastructure components recently open-sourced by DeepSeek use supernodes of only 128 GPUs, rather than the widely publicized deployments of supernodes with thousands or even tens of thousands of GPUs.
Under these constraints, if ScaleUP needs to support at most 128 GPUs, then a switch with Radix = 128 provides 12.8 Tbps using 112G SerDes and 25.6 Tbps using 224G SerDes. We do not actually need a ScaleUP system with Radix = 1024, nor perhaps products such as 448G SerDes, because the optical-transmission reliability problems introduced by 448G SerDes have not yet been resolved, while the latency increase caused by FEC and similar mechanisms is significant.
Under a constraint of Radix = 128, or at most Radix = 256, figuring out how to build a switch chip and a ScaleUP protocol is actually very easy.
2.2 Infrastructure Stability
Training a model at the 10T scale may require 200,000 GPUs for a single training job. The reliability of the overall system is therefore an issue well worth discussing, especially the trade-off between ScaleUP and ScaleOut. My view is very clear: use a lossless credit-based scheme within the ScaleUP domain; for now, keep the ScaleUP domain predominantly copper-interconnected and confined to one or two racks; and focus more on improving the reliability of lossy ScaleOut. For details, see “Discussing Reliable Transport for RDMA and ScaleUP”.
Reliable transport is, in essence, about spending compute to compensate for link instability. When a single training job runs across a 200,000-GPU cluster, lossless RDMA will inevitably reach a state in which it becomes unusable. That is why AMD / BRCM / Meta and some Chinese vendors have all promoted Lossy support in their customized RoCE protocols this year. I have been raising these issues for several years. Once training reaches a certain scale, Lossless does not work at scale in the way some people advertise.
However, when these vendors heavily modify RoCE to support SACK, their capabilities still seem insufficient. For example, a well-known vendor reported at this year's Hot Chips that, under 1% packet loss, a 400 Gbps NIC could achieve 160 Gbps of actual throughput with the help of SACK. This is what they consider SOTA performance. What we want to say is that, when development of CIPU 2.0 was completed three years ago, we had already achieved 90% Goodput at 5% packet loss, reaching an industry-leading level.
Of course, many people will say that packet loss currently observed in networks is generally only around 0.1%; make that work well and it is enough—why require near-full-load operation at 5% packet loss while avoiding the bandwidth consumed by spurious retransmissions? The 5% figure is not arbitrary; it comes from real business requirements. In large-scale ScaleOut networks, particularly when scaling to 200,000 GPUs, multi-tier switch fabrics and the extensive use of optical modules and fibers mean that you certainly do not want to see large fluctuations in MFU throughout training. Yet excessive phase noise in the crystal oscillators of optical-module DSPs, together with hotspots in high-density racks that cause DSP CTLE/DFE drift or contamination of fiber connectors, can all cause packet loss and link instability. There is another significant problem: assuming a switch has Uplink Radix = 32, the routing black hole caused by one failed uplink produces 3% packet loss, while two failed uplinks produce 6%. There are also packet-loss problems caused by the Scale-across interconnects required for a 200,000-GPU fabric. This is why I set the reliable-transport target such that the system must maintain high goodput even at a 5% packet-loss rate.
2.3 Infrastructure Security
I have recently been discussing a number of security-related issues. At last week's Apsara Conference, we also discussed multi-tenant security in cloud environments in “What Is a DPU in the Agentic Era? How CIPU Builds Alibaba Cloud's Core Technological Competitiveness in AI Infrastructure for AI Agents, Inference, Training, and More”. As the first team in the world to launch bare-metal instances, we actually have many security-related designs, but they cannot be discussed publicly. NVIDIA has just released the Open Agent Safety Platform.
It divides Agent security into two complementary layers. OpenShell, close to the execution environment, restricts the files, networks, services, and credentials that an agent can actually access. Sentry places monitoring and some policy enforcement in an independent Trust Domain on the BlueField-4 DPU, adding an independent line of defense for scenarios in which the host is compromised. However... some fundamental security issues still have not been thought through. In essence, it still approaches the problem from a networking perspective, using some Telemetry and then blocking traffic through flow tables on BF4. They have not adequately addressed many aspects of host security...
The most fundamental problem is that Agentic workflows have broken the original security boundary. Previously, enterprise software was largely deployed in a trusted and controlled manner, so no multi-tenant isolation technology was needed within a user's single data center; simple security access control was sufficient. But when LLMs generate large amounts of code, the entire trust boundary changes. Even Agentic RL training at many foundation-model teams may need task-level multi-tenant security isolation, especially for large numbers of sandbox environments. Yet the industry seems never to have clearly understood the distinction between bare machine and bare metal. As for some security problems, they will understand once they suffer the consequences.
2.4 Infrastructure Performance
A few days ago, I read a Zhihu article titled “Debate Series VI: Superscalar or Static Pipeline Scheduling”. Both it and the OpenAI Janapeno article “From the Shortcomings of NVIDIA GPUs to OpenAI Jalapeño” raised one issue: “superscalar.”
I actually raised this issue last year in “Inside NVIDIA GPU: Discussing Blackwell's Shortcomings and Predicting Rubin's Microarchitecture”. I remember discussing it with Dr. Liao Heng at some event at Huawei HiSilicon last year. Jensen Huang also forwarded this article internally at NVIDIA; I really do not know how much NVIDIA's engineers took to heart. Unfortunately, some of the design considerations behind it are confidential, so I will not say more...
That is roughly it. Having seen some erroneous claims, I wrote down a few thoughts offhand. End of article.

Activity