<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://fables.day/feed.xml" rel="self" type="application/atom+xml" /><link href="https://fables.day/" rel="alternate" type="text/html" /><updated>2026-11-18T03:36:13+00:00</updated><id>https://fables.day/feed.xml</id><title type="html">Daily Fables</title><subtitle>Graduate-level cloud &amp; distributed-systems concepts, told as bilingual fables.</subtitle><entry><title type="html">不回前门的锦船 · The Barges That Left by the River</title><link href="https://fables.day/2026/11/17/direct-server-return/" rel="alternate" type="text/html" title="不回前门的锦船 · The Barges That Left by the River" /><published>2026-11-17T00:00:00+00:00</published><updated>2026-11-17T00:00:00+00:00</updated><id>https://fables.day/2026/11/17/direct-server-return</id><content type="html" xml:base="https://fables.day/2026/11/17/direct-server-return/"><![CDATA[<section class="zh">
  <p>沧浪江畔的”云津渡锦行”，接的是全天下最轻的帖子，发的是全天下最重的船。</p>

  <p>各州绸商派快艇来订货，递进前津总署窗口的不过是一枚三寸长的竹筹，上面刻着一行小字：”扬州四号埠，订云锦二百匹。”总署掌事沈阔接过竹筹，看一眼不过半息工夫；可后院十二座临江水阁织坊要交这笔货，却得装满整整二十口楠木大箱、压沉一艘吃水五尺的乌篷重船。</p>

  <p>偏偏锦行立过一条传了二十年的老规矩：<strong>外州绸商只认前津总署的大门与”云津渡锦行”的总号官印。</strong></p>

  <p>为了守这条规矩，沈阔每次收下竹筹，都要先刮去商家的埠头名号，改成”交前津总署沈阔收”，派学徒送进某座水阁；等水阁织完二百匹云锦，又得套上牛车，吭哧吭哧拉回前津总署那扇一丈宽的青石窄门里，由沈阔核对旧账、盖上”云津渡锦行”总印、重新写上”发往扬州四号埠”，再从总署门前的小码头装船出海。</p>

  <p>起初只有三座水阁时，窄门还勉强转得开。等到生意兴旺、后院扩到十二座水阁，怪事发生了——</p>

  <p>十二座水阁的织机才开了一半，前津总署那扇一丈宽的青石门却从五更堵到了半夜。门外递竹筹的快艇三息就能收完十支，门洞里往外拉云锦的牛车却顶死了辕木，进不去也出不来。更叫人憋闷的是，十二座水阁明明背靠着浩浩荡荡的沧浪大江，各自都有宽敞的后水门，却只能眼睁睁看着大江空荡荡，硬把万斤重箱往沈阔那道窄门缝里塞。</p>

  <p>水阁总管林清晏登上总署望楼，看了一眼堵成长龙的牛车，只问了沈阔一句话：</p>

  <p><strong>“进门的是三寸竹筹，出门的是万斤重船——凭什么重船非要回头钻你这扇收竹筹的针眼？”</strong></p>

  <p>当晚，林清晏把老规矩彻底翻了过来：</p>

  <p>从明日起，商家递进来的三寸竹筹——上面写着<strong>“扬州四号埠呈，云津渡锦行收”</strong>——沈阔<strong>一个字也不许改</strong>。他只管挑一座最清闲的水阁，把原封不动的竹筹塞进一只院内布袋，袋口挂一块木牌写上<strong>“送七号水阁侧门”</strong>，让人隔墙递过去。七号水阁拆开布袋，照着竹筹上原本留着的”扬州四号埠”，直接在自家后水门装满云锦大船，在封条上盖上<strong>“云津渡锦行发”</strong>，推开后闸直入沧浪大江，顺流直奔扬州——<strong>连一片船帆都不必再绕回前津总署！</strong></p>

  <p>沈阔听得眼睛发亮，连夜让人铸了十二块一模一样的”云津渡锦行”铜牌，分挂在十二座水阁檐下——毕竟水阁账房若不挂总号铜牌，见了写着”云津渡锦行收”的竹筹只会当成投错门的外信扔出去，封船时也没资格盖总号的名头。</p>

  <p>谁知第二天清晨第一声鸡叫，乱子就出了。</p>

  <p>一名初来乍到的徽州船老大摇橹停在渡口三岔河心，扯着嗓子朝岸上喊了一嗓子：<strong>“敢问哪一扇门是云津渡锦行？”</strong></p>

  <p>话音未落，后院墙头十二座水阁的伙计瞧见自家檐下挂着铜牌，齐刷刷探出脑袋高喊：”我这儿是！”“往我这儿摇！”船老大哪分得清总署和水阁，顺手把船摇进了喊声最响的一号水阁。不到半个时辰，一半商船全挤进了一号和二号水阁，后头十座水阁冷冷清清，沈阔手里的分派账册成了一张废纸。</p>

  <p>沈阔一拍脑门，当天晌午便立下<strong>“哑牌规矩”</strong>：</p>

  <p>十二座水阁檐下的”云津渡锦行”铜牌统统摘下来，锁进各阁不见天日的<strong>内堂密室</strong>里。往后凡有外客在三岔河心高喊”谁是云津渡锦行”，十二座水阁一律闭嘴装哑，<strong>唯独前津总署的沈阔一个人许开口应声</strong>；可一旦沈阔用院内布袋把竹筹塞进了某座水阁的侧门，阁内账房抬头看见密室里供着的铜牌，便认下这桩买卖，后水门发出的锦船也照旧堂堂正正盖上”云津渡锦行”的大印。</p>

  <p>这一招立竿见影。前津总署门前再不见一辆牛车，沈阔摇着折扇收竹筹，十二座水阁后闸千帆竞发，吞吐量一夜翻了十倍。</p>

  <p>直到那年深秋，皇商大单涌来，江边地界不够用了。林清晏翻过东边的青石岭，在十里外的<strong>栖霞谷</strong>新开了八座大水阁，谷口溪流同样直通沧浪大江，只是从前津总署往栖霞谷送信，必须跨过三座官府设立的<strong>官驿石桥</strong>。</p>

  <p>开张头一天早上，沈阔挑出一支竹筹，照老法子塞进写着”送栖霞谷三号阁”的院内布袋，交给跑腿小厮上路。</p>

  <p>一炷香后，沈阔正喝着茶，忽见官驿驿卒急匆匆跑进总署，恭恭敬敬递上一支竹筹：”沈掌事，刚从一号石桥转回来的信，请您签收！”</p>

  <p>沈阔低头一看，差点把茶盏摔了——<strong>正是他一炷香前亲手发出去的那支扬州竹筹！</strong> 他不信邪，重装一只布袋再发；又过一炷香，驿卒满头大汗又把同一支竹筹送回了总署案头。整整一上午，几十支竹筹在总署和一号石桥之间来回打转，栖霞谷八座水阁连一根丝都没织上。</p>

  <p>老巡驿赵叔路过桥头，看了一眼就笑了：”沈掌事，你那院内布袋上的木牌，只在自家院墙里管用！一上官驿石桥，桥官按朝廷驿规，<strong>第一件事就是扒掉你那只私家布袋</strong>，只看里头竹筹正身写的收信人。竹筹上明晃晃刻着’云津渡锦行收’——桥官翻开官驿舆图一查，’云津渡锦行’在哪儿？不就在你前津总署吗！他不往你这儿送往哪儿送？”</p>

  <p>沈阔愣在当场：改竹筹上的收信人吧，到了栖霞谷没法用总号大印直发商船；不改吧，一过官驿石桥就被桥官按舆图踢回总署！</p>

  <p>林清晏赶到桥头，取过一支竹筹端详片刻，转身从驿站买来一摞过境专用的<strong>火漆竹筒</strong>。</p>

  <p>“竹筹一字不改，石桥规矩也不破。”林清晏把刻着<strong>“扬州四号埠呈，云津渡锦行收”</strong>的竹筹原封不动塞进竹筒，封上蜡，在<strong>竹筒外壁</strong>清清楚楚写上一行官驿地址：<strong>“前津总署发，栖霞谷三号阁亲启。”</strong></p>

  <p>这一次，三座官驿石桥的桥官只验竹筒外壁的”栖霞谷三号阁”，一路快马放行。竹筒到了栖霞谷三号阁，账房啪地敲碎外筒扔进废纸篓，露出里头完好无损的三寸竹筹——既看得见”扬州四号埠”的真址，又对得上内堂密室里的”云津渡锦行”哑牌。当天傍晚，一艘艘满载云锦的大船便从栖霞谷溪口直驶沧浪大江，扬帆远去。</p>

  <p>——到这儿你大概已经认出来了：这就是大规模高性能四层负载均衡里的<strong>直接服务器返回</strong>（<em>Direct Server Return, DSR</em>）与<strong>三层直接服务器返回</strong>（<em>L3DSR</em>）。前津总署是四层负载均衡器（如 <em>IPVS DR</em> 模式、Meta <em>Katran</em>、<em>Cilium DSR</em>）；”云津渡锦行”总号是公开的<strong>虚拟 IP</strong>（<em>VIP</em>）；三寸竹筹与万斤锦船是极不对称的请求包与响应流；最初隔墙递送的院内布袋是<strong>二层 MAC 地址重写</strong>；锁在内堂闭嘴不应声的”哑牌”是后端挂在本地回环网卡（<code class="language-plaintext highlighter-rouge">lo</code>）上并开启 <strong>ARP 抑制</strong>的 <em>VIP</em>；而跨过官驿石桥（三层路由器）时套在外面的火漆竹筒，正是 <em>IP-in-IP</em> 或 <em>GUE / Geneve</em> 隧道封装的 <strong><em>L3DSR</em></strong>。</p>

  <h3 id="section">这是什么</h3>

  <p>在绝大多数互联网业务（视频流、对象存储下载、CDN、高吞吐 API）里，进出流量天生极度不对称：<strong>客户端发来的请求往往只有几百字节（一枚 <code class="language-plaintext highlighter-rouge">HTTP GET</code>），而服务端吐回去的响应动辄几兆甚至上百兆字节。</strong></p>

  <p>传统的对称式负载均衡（七层反向代理或四层 <em>SNAT / Full-NAT</em>）要求请求和响应都必须原路穿过负载均衡器：入站时负载均衡器把目的地址改成后端真实 IP（<em>RIP</em>）、把源地址改成自己；出站时后端把庞大响应发回负载均衡器，再由它把源地址改回虚拟 IP（<em>VIP</em>）转给客户端。结果，后端机器扩了十倍，所有出站大象流却全卡在负载均衡器的网卡带宽与连接跟踪表（<em>conntrack</em>）瓶颈上。</p>

  <p><strong>直接服务器返回（<em>DSR</em>）</strong> 彻底拆开了进出路径，让轻巧的请求走负载均衡器，沉重的响应由后端服务器直连客户端返回：</p>

  <ol>
    <li><strong>二层 DSR（<em>L2 DSR</em>，MAC 重写）</strong>：当负载均衡器与后端处在同一二层广播域时，负载均衡器收到发往 <code class="language-plaintext highlighter-rouge">VIP</code> 的数据包后，<strong>完全不碰三层 IP 头</strong>（保留客户端真实源 IP 和目的 <code class="language-plaintext highlighter-rouge">VIP</code>），只把二层以太网帧的目的 MAC 地址改写成选定后端的网卡 MAC。</li>
    <li><strong>回环网卡绑定 <code class="language-plaintext highlighter-rouge">VIP</code> 与 ARP 抑制</strong>：后端服务器之所以不会丢弃目的地址为 <code class="language-plaintext highlighter-rouge">VIP</code> 的包、且能用 <code class="language-plaintext highlighter-rouge">src IP = VIP</code> 直接给客户端回包，是因为每台后端都在本地回环接口 <code class="language-plaintext highlighter-rouge">lo</code> 上配置了 <code class="language-plaintext highlighter-rouge">VIP/32</code>。为了防止故事里”十二座水阁齐声抢答”的 <strong>ARP 抢劫（ARP Flux）</strong>，Linux 后端必须配置内核参数 <code class="language-plaintext highlighter-rouge">arp_ignore=1</code> 与 <code class="language-plaintext highlighter-rouge">arp_announce=2</code>——物理网卡对局域网内询问 <code class="language-plaintext highlighter-rouge">VIP</code> 的 ARP 广播一律保持沉默，只有负载均衡器对外宣告 <code class="language-plaintext highlighter-rouge">VIP</code>。</li>
    <li><strong>三层 DSR（<em>L3DSR</em>，隧道封装）</strong>：现代云数据中心普遍采用跨机架的 Spine-Leaf 三层路由网络。一旦跨过三层路由器，二层 MAC 头就会像私家布袋一样被剥离；路由器一看内层目的 IP 是 <code class="language-plaintext highlighter-rouge">VIP</code>，就会按路由表把包踢回负载均衡器形成死循环。<em>L3DSR</em>（如 Meta <em>Katran</em> 的 <em>IP-in-IP</em>、Google <em>Maglev</em> 的 <em>GUE</em>、或 <em>Cilium</em> 的 <em>Geneve / TCP Option DSR</em>）把原始 IP 包原封不动塞进一层外层隧道头（外层目的 IP 为后端 <code class="language-plaintext highlighter-rouge">RIP</code>）。三层路由网按外层 <code class="language-plaintext highlighter-rouge">RIP</code> 把包送到任意机架的后端；后端剥掉外层壳，拿到客户端真实 IP 与 <code class="language-plaintext highlighter-rouge">VIP</code>，随后不走隧道、直接以 <code class="language-plaintext highlighter-rouge">src = VIP</code> 把响应大流送上公网或骨干网。</li>
  </ol>

  <h3 id="section-1">为什么重要</h3>

  <p><em>DSR / L3DSR</em> 让四层负载均衡摆脱了出站带宽的物理枷锁——几台仅处理入站小包的负载均衡器，就能轻松调度身后数百 Gbps 甚至数 Tbps 的出站流量，同时<strong>无需任何 <code class="language-plaintext highlighter-rouge">X-Forwarded-For</code> 或 <code class="language-plaintext highlighter-rouge">PROXY Protocol</code> 就能在内核四层保留客户端真实源 IP</strong>。</p>

  <p>但在生产环境落地 <em>L3DSR</em> 时，有三个极容易踩中的深水暗礁：</p>

  <ul>
    <li><strong>外层火漆竹筒撑爆桥洞（MTU 与分片黑洞）</strong>：<em>L3DSR</em> 在原始包外多套了一层 <code class="language-plaintext highlighter-rouge">IP-in-IP</code>（20 字节）或 <code class="language-plaintext highlighter-rouge">GUE / Geneve</code>（28～50 字节）隧道头。如果客户端发来一个满载 1500 字节的上传大包（如 <code class="language-plaintext highlighter-rouge">HTTP POST</code>），套壳后就会超出机房物理网络 1500 字节的 MTU 上限！只测短小的 <code class="language-plaintext highlighter-rouge">GET</code> 请求时一切正常，一传大文件连接就神秘卡死。工程上必须在底层开启巨型帧（Jumbo Frames）、在边缘做 TCP MSS 钳制（MSS Clamping），或正确处理路径 MTU 探测。</li>
    <li><strong>门卫把自家人当假冒犯拦下（反向路径过滤 <code class="language-plaintext highlighter-rouge">rp_filter</code> 与云厂商防伪检查）</strong>：因为后端进站走的是 <code class="language-plaintext highlighter-rouge">RIP</code> 隧道包，出站发的却是源地址为 <code class="language-plaintext highlighter-rouge">VIP</code> 的裸包，路径天生不对称。如果后端开启了严格的 Linux 反向路径校验（<code class="language-plaintext highlighter-rouge">rp_filter=1</code>），或者公有云弹性网卡开着默认的”源/目的 IP 检查”（Source/Destination Check），网卡和内核会把后端发出的 <code class="language-plaintext highlighter-rouge">src = VIP</code> 响应包当成伪造源地址攻击当场丢弃。</li>
    <li><strong>退信也得找回同一座水阁（ICMP 差错报文一致性哈希）</strong>：如果客户端到后端的回程路上遇到窄路，中间路由器退回来的 <code class="language-plaintext highlighter-rouge">ICMP Packet Too Big</code> 差错包目的地址也是 <code class="language-plaintext highlighter-rouge">VIP</code>，会先落到负载均衡器头上。负载均衡器必须拆开 ICMP 载荷里嵌着的内层五元组，用完全相同的一致性哈希算法（如 <em>Maglev</em> 环）把这封退信精准转给正在发船的那一台后端，否则连接就会永远停在重传死局里。</li>
  </ul>

  <p>它道出了高性能网络架构里一条极迷人的不对称法则：<strong>调度权不必等于搬运权——谁最懂分派，就让谁只碰那枚三寸竹筹；至于万斤重船，大江就在身后，何必回头。</strong></p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li><strong>三寸订货竹筹与压沉五尺的万斤云锦船</strong> —— 极度不对称的入站小请求（<code class="language-plaintext highlighter-rouge">HTTP GET</code> / 信令）与出站大响应流（视频块、镜像层、数据流）</li>
    <li><strong>“云津渡锦行”总号与前津总署沈阔</strong> —— 对外公布的虚拟 IP（<em>VIP</em>）与四层负载均衡器（<em>IPVS</em> / <em>Katran</em> / <em>Cilium</em>）</li>
    <li><strong>牛车把云锦拉回一丈宽的前津窄门换票盖章</strong> —— 传统对称式代理 / <em>SNAT</em>，出站大流量被迫回穿负载均衡器造成带宽与状态表瓶颈</li>
    <li><strong>不改竹筹一字、只套”送七号阁侧门”院内布袋</strong> —— 二层 <em>DSR</em>（仅重写以太网目的 MAC 地址，完整保留客户端源 IP 与目的 <code class="language-plaintext highlighter-rouge">VIP</code>）</li>
    <li><strong>十二座水阁齐声抢答三岔河心船老大的问路</strong> —— 未做 ARP 抑制时的 <strong>ARP 抢劫 / ARP Flux</strong>，后端直接响应 <code class="language-plaintext highlighter-rouge">VIP</code> 的 ARP 广播导致流量失控</li>
    <li><strong>锁进内堂密室、对外闭嘴装哑的”哑牌规矩”</strong> —— 在后端 <code class="language-plaintext highlighter-rouge">lo</code> 回环网卡绑定 <code class="language-plaintext highlighter-rouge">VIP/32</code>，并设 <code class="language-plaintext highlighter-rouge">arp_ignore=1</code>、<code class="language-plaintext highlighter-rouge">arp_announce=2</code> 静默物理口 ARP</li>
    <li><strong>官驿石桥扒掉私家布袋、按竹筹把信踢回总署</strong> —— 三层路由器剥离二层 MAC 头并按目的 <code class="language-plaintext highlighter-rouge">VIP</code> 路由，导致二层 <em>DSR</em> 跨网段形成路由死循环</li>
    <li><strong>套在竹筹外层的过境”火漆竹筒”</strong> —— <em>L3DSR</em> 隧道封装（<em>IP-in-IP</em> / <em>GUE</em> / <em>Geneve</em>），外层按后端 <code class="language-plaintext highlighter-rouge">RIP</code> 跨三层路由，内层保留原包直回江面</li>
  </ul>
</section>
<section class="en">
  <p>At the Cloud-Ford Brocade Guild beside the Canglang River, what comes in through the front window weighs less than an ounce, and what leaves out the back weighs ten tons.</p>

  <p>Silk merchants from downriver arrive in swift skiffs and slide a three-inch bamboo tally through the lattice of the Front Dispatch Gatehouse: <em>Yangzhou Pier Four — two hundred bolts of cloud-brocade.</em> Dispatch Master Shen Kuo needs half a breath to read a tally. Fulfilling it, however, takes twenty camphorwood crates heavy enough to sink a cargo barge five feet into the water.</p>

  <p>For twenty years the guild obeyed one iron rule: <strong>out-of-province merchants deal only with the Front Gatehouse and trust only crates bearing the grand seal of the Cloud-Ford Brocade Guild.</strong></p>

  <p>To keep that rule, every time Shen Kuo accepted a bamboo tally he scraped off the merchant’s pier address, wrote <em>“Return to Shen Kuo, Front Gatehouse,”</em> and sent a runner to one of the twelve riverside weaving houses behind the wall. Once two hundred bolts were woven, ox-carts had to haul all twenty crates <em>back</em> into the ten-foot stone archway of the Front Gatehouse, where Shen Kuo checked his ledger, stamped <strong>“Cloud-Ford Brocade Guild”</strong> on the wax, rewrote <em>“To Yangzhou Pier Four,”</em> and loaded the waiting river barges at the narrow front dock.</p>

  <p>With three weaving houses, the archway held. With twelve, absurdity set in.</p>

  <p>Half the looms in the rear weaving houses stood idle while ox-carts locked axles inside the ten-foot stone gatehouse from dawn until midnight. Outside, skiffs handing in bamboo tallies cleared the window in three heartbeats; inside, outbound brocade wagons choked the passage solid. Worst of all, every single weaving house backed directly onto the broad, open Canglang River with its own wide water-gate — yet all twelve left the great river empty so they could jam ten-ton loads back through Shen Kuo’s keyhole.</p>

  <p>Chief Mistress Lin Qingyan climbed the gatehouse watchtower, looked down at the snarled ox-carts, and asked Shen Kuo a single question:</p>

  <p><strong>“What enters is a three-inch bamboo tally; what leaves is a ten-ton barge. Why must the barge squeeze back through the needle’s eye that took the tally?”</strong></p>

  <p>That evening she turned the twenty-year rule inside out.</p>

  <p>Starting at dawn, when a merchant’s bamboo tally arrived — carved <strong>“From Yangzhou Pier Four, To Cloud-Ford Brocade Guild”</strong> — Shen Kuo was forbidden to alter a single stroke on it. He simply picked the least busy weaving house, dropped the untouched tally into a courtyard linen pouch tagged <strong>“To Weaving House Seven, Side Door,”</strong> and tossed it over the inner wall. Weaving House Seven opened the pouch, read <em>Yangzhou Pier Four</em> straight off the original tally, loaded twenty crates onto a barge at its <strong>own rear water-gate</strong>, stamped the wax <strong>“From Cloud-Ford Brocade Guild,”</strong> and poled straight out onto the broad Canglang River toward Yangzhou — <strong>never showing a sail at the Front Gatehouse again.</strong></p>

  <p>Shen Kuo was delighted. Overnight he had twelve identical brass plaques cast reading <em>Cloud-Ford Brocade Guild</em> and hung one under the eaves of every weaving house — for without the guild’s own name over its door, a workshop clerk seeing a tally addressed <em>To Cloud-Ford Brocade Guild</em> would toss it aside as misdelivered mail, and would have no right to stamp the guild’s seal on outgoing barges.</p>

  <p>At first cockcrow the next morning, chaos broke loose.</p>

  <p>A newcomer boatman from Huizhou rested his oar at the three-way canal fork outside the port and cupped his hands to shout: <strong>“Which door here is the Cloud-Ford Brocade Guild?”</strong></p>

  <p>Before Shen Kuo could open his mouth, apprentices at all twelve weaving houses glanced up at the shiny new brass plaques under their eaves, leaned over the wall, and bellowed back at once: <em>“Right here!”</em> <em>“Row to Dock One!”</em> <em>“No, Dock Two!”</em> Knowing no better, the boatman steered straight into whichever workshop shouted loudest. Within half an hour, half the morning’s skiffs were wedged outside Houses One and Two, Houses Three through Twelve sat deserted, and Shen Kuo’s careful balancing ledger at the Front Gatehouse was worthless.</p>

  <p>By noon Shen Kuo laid down the <strong>Silent Plaque Rule</strong>:</p>

  <p>Every weaving house took down the <em>Cloud-Ford Brocade Guild</em> plaque from its outer eaves and locked it inside a <strong>windowless inner seal-room</strong>. Whenever a stranger out at the canal fork shouted <em>“Who here is the Cloud-Ford Brocade Guild?”</em>, all twelve weaving houses kept dead silence — <strong>only Shen Kuo at the Front Gatehouse was permitted to answer public calls.</strong> Yet the instant Shen Kuo’s courtyard pouch came through a workshop’s side door, the clerks inside saw the brass plaque hanging in their inner room, accepted the tally as addressed to themselves, and stamped outbound river barges with the grand guild seal as before.</p>

  <p>The bottleneck vanished overnight. Not a single ox-cart crossed the Front Gatehouse courtyard; Shen Kuo sat fanning himself over feather-light bamboo slips while behind the wall twelve rear water-gates launched heavy barges straight onto the river at ten times the old pace.</p>

  <p>Then came autumn, and an imperial tribute order too large for the riverbank. Lin Qingyan crossed the stone ridge ten miles east into <strong>Mist-Creek Valley</strong>, where eight new weaving houses opened directly onto a deep tributary of the Canglang River. Between the Front Gatehouse and Mist-Creek Valley, however, stood three <strong>Imperial Post Bridges</strong> staffed by government relay guards.</p>

  <p>On opening morning, Shen Kuo slipped a Yangzhou bamboo tally into a courtyard linen pouch tagged <em>“Mist-Creek House Three”</em> and sent a runner up the ridge.</p>

  <p>One incense-stick later, Shen Kuo was sipping tea when an imperial bridge courier trotted into the Front Gatehouse and bowed: <em>“Master Shen — mail routed back from Post Bridge One!”</em></p>

  <p>Shen Kuo looked down and nearly dropped his cup. <strong>It was the exact Yangzhou bamboo tally he had dispatched twenty minutes earlier.</strong> Thinking the runner had blundered, he bagged it afresh and sent it out again; twenty minutes later a sweating bridge courier laid the very same tally back on his desk. All morning long, dozens of tallies spun in a tight loop between the Front Gatehouse and Post Bridge One while eight brand-new weaving houses in Mist-Creek Valley wove not an inch of silk.</p>

  <p>Old Inspector Zhao happened past the bridge, took one look, and laughed. “Master Shen, your courtyard linen pouches only mean something inside your own compound! The moment a parcel steps onto an Imperial Post Bridge, bridge regulations say <strong>strip off local pouches</strong> and route strictly by the addressee carved on the letter itself. And what is carved on that bamboo tally? <em>To Cloud-Ford Brocade Guild!</em> The bridge officer opens his imperial road atlas, sees <em>Cloud-Ford Brocade Guild = Shen Kuo’s Front Gatehouse</em>, and dutifully marches it right back to your desk!”</p>

  <p>Shen Kuo froze. Scrape <em>Cloud-Ford Brocade Guild</em> off the tally, and Mist-Creek House Three couldn’t ship directly to the merchant under the guild seal; leave it on, and Post Bridge One kicked every tally straight back up the hill.</p>

  <p>Lin Qingyan walked to the bridgehead, turned the bamboo tally over in her fingers, and bought a stack of <strong>wax-sealed transit tubes</strong> from the relay station.</p>

  <p>“Don’t touch a stroke on the bamboo tally, and don’t ask the bridge guards to bend imperial law,” she said. She slid the untouched tally — still carved <em>From Yangzhou Pier Four, To Cloud-Ford Brocade Guild</em> — inside a hollow bamboo transit tube, sealed the cap, and brushed an outer relay address across the cylinder: <strong>“From Front Gatehouse — Deliver to Mist-Creek House Three.”</strong></p>

  <p>At all three Imperial Post Bridges, guards read only <em>Mist-Creek House Three</em> on the outer tube and waved the rider straight over the ridge. At Mist-Creek House Three, the clerk cracked open the outer transit tube, tossed the shell into the scrap basket, and pulled out the pristine bamboo tally inside — carrying <em>Yangzhou Pier Four</em>’s true return address and matching the silent brass plaque in the inner room. Before sunset, deep-laden brocade barges glided out of the valley creek straight into the Canglang River, bound for Yangzhou without ever looking back.</p>

  <p>— By now you’ve probably recognised it: this is <strong>Direct Server Return (<em>DSR</em>)</strong> and <strong>Layer-3 Direct Server Return (<em>L3DSR</em>)</strong> in high-throughput Layer-4 load balancing (powering Linux <em>IPVS DR</em> mode, Meta’s <em>Katran</em>, Google’s <em>Maglev</em>, and <em>Cilium DSR</em>). Shen Kuo’s Front Gatehouse is the Layer-4 load balancer; <em>Cloud-Ford Brocade Guild</em> is the public <strong>Virtual IP (<em>VIP</em>)</strong>; the three-inch bamboo tallies and ten-ton barges are tiny request packets and massive response streams; the local courtyard pouch is <strong>Layer-2 MAC address rewriting</strong>; the brass plaque locked inside a silent inner room is binding <code class="language-plaintext highlighter-rouge">VIP/32</code> to <code class="language-plaintext highlighter-rouge">lo</code> (<em>loopback</em>) with <strong>ARP suppression</strong>; and the outer wax-sealed transit tube that crosses Imperial Post Bridges (Layer-3 routers) is <strong><em>L3DSR</em></strong> tunnel encapsulation (<em>IP-in-IP</em>, <em>GUE</em>, or <em>Geneve</em>).</p>

  <h3 id="what-it-is">What it is</h3>

  <p>Most real-world network traffic — video streaming, object storage downloads, CDN asset delivery, large RPC responses — is wildly asymmetric: <strong>inbound client requests are tiny (a few hundred bytes for an <code class="language-plaintext highlighter-rouge">HTTP GET</code>), while outbound server responses are orders of magnitude larger.</strong></p>

  <p>In traditional <strong>symmetric load balancing</strong> (a Layer-7 reverse proxy or Layer-4 <em>SNAT / Full-NAT</em> load balancer), both directions must pass through the load balancer: inbound packets have their destination rewritten to a backend’s <em>Real IP (RIP)</em> and their source rewritten to the load balancer, so outbound megabytes must flow <em>back</em> through the load balancer to have their source IP rewritten back to the public <em>VIP</em>. Add ten times more backend servers and the entire fleet still chokes on the load balancer tier’s egress NIC bandwidth, CPU, and connection-tracking (<em>conntrack</em>) memory.</p>

  <p><strong>Direct Server Return (<em>DSR</em>)</strong> splits the request path from the response path so tiny requests traverse the load balancer while heavy responses travel straight from the backend server to the client:</p>

  <ol>
    <li><strong>Layer-2 DSR (<em>MAC Rewrite</em>)</strong>: When load balancers and backends share a Layer-2 VLAN, the load balancer receives a packet addressed to <code class="language-plaintext highlighter-rouge">VIP</code>, <strong>leaves the Layer-3 IP header completely untouched</strong> (<code class="language-plaintext highlighter-rouge">src = Client IP</code>, <code class="language-plaintext highlighter-rouge">dst = VIP</code>), and rewrites only the destination Ethernet MAC address to the chosen backend’s NIC MAC.</li>
    <li><strong>Loopback <code class="language-plaintext highlighter-rouge">VIP</code> + ARP Suppression</strong>: Why doesn’t the backend kernel drop a packet addressed to <code class="language-plaintext highlighter-rouge">VIP</code> as foreign, and how can it reply with <code class="language-plaintext highlighter-rouge">src IP = VIP</code>? Every backend assigns <code class="language-plaintext highlighter-rouge">VIP/32</code> to its own <code class="language-plaintext highlighter-rouge">lo</code> (<em>loopback</em>) interface. To prevent the <strong>“twelve workshops shouting at once” bug (<em>ARP flux / VIP hijacking</em>)</strong> on the shared Layer-2 network, Linux backends set <code class="language-plaintext highlighter-rouge">net.ipv4.conf.all.arp_ignore=1</code> and <code class="language-plaintext highlighter-rouge">net.ipv4.conf.all.arp_announce=2</code> — silencing physical NICs whenever routers broadcast <code class="language-plaintext highlighter-rouge">ARP Who-Has VIP</code>, so only the load balancer answers publicly while every backend kernel quietly recognizes <code class="language-plaintext highlighter-rouge">VIP</code> internally.</li>
    <li><strong>Layer-3 DSR (<em>L3DSR</em> — Tunnel Encapsulation)</strong>: Modern Spine-Leaf cloud fabrics route Layer-3 IP across racks rather than stretching flat Layer-2 VLANs. The first Layer-3 router strips the Ethernet MAC header, inspects the inner <code class="language-plaintext highlighter-rouge">dst IP = VIP</code>, and routes the packet straight back to the load balancer in a loop. <strong><em>L3DSR</em></strong> (such as <em>IP-in-IP</em> in Meta’s <em>Katran</em>, <em>GUE</em> in Google’s <em>Maglev</em>, or <em>Geneve / TCP-option DSR</em> in <em>Cilium</em>) wraps the untouched client packet inside an outer IP/UDP header addressed to the backend’s <code class="language-plaintext highlighter-rouge">RIP</code>. Routers forward across any number of hops to the backend, which strips the outer header, reads the true <code class="language-plaintext highlighter-rouge">Client IP</code> and <code class="language-plaintext highlighter-rouge">VIP</code>, and transmits heavy response packets (<code class="language-plaintext highlighter-rouge">src = VIP</code>, <code class="language-plaintext highlighter-rouge">dst = Client IP</code>) directly onto the network fabric without returning through the load balancer.</li>
  </ol>

  <h3 id="why-it-matters">Why it matters</h3>

  <p><em>DSR</em> and <em>L3DSR</em> decouple ingress dispatch capacity from egress bandwidth: a small pair of Layer-4 load balancers processing a few gigabits of tiny request packets can orchestrate <strong>hundreds of gigabits or terabits per second</strong> of direct backend egress — while <strong>preserving the client’s true source IP natively at Layer 4</strong> without <code class="language-plaintext highlighter-rouge">X-Forwarded-For</code> headers or <code class="language-plaintext highlighter-rouge">PROXY protocol</code> parsing.</p>

  <p>In production, however, three subtle failure modes catch teams off guard when rolling out <em>L3DSR</em>:</p>

  <ul>
    <li><strong>The outer transit tube makes fat parcels too wide for the bridge (<em>MTU &amp; PMTU black holes</em>)</strong>: Wrapping a client packet in <em>IP-in-IP</em> (+20 bytes) or <em>GUE / Geneve</em> (+28 to 50 bytes) inflates packet size. Tiny <code class="language-plaintext highlighter-rouge">HTTP GET</code> tallies sail through easily, so health checks look green — until a client uploads a full 1500-byte <code class="language-plaintext highlighter-rouge">POST</code> segment, which exceeds a 1500-byte underlay MTU after encapsulation and silently stalls mid-stream. Production <em>L3DSR</em> fabrics require underlay jumbo frames, TCP MSS clamping, or rock-solid ICMP Packet-Too-Big handling.</li>
    <li><strong>Anti-spoofing guards block your own barges (<em><code class="language-plaintext highlighter-rouge">rp_filter</code> &amp; cloud source/dest checks</em>)</strong>: Because packets enter a backend addressed to <code class="language-plaintext highlighter-rouge">RIP</code> (outer header) and leave bare with <code class="language-plaintext highlighter-rouge">src IP = VIP</code>, routing is intentionally asymmetric. Strict Linux reverse-path filtering (<code class="language-plaintext highlighter-rouge">rp_filter=1</code>) or cloud VPC <em>Source/Destination Checks</em> will mistake outbound <code class="language-plaintext highlighter-rouge">src = VIP</code> packets for IP spoofing and drop every <code class="language-plaintext highlighter-rouge">SYN-ACK</code> on the floor unless explicitly configured.</li>
    <li><strong>Bounced mail still has to reach the right workshop (<em>ICMP error consistent hashing</em>)</strong>: When a router along the return path sends <code class="language-plaintext highlighter-rouge">ICMP Fragmentation Needed</code> back toward <code class="language-plaintext highlighter-rouge">VIP</code>, that error lands at the load balancer, not the backend. The load balancer must parse the inner quoted TCP 5-tuple inside the ICMP payload and run the <strong>exact same consistent hash</strong> (e.g., <em>Maglev</em> ring) so the error reaches the specific backend holding that connection.</li>
  </ul>

  <p>The architectural lesson reaches well beyond packet headers: <strong>deciding where work goes does not require carrying the freight back — let the gatekeeper touch only the three-inch tally, and let the heavy barges take the open river.</strong></p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li><strong>Three-inch bamboo order tallies vs. ten-ton cloud-brocade barges</strong> — asymmetric network traffic: tiny inbound request packets (<code class="language-plaintext highlighter-rouge">HTTP GET</code>) vs. massive outbound response streams</li>
    <li><strong>“Cloud-Ford Brocade Guild” title and Shen Kuo’s Front Gatehouse</strong> — the public Virtual IP (<em>VIP</em>) and the Layer-4 load balancer (<em>IPVS</em> / <em>Katran</em> / <em>Cilium</em>)</li>
    <li><strong>Hauling brocade ox-carts back through the ten-foot gatehouse archway</strong> — symmetric reverse proxying / <em>SNAT</em>, forcing all egress bandwidth and connection state back through the load balancer</li>
    <li><strong>Tossing the untouched tally in a local courtyard pouch (“House Seven, Side Door”)</strong> — Layer-2 <em>DSR</em>: rewriting only the destination Ethernet MAC while leaving <code class="language-plaintext highlighter-rouge">src = Client IP</code> and <code class="language-plaintext highlighter-rouge">dst = VIP</code> intact</li>
    <li><strong>All twelve weaving houses shouting back to the boatman at the canal fork</strong> — <em>ARP flux / VIP hijacking</em> when backends answer <code class="language-plaintext highlighter-rouge">ARP Who-Has VIP</code> broadcasts on the shared network</li>
    <li><strong>Locking the guild plaque in a windowless inner room (“Silent Plaque Rule”)</strong> — binding <code class="language-plaintext highlighter-rouge">VIP/32</code> to <code class="language-plaintext highlighter-rouge">lo</code> (<em>loopback</em>) with <code class="language-plaintext highlighter-rouge">arp_ignore=1</code> and <code class="language-plaintext highlighter-rouge">arp_announce=2</code></li>
    <li><strong>Imperial Post Bridge stripping the pouch and routing the tally back to Shen Kuo</strong> — Layer-3 routers stripping Ethernet MAC headers and routing <code class="language-plaintext highlighter-rouge">dst = VIP</code> back to the load balancer in a loop</li>
    <li><strong>The wax-sealed outer transit tube addressed to Mist-Creek House Three</strong> — <em>L3DSR</em> tunnel encapsulation (<em>IP-in-IP</em> / <em>GUE</em> / <em>Geneve</em>) routing to backend <code class="language-plaintext highlighter-rouge">RIP</code> while preserving direct <code class="language-plaintext highlighter-rouge">src = VIP</code> return</li>
  </ul>
</section>]]></content><author><name></name></author><category term="networking" /><category term="linux" /><category term="performance" /><summary type="html"><![CDATA[沧浪江畔的”云津渡锦行”，接的是全天下最轻的帖子，发的是全天下最重的船。]]></summary></entry><entry><title type="html">扔进窑火的印 · The Seal Thrown Into the Kiln</title><link href="https://fables.day/2026/11/16/sigstore-keyless-signing/" rel="alternate" type="text/html" title="扔进窑火的印 · The Seal Thrown Into the Kiln" /><published>2026-11-16T00:00:00+00:00</published><updated>2026-11-16T00:00:00+00:00</updated><id>https://fables.day/2026/11/16/sigstore-keyless-signing</id><content type="html" xml:base="https://fables.day/2026/11/16/sigstore-keyless-signing/"><![CDATA[<section class="zh">
  <p>南陵药局烧出来的青瓷药瓮，封上火漆，要在各州府的常平仓里放上整整十年，等大疫之年才启封救命。</p>

  <p>各州医官不认药工的脸，只认瓮口火漆上那方铁印——那是南陵三号窑主温衡的私章。规矩传了三代：<strong>药瓮要存十年，那方铁印就得管用十年。</strong></p>

  <p>既然铁印十年不能废，温衡就得替它操十年的心。白日里拴在腰间，夜里锁进床底樟木匣，连沐浴都不敢离身。可千日防贼终有一失：第七年冬夜，梁上君子撬开樟木匣，把那方铁印摸走了。</p>

  <p>第二天一早，整座南陵药局陷入了死局。</p>

  <p>若发公文宣告温衡的铁印作废，十二州府仓里存着的四万只真药瓮一夜之间全成了无凭之物——倒掉，大疫来时无药可救；留着，谁也分不清哪只是三年前封的真药，哪只是贼人昨夜拿真印盖出来的砒霜。</p>

  <p>新请来的验封司主事苏绾站在窑口，听众掌柜吵了一上午，只说了一句话：</p>

  <p><strong>“印之所以会被偷，是因为你们非要留着它过夜。”</strong></p>

  <p>她定了一条让所有老师傅瞪圆眼睛的新规矩：<strong>从今日起，谁也不许存印。每封一批药瓮，当场抓一把软泥刻一方新章，往火漆上一按，转身就把泥章扔进白炽的窑火里烧成碎渣——每一方印，只许活一刻钟。</strong></p>

  <p>温衡第一个摇头：”苏主事，我今日捏个梅花纹扔了，明日捏个水波纹又扔了。十年后千里之外的医官开仓，他连这花纹见都没见过，凭什么信这是我温衡封的药，不是街口卖假药的随便捏的泥巴？”</p>

  <p>苏绾指了指窑门外的牌坊：”你每日巳时进三号窑当值，总门不是刚给你发了一块当日当值腰牌吗？你捏好新泥章，先拿当值腰牌到我的验封窗前亮一亮。我核准是你本人当班，就在一张官凭纸上拓下你这枚新泥章的印样，再盖上我验封司的总印，写明——<strong>此临时泥章确系三号窑温衡所持，仅限今日巳时二刻至三刻有效。</strong> 你把官凭贴上瓮口，泥章照旧扔进窑火。”</p>

  <p>满堂掌柜刚要拍手叫好，站在角落里的巡检官陆峥忽然按刀上前，拦住了众人。</p>

  <p>“慢着！”陆峥盯着那只药瓮，一字一句地问，”苏主事，官凭上写的可是’仅限今日巳时二刻至三刻有效’？”</p>

  <p>“不错，只活一刻钟。”</p>

  <p>“那十年之后，外州医官开仓验药时，这张官凭岂不是<strong>已经过期整整十年了</strong>？到时候医官问起来，你必定教他们：’虽然官凭早已过期，但只要验封司总印是真的，照样算数。’既然过期十年也算数——”陆峥目光一凛，”倘若有个心怀鬼胎的学徒，在巳时二刻领了官凭，<strong>假装</strong>把泥章扔进窑火，实则藏进靴筒里带回家呢？三年之后，他半夜摸出这枚旧泥章，盖在一百瓮砒霜上，硬说这是三年前那一刻钟里封的旧货！火漆干了可不会开口说话，谁能戳穿他？再退一步，倘若有人半夜偷了温衡的当值腰牌，悄悄去你窗前领了一张一刻钟官凭，偷偷封十瓮毒药专害一家人，温衡天亮还蒙在鼓里，又当如何？”</p>

  <p>窑房里鸦雀无声。泥印毁没毁，外人看不见；印是哪一年按下去的，火漆自己证不出。</p>

  <p>苏绾却笑了。她推开后窗，指着镇子正中央那座人来人往的钟楼。</p>

  <p>钟楼底下立着一卷十丈长的《公验长编》，各省药商每日傍晚都要抄录副本带回本省，谁也涂改不得半个字。</p>

  <p>“印自己当然证不出时辰，”苏绾缓缓道，”所以规矩还有后半截——<strong>趁你那张官凭还活着的这一刻钟里</strong>，你必须抱着瓮号、印迹和官凭，穿过院子去钟楼底下登簿。守楼人抬头看一眼铜漏：眼下是巳时二刻半，官凭还活着！他当场把你的瓮号与印迹写进长编第八万四千九百一十二行，再给你开一张盖着钟楼铁戳的<strong>入簿时辰回执</strong>，一并封在瓮口。等巳时三刻一过，哪怕你的泥章没烧化、被人藏进了靴筒里，再去钟楼，守楼人连门都不会让他进。”</p>

  <p>陆峥怔住了，在心里默默推演了一遍十年后的场景：</p>

  <p>十年后，外州医官启封一只药瓮，只看三样东西——</p>

  <ol>
    <li>看验封司总印与官凭：<strong>这枚临时印章当年确实发给了三号窑的温衡；</strong></li>
    <li>看官凭效期：<strong>早已过期十年——好得很，说明今天世上没人能再拿它去钟楼登新账；</strong></li>
    <li>看钟楼的入簿时辰回执：<strong>这只药瓮的印迹，确确实实是在十年前那活着的十五分钟窗口里写进《公验长编》的。</strong></li>
  </ol>

  <p>更妙的是，第二年春天，真有个内鬼半夜偷拿了三号窑的腰牌去领临时官凭。可只要登进《公验长编》，天下药商人人可见——天还没亮，三号窑负责盯账的学徒就在长编新页上看见一行”温衡寅时三刻封瓮十只”，当场敲钟追到码头，连人带船截在了闸口。</p>

  <p>从那以后，南陵药局再也没有锁过一只樟木匣。每炉药封完，只听窑火里”啪”的一声轻响，泥印化作青烟，而真凭实据早已刻进了全天下都看得见的石头与长卷里。</p>

  <p>——到这儿你大概已经认出来了：这就是云原生软件供应链里的<strong>无密钥签名</strong>（<em>Keyless Code Signing</em>，以 <em>Sigstore / Cosign</em> 为代表）。温衡腰间那块当日当值腰牌是 CI 流水线里的短期 <em>OIDC ID Token</em>；当场捏完就扔进窑火的泥章是内存里临时生成的<strong>一次性密钥对</strong>（<em>ephemeral keypair</em>）；苏绾开出的”仅活一刻钟”官凭是 <em>Fulcio</em> 签发的短期 X.509 代码签名证书；而钟楼底下那卷公开防篡改的《公验长编》与入簿时辰回执，正是透明日志 <em>Rekor</em> 与它的<strong>签名录入时间戳</strong>（<em>Signed Entry Timestamp, SET</em>）。</p>

  <h3 id="section">这是什么</h3>

  <p>传统代码签名（PGP 或长期私钥文件）有一个根本悖论：<strong>签名必须比软件活得一样久，所以私钥也跟着被迫活那么久。</strong> 你今天构建一个容器镜像，两年后生产集群扩容拉取时还要验签。为了两年后能验过，那把私钥就得在开发者的笔记本、CI 系统的 Secret 变量或密钥库里躺上好几年——成了一块高价值的静态靶子。一旦私钥泄露撤销，要么过去几年的合法镜像集体失效，要么攻击者拿旧钥匙随意伪造签名。</p>

  <p><em>Sigstore</em>（由 Linux 基金会托管，核心组件为 <em>Cosign</em>、<em>Fulcio</em>、<em>Rekor</em>）把”管好长期钥匙”彻底改写成了<strong>“把身份换成十分钟短证书，再用公开透明日志钉死签名时刻”</strong>：</p>

  <ol>
    <li><strong>临时密钥与 OIDC 身份交换（<em>Fulcio</em>）</strong>：构建任务（如 GitHub Actions Pod）在内存里生成一对全新的临时公私钥，私钥从不落盘。接着任务向 OIDC 提供方申请一枚短期身份令牌（证明”我是某仓库 <code class="language-plaintext highlighter-rouge">main</code> 分支某次构建”），连同临时公钥发给证书机构 <em>Fulcio</em>。<em>Fulcio</em> 校验令牌后，签发一张把<strong>临时公钥</strong>与<strong>工作负载身份（SAN 扩展）</strong> 绑在一起的 X.509 证书，有效期通常只有 <strong>10 分钟</strong>。</li>
    <li><strong>签名并写入透明日志（<em>Rekor</em>）</strong>：构建任务用临时私钥对镜像摘要（<em>digest</em>）签名，随后立刻把<strong>签名、公钥证书与制品哈希</strong>提交进基于 Merkle 树的只追加公开透明日志 <em>Rekor</em>，然后<strong>直接在内存里销毁临时私钥</strong>。<em>Rekor</em> 校验证书在当前时刻仍然有效后，返回一条包含可信时间戳的录入回执（<em>SET</em> / 包含性证明 <em>inclusion proof</em>）。</li>
    <li><strong>跨年离线/在线验证</strong>：哪怕两年后那张 10 分钟证书早已过期，验证方（如 Kubernetes 准入控制器）也不看”证书今天有没过期”，而是验证三件事：<strong>证书链归属可信 <em>Fulcio</em> 根且身份匹配预期仓库</strong> + <strong>签名与镜像摘要匹配</strong> + <strong><em>Rekor</em> 的时间戳证明这次签名就发生在该证书当年存活的那 10 分钟窗口之内</strong>。</li>
  </ol>

  <h3 id="section-1">为什么重要</h3>

  <p>因为它第一次让开源社区与企业 CI/CD 在<strong>不保管任何长期私钥</strong>的前提下，拿到了密码学级别的供应链溯源与防篡改保证——这也是为什么 Kubernetes 官方镜像、Python（PyPI）、npm 来源证明（<em>provenance</em>）和 Homebrew 纷纷转向 <em>Sigstore</em>。</p>

  <p>工程上落地这套机制时，故事里陆峥挑出的两个死穴，恰恰是生产配置最容易踩空的坑：</p>

  <ul>
    <li><strong>只验签名、不绑身份，等于没验。</strong> 在传统公私钥里，你手里只有一把自家公钥，验过就代表是自家人；但在 <em>Sigstore</em> 里，全世界所有 GitHub 仓库都在用同一个公共 <em>Fulcio</em> 根证书！如果你运行 <code class="language-plaintext highlighter-rouge">cosign verify</code> 或配置 <em>Kyverno / PolicyController</em> 时不显式锁定 <code class="language-plaintext highlighter-rouge">--certificate-identity</code>（哪条工作流）和 <code class="language-plaintext highlighter-rouge">--certificate-oidc-issuer</code>（哪家身份源），任何攻击者用自己的免费 GitHub 账号跑一遍 Actions，同样能拿到一张 <em>Fulcio</em> 签发的合法证书与 <em>Rekor</em> 记录，大摇大摆通过你的准入检查。</li>
    <li><strong>没有公开监控，透明日志只用了一半。</strong> 如果攻击者窃取了你的 CI 权限或某次构建被注入恶意步骤，他确实能拿到 10 分钟真证书完成签名；但正因为 <em>Fulcio</em> 的证书透明日志与 <em>Rekor</em> 全网公开可查，安全团队只要订阅自己仓库身份的日志监控（如 <code class="language-plaintext highlighter-rouge">rekor-monitor</code>），任何半夜冒出来的非预期签名都会像三号窑那十只寅时药瓮一样无所遁形。</li>
  </ul>

  <p>它揭示了一个极漂亮的密码学设计范式：<strong>最安全的私钥，是签完名就已经不存在的那一把；而让一把”已死之钥”在十年后依然可信的秘诀，是把”我是谁”和”我在哪一刻出过手”拆开证明。</strong></p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li><strong>要在常平仓存放十年的青瓷药瓮</strong> —— 生命周期长达数年的软件制品、容器镜像或二进制包</li>
    <li><strong>锁在床底樟木匣里的十年铁印</strong> —— 传统长期代码签名私钥（PGP / 静态 <code class="language-plaintext highlighter-rouge">cosign.key</code>），泄露与轮换代价极高</li>
    <li><strong>当场捏制、盖完即扔进窑火烧碎的泥章</strong> —— 仅存在于内存中的一次性临时密钥对（<em>ephemeral keypair</em>），签完即毁</li>
    <li><strong>总门发放的当日当值腰牌</strong> —— CI/CD 或工作负载持有的短期 <em>OIDC ID Token</em>（证明仓库、分支与运行身份）</li>
    <li><strong>苏绾的验封司窗口与”仅活一刻钟”官凭</strong> —— <em>Fulcio</em> CA 校验 OIDC 令牌后签发的 10 分钟短期 X.509 签名证书</li>
    <li><strong>把泥章偷藏靴筒、三年后盖砒霜冒充旧货</strong> —— 数字签名自身不包含可信时间，若仅凭过期证书验签会遭事后盗钥回溯伪造</li>
    <li><strong>钟楼底下公开只追加的《公验长编》</strong> —— 基于 Merkle 树的公开防篡改透明日志 <em>Rekor</em></li>
    <li><strong>盖着钟楼铁戳的入簿时辰回执</strong> —— <em>Rekor</em> 返回的签名录入时间戳（<em>SET</em>）与树包含性证明（<em>inclusion proof</em>）</li>
    <li><strong>十年后核验”官凭真伪 + 回执落在当年一刻钟窗口内”</strong> —— 验证证书链、OIDC 身份扩展（SAN）及签名时间处于 <code class="language-plaintext highlighter-rouge">[notBefore, notAfter]</code> 有效期内</li>
    <li><strong>天亮前在长编新页抓出”寅时三刻封瓮十只”</strong> —— 持续监控公开透明日志，第一时间发现身份被盗用签发的异常制品</li>
  </ul>
</section>
<section class="en">
  <p>The celadon medicine jars fired at the Nanling Apothecary Kiln are sealed in red lacquer and stacked inside provincial granaries for ten full years, waiting for a plague year before anyone breaks the clay.</p>

  <p>County physicians hundreds of miles away do not know the potters’ faces. They know only the iron seal pressed into the lacquer on the lid — the private mark of Master Wen Heng of Kiln Three. For three generations the rule never changed: <strong>if a jar must sit ten years in storage, the seal that vouches for it must stay valid for ten years.</strong></p>

  <p>Because the seal could not be retired for ten years, Wen Heng spent ten years losing sleep over it. By day it hung from his belt; by night it sat triple-locked in a camphor box beneath his bed. Yet a thief needs only one night in three thousand. In the seventh winter, someone pried open the camphor box and walked off with the iron seal.</p>

  <p>By dawn the entire Nanling Kiln was trapped.</p>

  <p>Declare Wen Heng’s iron seal void, and forty thousand genuine medicine jars already sitting in granaries across twelve provinces turn untrusted overnight — dump them and people die when the plague comes; keep them and no physician on earth can tell a genuine jar sealed three years ago from a jar of arsenic stamped by the thief last night.</p>

  <p>Su Wan, newly appointed mistress of the Assay Office, stood by the kiln mouth listening to the elders argue until noon. Then she spoke one sentence:</p>

  <p><strong>“A seal can only be stolen because you insist on keeping it overnight.”</strong></p>

  <p>She laid down a rule that made every old master stare: <strong>from today on, nobody keeps a seal. For each batch of jars, grab a lump of wet clay on the spot, carve a fresh stamp, press it into the lacquer once, and hurl the clay stamp straight into the white-hot kiln to shatter into slag. No seal may live longer than one quarter-hour.</strong></p>

  <p>Wen Heng shook his head first. “Mistress Su, if I carve plum blossoms today and smash the stamp, then carve river waves tomorrow and smash that too, a county physician ten years from now has never laid eyes on either pattern. Why should he trust it came from Wen Heng of Kiln Three and not a street peddler?”</p>

  <p>Su Wan pointed to the archway outside. “When you enter Kiln Three for the morning shift, the Guild Gate already hands you today’s shift tally, does it not? Bring your freshly carved clay stamp and your live shift tally to my Assay Window. Once I see you are on shift in person, I take an impression of your new stamp on a slip of official paper, press my Assay Master Crest beside it, and write: <strong>This temporary stamp belongs to Wen Heng of Kiln Three — valid today from the second quarter of the Si hour to the third quarter only.</strong> Paste that voucher onto the jar lid, and throw the clay stamp into the fire as promised.”</p>

  <p>The elders were halfway into applause when Inspector Lu Zheng stepped out from the corner, hand on his sabre, and stopped them.</p>

  <p>“Wait,” Lu Zheng said, eyes fixed on the jar. “Your voucher reads <em>valid today from the second quarter to the third quarter only</em>?”</p>

  <p>“One quarter-hour,” Su Wan said.</p>

  <p>“Then ten years from now, when a provincial physician opens the granary, that voucher will be <strong>ten years expired</strong>. You will have to instruct every physician in the empire: <em>Even though the voucher expired ten years ago, so long as the Assay Crest is genuine, trust the jar.</em> Well, if an expired voucher still counts —” Lu Zheng’s voice dropped, “what if a crooked apprentice claims his voucher at the second quarter, <strong>pretends</strong> to toss his clay stamp into the furnace, slips it into his boot instead, and walks home? Three years later he pulls that old stamp out at midnight, presses it onto a hundred jars of arsenic, and swears they were sealed back during that valid quarter-hour three years ago! Dried lacquer cannot tell you what year it hardened. Worse still — what if a night-thief steals Wen Heng’s shift tally at three in the morning, gets a valid fifteen-minute voucher, seals ten poisoned jars for one chosen victim, and Wen Heng wakes at dawn never knowing his name was used?”</p>

  <p>Silence fell across the kiln floor. Nobody outside can see whether a stamp truly burned, and a seal impression never proves <strong>when</strong> it was pressed.</p>

  <p>Su Wan smiled and pushed open the rear lattice window, pointing to the drum tower in the middle of the market square.</p>

  <p>Beneath the drum tower stood the <em>Grand Public Chronicle</em> — a hundred-foot ledger copied every evening by medicine merchants from every province, where no clerk could scratch out a single line without a hundred copies betraying him.</p>

  <p>“A seal alone never proves the hour,” Su Wan said. “That is why the rule has a second half. <strong>While your fifteen-minute voucher is still alive</strong>, you carry the jar’s batch number, seal impression, and voucher across the square to the Chronicle. The keeper glances up at the bronze water-clock: <em>half-past the second quarter — the voucher still breathes.</em> He writes your batch and seal impression onto Line 84,912 of the public scroll, and hands you a stamped <strong>Chronicle Time-Receipt</strong> to seal onto the lid. Once the third quarter strikes, even if someone hid an unbroken stamp in his boot and crept back three years later, the Chronicle keeper wouldn’t let him through the door.”</p>

  <p>Lu Zheng stood still, tracing what a physician ten years and a thousand miles away would check:</p>

  <ol>
    <li>Check the Assay Crest and voucher: <strong>this temporary stamp really was issued to Wen Heng of Kiln Three.</strong></li>
    <li>Check the voucher’s window: <strong>expired ten years ago — good, which means nobody alive today can register new jars under it.</strong></li>
    <li>Check the Chronicle Time-Receipt: <strong>this jar’s exact impression entered the Public Chronicle at <code class="language-plaintext highlighter-rouge">Si-hour, 2.5 quarters</code> ten years ago — squarely inside the fifteen minutes when the voucher was alive.</strong></li>
  </ol>

  <p>Better yet, the following spring an insider really did steal a Kiln Three shift tally before dawn to claim a temporary voucher. Because every entry went straight onto the public scroll in the open square, Kiln Three’s morning clerk spotted an unexpected line — <em>Wen Heng, Yin-hour third quarter, ten jars</em> — before sunrise, rang the alarm bell, and intercepted the boat at the river lock.</p>

  <p>From that day on, Nanling Kiln never locked another camphor box. Whenever a batch was sealed, a soft <em>crack</em> sounded inside the white-hot kiln as the clay stamp turned to ash — while the proof lived on in open ink for ten years and more.</p>

  <p>— By now you’ve probably recognised it: this is <strong>keyless code signing</strong> in cloud-native software supply chains, pioneered by <strong>Sigstore</strong> (<em>Cosign</em>, <em>Fulcio</em>, and <em>Rekor</em>). Wen Heng’s daily shift tally from the gate is a short-lived CI <em>OIDC ID Token</em>; the wet-clay stamp hurled into the kiln right after pressing is an in-memory <strong>ephemeral keypair</strong> whose private key is never written to disk; Su Wan’s fifteen-minute voucher is a short-lived X.509 code-signing certificate from <em>Fulcio</em>; and the open, append-only <em>Grand Public Chronicle</em> with its time-receipt is the <em>Rekor</em> transparency log and its <strong>Signed Entry Timestamp (<em>SET</em>)</strong>.</p>

  <h3 id="what-it-is">What it is</h3>

  <p>Traditional code signing (PGP keys or long-lived <code class="language-plaintext highlighter-rouge">.key</code> files) suffers from a structural paradox: <strong>a signature must remain verifiable for the entire multi-year lifetime of an artifact, which forces the signing key to stay valid for years too.</strong> Build a container image today and a production cluster scaling up two years from now still needs to verify it. So that long-lived private key sits on a developer laptop, in a CI secret store, or inside an HSM wrapper year after year — a high-value static target. When it leaks and gets revoked, either years of legitimate releases break at once or attackers forge signatures indistinguishable from old ones.</p>

  <p><em>Sigstore</em> (hosted by the Linux Foundation around <em>Cosign</em>, <em>Fulcio</em>, and <em>Rekor</em>) replaces long-lived secret custody with <strong>short-lived workload identity plus an immutable timestamped log</strong>:</p>

  <ol>
    <li><strong>Ephemeral keys + OIDC identity exchange (<em>Fulcio</em>)</strong>: A build job (such as a GitHub Actions runner or Kubernetes workload) generates a fresh keypair purely in memory. It requests a short-lived <em>OIDC ID Token</em> from its identity provider (proving <em>“I am workflow <code class="language-plaintext highlighter-rouge">release.yml</code> on <code class="language-plaintext highlighter-rouge">main</code> at commit SHA <code class="language-plaintext highlighter-rouge">...</code>“</em>), and sends that token plus the ephemeral public key to the <em>Fulcio</em> certificate authority. <em>Fulcio</em> verifies the token and issues an X.509 certificate binding the <strong>ephemeral public key</strong> to the <strong>OIDC workload identity</strong> (in X.509 SAN extensions), valid for just <strong>10 minutes</strong>.</li>
    <li><strong>Sign, log, and destroy (<em>Rekor</em>)</strong>: The job signs the artifact digest with the ephemeral private key, immediately submits the <strong>signature, certificate, and artifact hash</strong> to <em>Rekor</em> — a public, append-only Merkle transparency log — and <strong>discards the private key from memory</strong>. <em>Rekor</em> verifies that the certificate is currently valid and returns a <strong>Signed Entry Timestamp (<em>SET</em>)</strong> and Merkle inclusion proof.</li>
    <li><strong>Verification years later</strong>: When a verifier (such as a Kubernetes admission controller) checks the image years after that 10-minute certificate expired, it does not ask whether the certificate is unexpired <em>today</em>. It verifies three facts together: <strong>the certificate chains to <em>Fulcio</em> and carries the expected OIDC identity + issuer</strong>, <strong>the signature matches the artifact digest</strong>, and <strong><em>Rekor</em>’s timestamp proves the signature was recorded inside that certificate’s 10-minute <code class="language-plaintext highlighter-rouge">[notBefore, notAfter]</code> window</strong>.</li>
  </ol>

  <h3 id="why-it-matters">Why it matters</h3>

  <p>Keyless signing gave open-source ecosystems and enterprise CI/CD cryptographic provenance <strong>without anyone ever storing, rotating, or leaking a signing private key</strong> — which is why official Kubernetes container releases, PyPI, npm provenance, and Homebrew adopted <em>Sigstore</em>.</p>

  <p>In production, the two loopholes Inspector Lu Zheng raised map directly onto the two sharpest operational failure modes:</p>

  <ul>
    <li><strong>Verifying a Sigstore signature without pinning identity verifies almost nothing.</strong> With a classic static keypair, trusting your own public key implicitly pins who signed. With public <em>Sigstore</em>, <strong>every repository on GitHub shares the same public <em>Fulcio</em> root CA</strong>. If you run <code class="language-plaintext highlighter-rouge">cosign verify</code> or configure <em>Kyverno / PolicyController</em> without explicitly pinning both <code class="language-plaintext highlighter-rouge">--certificate-identity</code> (which repo/workflow) and <code class="language-plaintext highlighter-rouge">--certificate-oidc-issuer</code> (which OIDC provider), any attacker can sign a malicious image from their own free GitHub repo, get a completely genuine <em>Fulcio</em> cert and <em>Rekor</em> entry, and sail straight past admission control.</li>
    <li><strong>Transparency only catches stealth compromise if someone watches the log.</strong> If an attacker briefly hijacks a CI workflow, they can obtain a real 10-minute certificate during that run. Because both <em>Fulcio</em>’s certificate log and <em>Rekor</em> are globally auditable, running a log monitor (<code class="language-plaintext highlighter-rouge">rekor-monitor</code>) watching for your identity turns silent supply-chain signing into a loud, public event that gets caught like those ten pre-dawn jars at the lock.</li>
  </ul>

  <p>The underlying cryptographic move is worth remembering far beyond containers: <strong>the safest private key is one that already ceased to exist five seconds after signing — and the way you keep a dead key trustworthy for a decade is to separate proving <em>who held it</em> from proving <em>when it struck</em>.</strong></p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li><strong>Celadon medicine jars stored ten years in provincial granaries</strong> — long-lived software artifacts, container images, and release binaries</li>
    <li><strong>Wen Heng’s ten-year iron seal locked in a camphor box</strong> — a traditional long-lived code-signing private key (<code class="language-plaintext highlighter-rouge">cosign.key</code> / PGP), vulnerable to theft and painful revocation</li>
    <li><strong>The wet-clay stamp pressed once and hurled into the white-hot kiln</strong> — an in-memory <em>ephemeral keypair</em> whose private key is never persisted and wiped immediately after signing</li>
    <li><strong>Today’s Guild Gate shift tally</strong> — a short-lived workload <em>OIDC ID Token</em> attesting repository, workflow, and build identity</li>
    <li><strong>Su Wan’s Assay Window voucher (“valid for one quarter-hour only”)</strong> — a 10-minute <em>Fulcio</em> X.509 certificate binding the ephemeral public key to the OIDC identity</li>
    <li><strong>Hiding the clay stamp in a boot to stamp arsenic three years later</strong> — why raw digital signatures do not prove <em>time</em>, making expired short-lived certs unsafe without trusted timestamping</li>
    <li><strong>The open, append-only <em>Grand Public Chronicle</em> beneath the drum tower</strong> — <em>Rekor</em>, the Merkle-tree-backed public transparency log</li>
    <li><strong>The stamped Chronicle Time-Receipt on the jar lid</strong> — <em>Rekor</em>’s <em>Signed Entry Timestamp (SET)</em> and inclusion proof</li>
    <li><strong>Checking Assay Crest + identity + Time-Receipt inside the 15-minute window ten years later</strong> — <code class="language-plaintext highlighter-rouge">cosign verify</code> checking the <em>Fulcio</em> trust chain, OIDC SAN identity/issuer, and <em>Rekor</em> timestamp within <code class="language-plaintext highlighter-rouge">[notBefore, notAfter]</code></li>
    <li><strong>Spotting “Yin-hour third quarter, ten jars” on the public scroll before dawn</strong> — transparency log monitoring (<code class="language-plaintext highlighter-rouge">rekor-monitor</code>) detecting unauthorized signatures issued under hijacked CI credentials</li>
  </ul>
</section>]]></content><author><name></name></author><category term="security" /><category term="ci-cd" /><category term="kubernetes" /><summary type="html"><![CDATA[南陵药局烧出来的青瓷药瓮，封上火漆，要在各州府的常平仓里放上整整十年，等大疫之年才启封救命。]]></summary></entry><entry><title type="html">他说他从未读过 · The Clerk Who Never Read It</title><link href="https://fables.day/2026/11/15/speculative-execution-side-channel/" rel="alternate" type="text/html" title="他说他从未读过 · The Clerk Who Never Read It" /><published>2026-11-15T00:00:00+00:00</published><updated>2026-11-15T00:00:00+00:00</updated><id>https://fables.day/2026/11/15/speculative-execution-side-channel</id><content type="html" xml:base="https://fables.day/2026/11/15/speculative-execution-side-channel/"><![CDATA[<section class="zh">
  <p>云隐阁凿在青崖半腰，藏书七万卷，向以”快”闻名。</p>

  <p>阁里有一条传了不知多少代的老规矩：<strong>不许等人开口。</strong></p>

  <p>掌门还没说要哪一卷，掌阁的陆老先生已经把人打发下去了。他站在门口，听脚步、看灯影、数风向，赌掌门下一步要开哪一扇门。赌对了，掌门话音未落，卷子已摊在案上，快得像本来就在那里；赌错了，就把人喊回来，卷子放回原处，案上什么也不留。</p>

  <p>阁里还有第二条规矩，比第一条更要紧：<strong>没开口的事，算没发生过。</strong> 拿错了卷子不算过失，账册上一个字都不记——那些卷子根本没”到过”案上。</p>

  <p>多少年来，云隐阁的名声是快，从没人说过乱。</p>

  <p>直到那年秋天，对家派来一个抄书人。</p>

  <p>他叫玄七，持着正经荐书，每日在阁里抄书，抄得极慢。他不问机密，不碰禁区，只是午后日头正好时，提着一支烛在架子之间慢慢走，走到哪一格，便把烛凑近，看焰子歪不歪。</p>

  <p>他做的事只有三步。</p>

  <p>第一步，把陆老先生的赌法喂熟。连着三日，他求掌门每日午后必问”玄字第七卷”。老先生果然记住了这个时辰。第四日午后，又有一封盖着假印的密札，被人混进了掌门案上的文书堆——那密札第一行只写一个字。</p>

  <p>第二步，他在密室到书架之间的廊下，摆了两百五十六个一模一样的空木匣，一个挨一个，匣底压着不同的青砖。</p>

  <p>第三步，他挑了个最老实的小厮，把一枚假符塞进他手里，说：等会儿老先生叫你，你照他的话走。</p>

  <p>午后，掌门果然开口：”取玄字第七卷。”</p>

  <p>陆老先生没有犹豫——这个时辰，这个语气，他听过三遍了。他一挥手，小厮便朝最深的那扇封条门跑去。</p>

  <p>那扇门后的规矩是：<strong>外面的架上有的，可以取；封条门后的，一步不许进。</strong> 这是阁里唯一的铁律，掌门亲口立的。</p>

  <p>可老先生喊得太快。小厮的脚步已经迈出去了，门上的封条在他眼里还没来得及变成一个”止”字。他跑了进去，一眼看见那封密札，看见了第一行那个字。</p>

  <p>那是个”柒”字。</p>

  <p>小厮是识数的。他脚步没停，转身就出来，径直走到廊下那两百五十六个空木匣前面，在第七个匣子跟前站了半瞬，又走回门口。</p>

  <p>这时陆老先生的第二声喊才追上来：”错了！回来！”</p>

  <p>小厮回来了。卷子放回原处。密札重新封好。账册上一个字都没有——他没取过任何东西，他只是跑错了一趟。</p>

  <p>事情到这里本该过去。</p>

  <p>可玄七还提着那支烛在廊下走。</p>

  <p>他一个匣子一个匣子地凑过去，看焰子。二百五十五个匣子前的焰子纹丝不动；走到第七个匣子跟前，焰子忽然歪了一下——那里的青砖，还是温的。</p>

  <p>那半瞬的站立，把那个”柒”字，原原本本地留在了砖上。</p>

  <p>——到这儿你大概已经认出来了：陆老先生那”不许等人开口”的抢跑，是 <em>speculative execution</em>；他凭时辰和语气赌的那一扇门，是 <em>branch prediction</em>；小厮穿过封条门的那一步，是<strong>乱序执行越过了边界检查</strong>；账册上的空白，是错误推测被<strong>回滚</strong>后的 <em>architectural state</em>；而那块还温着的青砖，是<strong>没有被回滚的缓存状态</strong>。玄七提着烛挨个量焰子，就是 <em>Flush+Reload</em> 计时；他三天里喂熟的那个时辰，是 <em>branch training</em>。</p>

  <p>这一整套，叫<strong>推测执行的侧信道</strong>，最出名的一支是 <em>Spectre</em>。</p>

  <h3 id="section">这是什么</h3>

  <p>现代 CPU 不肯让流水线空着。遇到一个分支，它不等你算出条件，先<strong>猜</strong>——用历史记录（<em>branch predictor</em>）猜你往哪走，然后把猜的那条路上的指令提前执行掉，这叫 <em>speculative execution</em>。猜对了，白赚一大截时间；猜错了，就把结果全部丢弃，寄存器、内存、标志位一概恢复原状。对软件来说，错误推测过的那条路<strong>从未发生</strong>——这被称为 <em>architectural state</em> 的精确回滚。</p>

  <p>问题在于：回滚只发生在<strong>架构状态</strong>这一层。微架构里的东西——缓存里被填进来的那一行、<em>TLB</em> 里新换的表项、<em>branch predictor</em> 自己更新过的历史——<strong>没有回滚机制，也从来没人打算做</strong>。它们只是”状态”。于是错误推测执行过的那条路，会在缓存里留下一道浅浅的印子。</p>

  <p><em>Spectre</em> 就是拿这道印子当信道。攻击者先<strong>训练</strong>分支预测器（反复让它走同一条路，喂出一个稳定的猜测），再让受害代码带着一个<strong>越界的下标</strong>走到那条路上：<code class="language-plaintext highlighter-rouge">if (x &lt; len) y = arr2[arr1[x] * 4096];</code>。<code class="language-plaintext highlighter-rouge">x &lt; len</code> 本该是护栏，但预测器已经笃定它成立，于是 <code class="language-plaintext highlighter-rouge">arr1[x]</code> 被<strong>推测性地</strong>读出来——它是一个受害者本来无权访问的字节——然后这个字节被用来算地址，把 <code class="language-plaintext highlighter-rouge">arr2</code> 里对应那一条缓存行<strong>拉进缓存</strong>。整段很快被判定为推测错误，全部撤销，程序继续若无其事地跑。</p>

  <p>可缓存里那一条，还在。</p>

  <p>攻击者只需再来一轮，把 <code class="language-plaintext highlighter-rouge">arr2</code> 的每一条缓存行都清掉，然后逐个计时访问：快的那个，就是被秘密字节拉进来的那一条。一次一个字节，重复几千次，一个密钥就漏完了。整个过程<strong>没有任何越权访问记录，没有异常，没有日志</strong>——因为架构上它真的什么都没做。</p>

  <h3 id="section-1">为什么重要</h3>

  <p>因为它把一句本来很结实的话打穿了：<strong>“没有日志，就是没发生。”</strong></p>

  <p>三十年来，隔离建立在”代码不可能执行到那一步”之上——边界检查、权限位、<em>page table</em> 的 <em>user/supervisor</em> 位、<em>KVM</em> 的地址空间、容器的 <em>namespace</em>。而 <em>Spectre</em> 攻击的恰恰是<strong>执行到那一步之前的猜测</strong>：真正的指令从未提交，权限从未被违反，操作系统看不见任何异常，可数据已经借着缓存的可观测性溜了出去。你的隔离边界还在，只是有人在边界外测了测温度。</p>

  <p>这也解释了它在云上为什么格外刺眼。多租户的物理机上，同一颗核的超线程（<em>SMT</em>）与共享的 L3 缓存，就是那块”公共廊道”——同一台机器上另一个租户的缓存时序，原则上就是可观测的。这就是为什么当年的补丁链条那么难看：</p>

  <ul>
    <li><strong>微码 / 固件补丁</strong>（<em>microcode</em>）改的是推测器本身，比如 <em>IBRS</em>/<em>STIBP</em> 之类的间接分支预测限制；</li>
    <li>内核里的 <em>retpoline</em> 把间接跳转换成一段”捕获并返回”的循环，让推测走进一条死路；</li>
    <li><strong>关掉超线程</strong>（<em>disable SMT</em>）——最有效，也最贵，吞吐直接掉一截；</li>
    <li><strong>站点隔离 / 进程隔离</strong>、给敏感代码加 <em>lfence</em> / <em>speculation barrier</em>，以及把代码写成<strong>常数时间</strong>（<em>constant-time</em>，无论秘密是什么都走一样的路，比如在密码学库里用无分支选择替代 <code class="language-plaintext highlighter-rouge">if</code>）。</li>
  </ul>

  <p>代价是实打实的：这些补丁普遍带来几位数百分比的吞吐损失，而它们防的不是”漏洞”这种可以补完的东西，是<strong>现代 CPU 为了快而做的一个根本取舍</strong>。这一类攻击（<em>Spectre</em> 家族）被反复重新发明，因为每一代新微架构都在同一处偷速度。</p>

  <p>工程上真正能落地的三条，故事里都给了：</p>

  <ul>
    <li><strong>在封条门前喊停</strong>——在信任边界的两侧插入序列化点（<em>lfence</em>、屏障、<em>constant-time</em> 比较）。位置要对：喊在门后毫无意义。</li>
    <li><strong>各院各烛</strong>——把真正敏感的东西和不可信的输入放进不同的进程、不同的核、不同的机器。租户隔离别只靠 <em>namespace</em>。</li>
    <li><strong>让人把每个匣子都走一遍</strong>——敏感代码里不许有依赖秘密的分支和内存下标。这条最难，也最根本。</li>
  </ul>

  <p>最后一条冷知识：你自己的代码通常不需要写 <code class="language-plaintext highlighter-rouge">lfence</code>。真正会中招的是<strong>编译器</strong>、<strong>JIT</strong> 和<strong>密码学库</strong>——它们会替你生成边界检查与间接跳转。所以修这件事的顺序是：先补微码和内核，看好你的运行时，最后才是自己的代码。</p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li><strong>陆老先生”不许等人开口”的抢跑</strong> —— <em>speculative execution</em></li>
    <li><strong>凭时辰、语气赌那扇门</strong> —— <em>branch prediction</em></li>
    <li><strong>三天里喂熟的”午后必问玄字第七卷”</strong> —— <em>branch training</em>（训练预测器）</li>
    <li><strong>封条门”外面有就可以取，门后一步不许进”</strong> —— 边界检查 / 权限检查（<code class="language-plaintext highlighter-rouge">if (x &lt; len)</code>）</li>
    <li><strong>喊得太快，脚步先迈了出去</strong> —— 推测执行越过未完成的条件判断</li>
    <li><strong>小厮看见密札第一行的字</strong> —— 推测性地读出一个越权字节</li>
    <li><strong>他走到第七个空匣子前站了半瞬</strong> —— 用秘密字节计算地址，把对应缓存行拉入缓存</li>
    <li><strong>二百五十六个一模一样的匣子</strong> —— 256 个候选值对应 256 条缓存行</li>
    <li><strong>“错了！回来！”，卷子放回，账册无字</strong> —— 错误推测的精确回滚（<em>architectural state</em> 无痕）</li>
    <li><strong>青砖还是温的</strong> —— 微架构状态（缓存）不回滚</li>
    <li><strong>玄七提烛挨个量焰子</strong> —— <em>Flush+Reload</em> 计时侧信道</li>
    <li><strong>“他从未读过”却字字皆知</strong> —— 隔离边界仍然完好，数据却借可观测性外泄</li>
    <li><strong>同一廊道、同一盏灯</strong> —— 共享 L3 缓存 / 同一颗核的 <em>SMT</em> 线程</li>
    <li><strong>在封条门前喊一声”停”</strong> —— 序列化 / <em>lfence</em> / <em>speculation barrier</em></li>
    <li><strong>让推测走进死路（原地默念再返回）</strong> —— <em>retpoline</em></li>
    <li><strong>各家分独立小院、各点各的烛</strong> —— 站点隔离 / 进程与核隔离 / 关掉 <em>SMT</em></li>
    <li><strong>无论取哪一卷都把每个匣子走一遍</strong> —— <em>constant-time</em>、无分支代码</li>
    <li><strong>补丁带来的吞吐下降</strong> —— 安全与性能的硬取舍，不是可以补完的 bug</li>
  </ul>
</section>
<section class="en">
  <p>The Cloud-Hidden Pavilion is cut into the flank of a green cliff. Seventy thousand scrolls, and a reputation built entirely on one word: fast.</p>

  <p>It has an old rule, older than anyone can trace: <strong>never wait for the master to speak.</strong></p>

  <p>Before the master has named a single scroll, old Lu, the keeper of the archive, has already sent a boy running. He stands at the doorway reading footsteps, lamp shadows, the turn of the wind, and guesses which door the master will open next. Guess right, and the scroll is laid on the desk before the sentence ends — so quick it seems it was always there. Guess wrong, and the boy is called back, the scroll returned, the desk left bare.</p>

  <p>And there is a second rule, more important than the first: <strong>what was never spoken never happened.</strong> A wrong scroll is not a fault. Not one character goes into the ledger — those scrolls never “reached” the desk at all.</p>

  <p>For generations the Pavilion’s name meant fast. Nobody ever said careless.</p>

  <p>Then one autumn, a rival house sent a copyist.</p>

  <p>His name was Xuan Qi. He carried proper letters of introduction and copied slowly, every day. He asked nothing confidential and touched nothing forbidden. He simply walked the aisles in the good afternoon light with a candle, and wherever he stopped, he held the flame close and watched whether it leaned.</p>

  <p>He did three things.</p>

  <p>First, he taught old Lu’s guessing habit. For three days running he begged the master to ask, every afternoon at that hour, for “the seventh scroll of the Xuan set.” The old man learned the hour. On the fourth afternoon, a sealed letter bearing a forged stamp was slipped in among the papers on the master’s desk — and the first line of that letter held a single character.</p>

  <p>Second, in the corridor between the sealed chamber and the shelves, he set out two hundred and fifty-six identical empty wooden boxes, side by side, each resting on a different blue brick.</p>

  <p>Third, he picked the most honest boy in the place, pressed a counterfeit tally into his hand, and said: when the old man calls you, do exactly as he says.</p>

  <p>That afternoon the master spoke: “Fetch the seventh scroll of the Xuan set.”</p>

  <p>Old Lu did not hesitate. That hour, that tone — he had heard it three times. He waved, and the boy ran for the deepest door in the archive.</p>

  <p>The rule behind that door was simple: <strong>what is on the outer shelves may be taken; what lies behind the seal may not be entered, not one step.</strong> It was the only iron law in the Pavilion, and the master himself had set it.</p>

  <p>But the old man had shouted too fast. The boy’s foot was already in the air, and the seal on the door had not yet become the character for <em>stop</em> in his eyes. He went in. He saw the sealed letter. He saw the first character of its first line.</p>

  <p>It was the character for <em>seven</em>.</p>

  <p>The boy could count. He did not break stride. He turned, came back out, walked straight to the two hundred and fifty-six empty boxes in the corridor, stood for half a breath in front of the seventh — and went back to the doorway.</p>

  <p>Only then did old Lu’s second shout catch up with him: “Wrong! Come back!”</p>

  <p>The boy came back. The scroll was returned to its place. The letter was resealed. Not one character went into the ledger — he had taken nothing. He had merely run the wrong way.</p>

  <p>That should have been the end of it.</p>

  <p>But Xuan Qi was still walking the corridor with his candle.</p>

  <p>He went box by box, holding the flame close. In front of two hundred and fifty-five of them the flame stood perfectly still. In front of the seventh, it leaned — the blue brick there was still warm.</p>

  <p>Half a breath of standing had left that character <em>seven</em> on the stone, whole and legible.</p>

  <p>— By now you’ve probably recognised it: old Lu’s rush to run before the order was spoken is <em>speculative execution</em>; the door he guessed from hour and tone is <em>branch prediction</em>; the boy’s step through the sealed door is <strong>out-of-order execution passing a bounds check</strong>; the blank in the ledger is <em>architectural state</em> after a <strong>rolled-back</strong> misprediction; and the brick still warm is <strong>cache state that is never rolled back</strong>. Xuan Qi measuring flame after flame with a candle is <em>Flush+Reload</em> timing; the hour he spent three days teaching is <em>branch training</em>.</p>

  <p>The whole arrangement is a <strong>speculative-execution side channel</strong>, and its most famous member is <em>Spectre</em>.</p>

  <h3 id="what-it-is">What it is</h3>

  <p>A modern CPU refuses to let its pipeline sit idle. Faced with a branch, it does not wait for the condition to resolve — it <strong>guesses</strong>, using history (<em>the branch predictor</em>), and executes down the guessed path ahead of time. That is <em>speculative execution</em>. Guess right and you win a large chunk of time for free. Guess wrong and everything is discarded: registers, memory, flags all restored. To the software, the mispredicted path <strong>never happened</strong> — this is the precise rollback of <em>architectural state</em>.</p>

  <p>The trouble is that the rollback stops at the architectural layer. Things in the microarchitecture — the line just pulled into cache, the <em>TLB</em> entry just installed, the predictor’s own updated history — <strong>have no rollback mechanism, and were never designed to have one.</strong> They are simply state. So the wrongly speculated path leaves a faint mark in the cache.</p>

  <p><em>Spectre</em> uses that mark as a channel. The attacker first <strong>trains</strong> the predictor — repeatedly taking one path until its guess is stable — then leads the victim’s code down that path with an <strong>out-of-range index</strong>: <code class="language-plaintext highlighter-rouge">if (x &lt; len) y = arr2[arr1[x] * 4096];</code>. The check <code class="language-plaintext highlighter-rouge">x &lt; len</code> is the guard rail, but the predictor is now certain it holds, so <code class="language-plaintext highlighter-rouge">arr1[x]</code> is read <strong>speculatively</strong> — a byte the victim has no right to touch — and that byte is then used to compute an address, pulling the corresponding line of <code class="language-plaintext highlighter-rouge">arr2</code> <strong>into cache</strong>. Moments later the whole thing is judged a misprediction and undone, and the program carries on as if nothing occurred.</p>

  <p>Except that one cache line is still there.</p>

  <p>The attacker needs one more round: evict every line of <code class="language-plaintext highlighter-rouge">arr2</code>, then time access to each. The fast one is the line the secret byte pulled in. One byte at a time, a few thousand repetitions, and a key is gone. And there is <strong>no record of any unauthorised access, no exception, no log entry</strong> — because architecturally nothing happened.</p>

  <h3 id="why-it-matters">Why it matters</h3>

  <p>Because it breaks a sentence that used to be solid: <strong>“no log means it never happened.”</strong></p>

  <p>For thirty years, isolation has rested on “the code cannot get to that step” — bounds checks, permission bits, the <em>user/supervisor</em> bit in the <em>page table</em>, <em>KVM</em>’s address spaces, container <em>namespaces</em>. <em>Spectre</em> attacks the <strong>guess that happens before that step</strong>: the real instruction never retires, no permission is ever violated, the operating system sees no anomaly — yet the data has already leaked through the observability of the cache. Your isolation boundary is intact. Someone just took its temperature from outside.</p>

  <p>Which is why it is especially sharp in the cloud. On a multi-tenant physical host, the hyperthreads of a single core (<em>SMT</em>) and the shared L3 cache are precisely that “public corridor” — another tenant’s cache timing on the same machine is, in principle, observable. It’s why the patch chain looked so ugly:</p>

  <ul>
    <li><strong>Microcode and firmware</strong> changes target the speculation machinery itself, such as <em>IBRS</em>/<em>STIBP</em> restrictions on indirect-branch prediction;</li>
    <li><em>retpoline</em> in the kernel turns indirect jumps into a “capture and return” loop so speculation walks into a dead end;</li>
    <li><strong>Disabling hyperthreading</strong> (<em>SMT</em>) is the most effective measure and the most expensive — throughput drops outright;</li>
    <li><strong>Site isolation / process isolation</strong>, <em>lfence</em> or other <em>speculation barriers</em> around sensitive code, and writing code <strong>constant-time</strong> — where every path looks identical no matter the secret, as crypto libraries do when they replace <code class="language-plaintext highlighter-rouge">if</code> with a branchless select.</li>
  </ul>

  <p>The cost is real: these mitigations routinely take a double-digit-percent bite out of throughput, and what they defend against is not a “vulnerability” that can be patched away, but <strong>a fundamental trade-off modern CPUs make in order to be fast</strong>. The <em>Spectre</em> family keeps being reinvented, because every new microarchitecture cuts the same corner.</p>

  <p>Three things that actually land in engineering practice, all of them in the story:</p>

  <ul>
    <li><strong>Shout stop at the sealed door</strong> — insert a serialisation point (<em>lfence</em>, a barrier, a <em>constant-time</em> comparison) on both sides of the trust boundary. Placement matters: shouting behind the door is useless.</li>
    <li><strong>A separate courtyard and its own candle per house</strong> — put genuinely sensitive material and untrusted input in different processes, different cores, different machines. Don’t rest tenant isolation on <em>namespaces</em> alone.</li>
    <li><strong>Make the boy walk every box</strong> — no secret-dependent branches and no secret-dependent memory indices in sensitive code. Hardest, and most fundamental.</li>
  </ul>

  <p>One last counter-intuitive note: your own code usually does not need an <code class="language-plaintext highlighter-rouge">lfence</code>. The things that get hit are <strong>compilers</strong>, <strong>JITs</strong>, and <strong>crypto libraries</strong> — they generate the bounds checks and indirect jumps for you. So the order of repair is: microcode and kernel first, then know your runtime, and your own code last.</p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li><strong>Old Lu’s rush to run before the order is spoken</strong> — <em>speculative execution</em></li>
    <li><strong>Guessing the door from hour and tone</strong> — <em>branch prediction</em></li>
    <li><strong>The three days spent teaching “the Xuan set, every afternoon”</strong> — <em>branch training</em> (teaching the predictor)</li>
    <li><strong>“What is outside may be taken; behind the seal, not one step”</strong> — a bounds check / permission check (<code class="language-plaintext highlighter-rouge">if (x &lt; len)</code>)</li>
    <li><strong>Shouting too late, the foot already in the air</strong> — speculative execution crossing an unresolved condition</li>
    <li><strong>The boy seeing the first character of the sealed letter</strong> — speculatively reading an out-of-bounds byte</li>
    <li><strong>Standing half a breath before the seventh empty box</strong> — computing an address from the secret byte, pulling that cache line in</li>
    <li><strong>Two hundred and fifty-six identical boxes</strong> — 256 candidate values mapping to 256 cache lines</li>
    <li><strong>“Wrong! Come back!”, scroll returned, ledger blank</strong> — precise rollback of architectural state on a misprediction</li>
    <li><strong>The blue brick still warm</strong> — microarchitectural state (cache) is never rolled back</li>
    <li><strong>Xuan Qi measuring flame after flame by candle</strong> — the <em>Flush+Reload</em> timing side channel</li>
    <li><strong>“He never read it”, yet every character known</strong> — the isolation boundary stays intact while data leaks through observability</li>
    <li><strong>The same corridor, the same lamp</strong> — shared L3 cache / <em>SMT</em> threads on one core</li>
    <li><strong>Shouting “stop” at the sealed door</strong> — serialisation / <em>lfence</em> / <em>speculation barrier</em></li>
    <li><strong>Making speculation walk into a dead end (chant in place, then return)</strong> — <em>retpoline</em></li>
    <li><strong>A courtyard and its own candle for each house</strong> — site isolation / process and core isolation / disabling <em>SMT</em></li>
    <li><strong>Walking every box no matter which scroll is asked for</strong> — <em>constant-time</em>, branchless code</li>
    <li><strong>The throughput lost to the patches</strong> — a hard security-versus-speed trade-off, not a bug you can finish fixing</li>
  </ul>
</section>]]></content><author><name></name></author><category term="security" /><category term="performance" /><category term="runtime" /><summary type="html"><![CDATA[云隐阁凿在青崖半腰，藏书七万卷，向以”快”闻名。]]></summary></entry><entry><title type="html">九千九百九十七骑 · Nine Thousand Nine Hundred and Ninety-Seven Riders</title><link href="https://fables.day/2026/11/14/hierarchical-timing-wheels/" rel="alternate" type="text/html" title="九千九百九十七骑 · Nine Thousand Nine Hundred and Ninety-Seven Riders" /><published>2026-11-14T00:00:00+00:00</published><updated>2026-11-14T00:00:00+00:00</updated><id>https://fables.day/2026/11/14/hierarchical-timing-wheels</id><content type="html" xml:base="https://fables.day/2026/11/14/hierarchical-timing-wheels/"><![CDATA[<section class="zh">
  <p>朔方关外的风雪一年刮八个月，斥候堂的灯一夜也不许灭。</p>

  <p>堂里有一条铁规矩：凡出关的斥候，临上马前必留一支铁箭，报清归期——近的翻一道梁，四十滴铜漏便回；远的入瀚海探营，要走三天三夜。到点不见人回来，堂上就要敲响救命钟，派轻骑出关接应。</p>

  <p>起初几十年，管箭架的老吏用的一直是最稳妥的笨法子：<strong>按归期先后，把铁箭从左到右排成一线。</strong> 最左边永远是最早到点的那一支，守堂人只要盯着排头看，一分神都不会出差错。</p>

  <p>后来边事吃紧，每日出关的斥候从百十骑涨到了上万骑。二十个书办围着三丈长的箭架，从早到晚汗流浃背：每出一骑，要顺着几千支箭比对时刻，把新箭塞进正中间，后面的箭全往右挪；更要命的是每归来一骑，又要从几千支箭里把那一支抽出来，再把空当合拢。关门口排起了长队，斥候在风雪里冻得直跺脚，不是等马，是在等书办排箭。</p>

  <p>新调来的掌堂官裴疏站在廊下看了一整天，傍晚只问了校尉沈烈一句话：”昨日出关一万骑，救命钟响了几回？”</p>

  <p>沈烈愣了愣：”三回。”</p>

  <p>“其余九千九百九十七骑呢？”</p>

  <p>“都在到点前平安入关，自己把箭抽走了。”</p>

  <p>裴疏点点头：”你们二十个人从早忙到晚，手心磨出血泡，是在给九千九百九十七声<strong>根本不会响的钟</strong>排队。”</p>

  <p>第二天，裴疏让人撤了长架，抬进一只圆木盘。木盘沿口凿了六十个深槽，铜漏每滴一滴水，盘心的铜针就往前拨一格。每支铁箭尾上系一根朱砂红绒线，线头直接挂在门口各自的鞍钩上。</p>

  <p>规矩改成了三句话：</p>

  <ul>
    <li>出关报十七滴后归？从铜针眼下往前数十七格，把箭直接丢进槽里——<strong>槽里哪怕已有几十支箭，也不许比先后</strong>，随手放进去就行。</li>
    <li>第九滴人就平安回来了？进门一拽自己鞍钩上的红绒线，铁箭直接从槽里提出来，左右邻箭纹丝不动。</li>
    <li>铜针拨到哪一格，哪一格里还剩着没被抽走的箭，一并敲钟发兵。</li>
  </ul>

  <p>一夜之间，十九个书办回去歇了。只留一个小厮守在盘边，一只手就管下了全关的近哨。</p>

  <p>真正的难关在入秋之后。</p>

  <p>大军开拔前，去瀚海深处的远哨多了起来——动辄两个时辰、一天、三天，折算成铜漏，是几十万滴水。木匠算了算，若按一滴一格凿一只三十万格的大木盘，把整座斥候堂拆了也放不下。</p>

  <p>沈烈想了个折中的主意：盘还是那只六十格的盘，只在远哨的铁箭杆上用白粉写一行字——<strong>“转满三千圈再响”</strong>。</p>

  <p>不到两天，那个小厮就累趴在了盘边。因为铜漏<strong>每一滴水落下</strong>，铜针拨进下一格，小厮都得把槽里八百支远哨铁箭抱出来，一支一支擦掉”三千”，改写成”两千九百九十九”，再塞回去。水滴不等他擦完粉笔字就落下了下一滴，铜针卡在半途，连翻山梁的近哨都误了钟点。</p>

  <p>裴疏走进堂里，拿袖口擦净一支箭杆上的白粉，说了一句点醒全堂的话：</p>

  <p><strong>“三天后才该响的钟，今天干吗要跟眼皮底下的近哨挤在一个盘里数水滴？离得远的事，本来就用不着细尺子量。”</strong></p>

  <p>她让木匠在同一根立轴上套了大、中、小三重木盘：</p>

  <ul>
    <li><strong>内盘（滴盘）</strong> 六十格，铜漏一滴拨一格，转一圈是一刻；</li>
    <li><strong>中盘（刻盘）</strong> 六十格，内盘转满一圈，中盘才拨一格；</li>
    <li><strong>外盘（辰盘）</strong> 七十二格，中盘转满一圈，外盘才拨一格，足足装得下三天三夜。</li>
  </ul>

  <p>一骑远哨报”五个时辰零二十三刻十四滴后归”，他的铁箭<strong>根本不进内盘</strong>，只丢进外盘往前数第五格。接下来的五个时辰里，内盘飞快地转上几百圈、铜漏滴下一万八千滴水，<strong>连碰都不会碰这支箭一下</strong>。倘若他在第三个时辰平安归关——十有八九都是如此——进门一拽红绒线，箭直接从外盘提走，自始至终没花过半点搬运功夫。</p>

  <p>只有当五个时辰真到了、外盘的铜针咔哒一声拨进第五格，而那支箭<strong>居然还在槽里</strong>时，小厮才把这一格里仅剩的几支箭捧出来，看一眼剩下的零头，<strong>往里降一盘</strong>，放进中盘的第二十三格；二十三刻后还没人抽走，再降进内盘的第十四格；十四滴水落尽，铜钟轰然敲响。</p>

  <p>那年腊月大雪封山，夜里全关只派出了四骑探路，最早的一骑也要五个时辰后才到点。半夜裴疏巡堂，却见小厮冻得眼圈通红，还守在铜漏边，”咔哒、咔哒”地一格一格空拨内盘——一宿要空拨一万八千下。</p>

  <p>裴疏笑着递给他一块巴掌大的小木牌：”傻孩子，别记箭，<strong>只记哪几个槽里有箭</strong>。就算一万骑齐出，三只盘加起来总共也才一百九十二个槽，牌子上最多不过一百九十二行字。”</p>

  <p>小厮低头一看：今夜整块牌子上只有四行字，排头那一行写着”外盘第五格”。他吹灭油灯，裹上羊皮袄睡了整整五个时辰，等铜漏积满五壶水，起身直接把指针拨到外盘第五格——一分不差。</p>

  <p>——到这儿你大概已经认出来了：这就是<strong>分层时间轮</strong>（<em>Hierarchical Timing Wheel</em>）。三丈长的排序箭架是按到期时间维护的最小堆（<em>min-heap</em>）；六十格木盘配红绒线是单层时间轮上的 <em>O(1)</em> 插入与双向链表 <em>O(1)</em> 撤单；在箭杆上写”转满三千圈”是带圈数计数的哈希时间轮（<em>hashed wheel with rounds</em>）；内、中、外三重木盘的逐级下放是<strong>级联降盘</strong>（<em>cascade</em>）；而腊月夜里那块只记”非空槽位”的掌心小牌，正是 <em>Kafka Purgatory</em> 用 <em>DelayQueue</em> 驱动时间轮的巧思。</p>

  <h3 id="section">这是什么</h3>

  <p>George Varghese 与 Tony Lauck 在 1987 年 SOSP 论文《Hashed and Hierarchical Timing Wheels》里提出时间轮，最初就是为了解决网络协议栈里的定时器瓶颈。在 TCP 重传、RPC 超时、心跳检测这类场景里，一台机器往往同时挂着几十万到上百万个定时器，但它们有一个极偏斜的共同特征：<strong>99.9% 的定时器在到期前就会因为收到 ACK 或响应而被提前取消</strong>，真正到点触发的不到千分之一。</p>

  <p>如果用传统的二叉最小堆（<em>min-heap</em>）或红黑树管定时器，每次新建连接插入是 $O(\log N)$，每次收到正常数据包取消定时器也是 $O(\log N)$。当 $N = 10^6$、每秒收发几十万包时，CPU 大把的缓存行和指令周期都浪费在<strong>给永远不会触发的超时重排堆序</strong>上。</p>

  <p>时间轮利用了超时管理的一个天然性质：<strong>超时的精度是有限的</strong>（比如 1 毫秒一跳，没人需要纳秒级精度的 30 秒超时）。于是它把时间切成固定步长（<em>tick</em>）的环形数组，每个槽位挂一条双向链表：</p>

  <ul>
    <li><strong>插入 $O(1)$</strong>：到期还剩 $\Delta t$ 个 tick，直接算 <code class="language-plaintext highlighter-rouge">(current + Δt) mod M</code>，插进对应槽位的链表头，槽内完全不排序；</li>
    <li><strong>取消 $O(1)$</strong>：请求对象自己持有链表节点指针（那根”红绒线”），收到响应时直接修改前后指针摘除节点，和总定时器数量 $N$ 完全无关；</li>
    <li><strong>分层级联（<em>Hierarchical Cascade</em>）</strong>：为了同时覆盖”1 毫秒精度”与”几天跨度”而不建几亿个空槽，用多层不同粒度的轮盘叠起来（如 20×1ms、20×20ms、20×400ms……）。远期任务只放在高层粗粒度轮里，内层快轮怎么转都不碰它；只有高层指针走到那一格时，才把槽里幸存的定时器取出，按剩余时间重插（<em>cascade</em>）进下一层细轮。每个定时器最多被级联搬运 $L$ 次（轮的层数，通常 3～4 层），均摊仍是 $O(1)$。</li>
  </ul>

  <h3 id="section-1">为什么重要</h3>

  <p>因为几乎所有高并发基础设施的底层都在转这只轮子，而且各自演化出了两记极其精彩的工程变招：</p>

  <p>第一记变招在 <strong>Apache Kafka 的 <em>DelayedOperationPurgatory</em></strong>（以及 Netty 的 <code class="language-plaintext highlighter-rouge">HashedWheelTimer</code> 对比）。Netty 用的是单层 <code class="language-plaintext highlighter-rouge">rounds</code> 圈数法，槽位冲突多时每跳仍要遍历扣减圈数；而 Kafka 面对成千上万个 <code class="language-plaintext highlighter-rouge">acks=all</code> 生产请求和长轮询拉取请求时，采用了标准<strong>分层时间轮 + <code class="language-plaintext highlighter-rouge">DelayQueue</code></strong>：如果像故事里的小厮那样每 1 毫秒由后台线程空拨一格，空闲集群也会白烧 CPU。Kafka 的做法是<strong>不把百万个定时任务放进最小堆，而是只把”非空槽位（Bucket）”放进 <code class="language-plaintext highlighter-rouge">DelayQueue</code>（最小堆）</strong>。无论挂了多少百万个请求，三四层轮盘加起来总共只有几十到一两百个槽位！堆的大小被死死锁在常数级，工作线程平时直接 <code class="language-plaintext highlighter-rouge">poll()</code> 阻塞休眠到下一个非空槽位到期，一举兼得了<strong>百万级任务 $O(1)$ 插删</strong>与<strong>零空转休眠</strong>。</p>

  <p>第二记变招在 <strong>Linux 内核定时器（<code class="language-plaintext highlighter-rouge">timer_list</code>）</strong>。Linux 从 1997 年起用了近二十年的经典级联时间轮（<code class="language-plaintext highlighter-rouge">tvec_base</code>），直到 2016 年（Linux 4.8，Thomas Gleixner 重构）发现了一个微妙的尾延迟毛刺：当高层轮盘每隔几秒咔哒拨动一格时，如果那一格恰好攒了几万个还没取消的定时器，瞬间集中级联重插会锁住轮盘，给软中断制造一次毫秒级抖动。内核团队看穿了本质——网络重传和套接字超时根本不在乎”30 秒整”还是”31.5 秒”触发，而高精度需求早有独立的红黑树 <code class="language-plaintext highlighter-rouge">hrtimer</code> 负责。于是新版内核干脆<strong>取消了级联搬运（<em>no cascade</em>）</strong>：高层轮用几何级数放宽刻度，换取约 6%～12.5% 的到期容差（<em>slack</em>），让远期定时器<strong>直接在高层槽位原地触发，永远不下放</strong>。</p>

  <p>这就是系统设计里最迷人的一课：<strong>动手选数据结构之前，先问哪件事发生一万次、哪件事只发生三次。</strong> 为千分之三的触发去拖累千分之九百九十七的撤单，再精巧的排序也是白费力气。</p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li><strong>出关留铁箭、到点敲救命钟</strong> —— 超时定时器（TCP 重传、RPC 超时、心跳检测）</li>
    <li><strong>一万骑里九千九百九十七骑提前平安归关</strong> —— 绝大多数定时器在到期前被正常 ACK/响应提前取消</li>
    <li><strong>三丈长按先后排队的箭架</strong> —— 最小堆（<em>min-heap</em>）或有序队列，插入与取消均需 $O(\log N)$ 重排</li>
    <li><strong>圆木盘上的六十个深槽与铜漏水滴</strong> —— 时间轮的环形槽位数组（<em>buckets</em>）与固定时间步长（<em>tick</em>）</li>
    <li><strong>箭入槽不比先后，随手丢入</strong> —— 双向链表头插法，$O(1)$ 插入</li>
    <li><strong>系在门口鞍钩上的朱砂红绒线</strong> —— 任务对象直接持有的链表节点引用，支持 $O(1)$ 原地摘除取消</li>
    <li><strong>在箭杆上写”转满三千圈再响”</strong> —— 单层带 <code class="language-plaintext highlighter-rouge">rounds</code> 计数的哈希时间轮，长跨度下每跳需遍历未到期节点</li>
    <li><strong>“离得远的事用不着细尺子量”</strong> —— 远期超时无需在当下维护细粒度精度，按剩余时间分层存放</li>
    <li><strong>同轴套叠的内盘（滴）、中盘（刻）、外盘（辰）</strong> —— 多级不同刻度（<code class="language-plaintext highlighter-rouge">tickMs</code> 与 <code class="language-plaintext highlighter-rouge">wheelSize</code>）的分层时间轮</li>
    <li><strong>外盘到点后把剩余箭捧出、放入中盘</strong> —— 级联降盘（<em>cascade / reinsert</em>），将高层到期桶拆解下放至低层细轮</li>
    <li><strong>大雪夜空拨一万八千下指针</strong> —— 空槽轮询（<em>empty-tick spinning</em>）导致的空闲 CPU 唤醒浪费</li>
    <li><strong>只记”哪几个槽里有箭”的掌心小牌</strong> —— Kafka 以 <code class="language-plaintext highlighter-rouge">DelayQueue</code>（最小堆）仅管理非空槽位（桶数量恒定），兼顾 $O(1)$ 增删与阻塞休眠</li>
  </ul>
</section>
<section class="en">
  <p>Outside Shuofang Pass the snow blows eight months of the year, and the lamps in the Scout Hall never go out.</p>

  <p>The hall keeps one iron rule: every scout who rides through the gate leaves an iron arrow behind and names a return mark — forty drips of the bronze water-clock to check the near ridge, two hours down the gorge, three days and nights into the salt desert. If the mark arrives and the rider has not, the hall strikes the rescue bell and sends fast cavalry out the gate.</p>

  <p>For decades the old clerks kept the arrows by the safest, plainest method anyone knew: <strong>line every arrow up along a thirty-foot cedar rack in strict order of who is due back first.</strong> The leftmost arrow is always the next deadline; watch the head of the rack and you can never miss an alarm.</p>

  <p>Then the northern frontier grew tense, and daily patrols swelled from a hundred riders to ten thousand. Twenty clerks stood drenched in sweat from dawn till midnight around that cedar rack. Every departing scout meant walking thousands of arrows to compare deadlines, wedging the new arrow into the middle, and shifting everything to its right; worse, every returning scout meant hunting down one arrow mid-rack, pulling it free, and closing the gap. Riders stood stamping their boots in the snow outside the gate — waiting not for fresh horses, but for clerks to finish sorting arrows.</p>

  <p>Pei Shu, newly appointed keeper of the hall, watched from the gallery for one full day. At dusk she asked Captain Shen Lie a single question: “Ten thousand riders went out yesterday. How many times did the rescue bell ring?”</p>

  <p>Shen Lie blinked. “Three.”</p>

  <p>“And the other nine thousand nine hundred and ninety-seven?”</p>

  <p>“Came home before their mark and pulled their arrows.”</p>

  <p>Pei Shu nodded. “Twenty men with blisters bleeding on their palms from dawn till dark — all to keep a neat queue of nine thousand nine hundred and ninety-seven bells <strong>that will never ring</strong>.”</p>

  <p>Next morning she had the long rack carried out and a round cedar wheel brought in. Sixty deep slots were cut around its rim. Each drip of the water-clock nudged a bronze hand forward by one slot. And tied to the tail of every iron arrow was a cord of vermillion silk, its loop hung directly on that rider’s saddle-hook beside the gate.</p>

  <p>Three short rules replaced the old ones:</p>

  <ul>
    <li>Riding out for seventeen drips? Count seventeen slots ahead of the bronze hand and drop the arrow straight in — <strong>even if thirty arrows are already lying in that slot, do not sort them</strong>. Just let it fall.</li>
    <li>Back safe on the ninth drip? Reach up by the gate and tug your vermillion cord. Your arrow lifts clean out of its slot without disturbing a single neighbour.</li>
    <li>Wherever the bronze hand lands, any arrow still lying in that slot is genuinely overdue — strike the bell for all of them at once.</li>
  </ul>

  <p>Overnight, nineteen clerks went home to sleep. One boy tended the entire near-pass patrol with one hand.</p>

  <p>The real wall arrived in autumn.</p>

  <p>Ahead of the main army’s march, long-range scouts began riding deep into the desert — two hours out, a full day, three days and nights. Measured in water-drips, three days was hundreds of thousands of beats. The carpenter did the sums: a single wheel with three hundred thousand slots would not fit inside the hall if you tore the walls down.</p>

  <p>Shen Lie proposed a clever compromise: keep the sixty-slot wheel, and chalk a line onto each desert arrow — <strong>“ring after three thousand laps.”</strong></p>

  <p>Within two days the boy collapsed beside the wheel. On <strong>every single drip</strong> of the water-clock, as the bronze hand clicked into the next slot, he had to haul out eight hundred desert arrows, rub out <em>3,000</em>, chalk <em>2,999</em>, and cram them back in before the next drop fell. The water did not wait for his chalk; the bronze hand jammed halfway round, and even short-range ridge scouts began missing their rescue bells.</p>

  <p>Pei Shu walked in, wiped the chalk off an arrow shaft with her sleeve, and said the sentence that untangled the hall:</p>

  <p><strong>“Why is a bell due three days from now crowding into the same ring to count water-drips beside a scout on the near ridge? What is far off does not yet need a fine ruler.”</strong></p>

  <p>She had the carpenter mount three concentric wooden wheels on the same central spindle:</p>

  <ul>
    <li><strong>The Inner Wheel (Drips)</strong> — sixty slots; one water-drip advances one slot, one full turn is a quarter-hour.</li>
    <li><strong>The Middle Wheel (Quarters)</strong> — sixty slots; one full turn of the Inner Wheel advances the Middle Wheel by one slot.</li>
    <li><strong>The Outer Wheel (Hours)</strong> — seventy-two slots; one full turn of the Middle Wheel advances the Outer Wheel by one slot, spanning three full days.</li>
  </ul>

  <p>When a desert scout named “five hours, twenty-three quarters, and fourteen drips,” his arrow <strong>never touched the Inner Wheel</strong>. It dropped straight into slot five of the Outer Wheel. For the next five hours the Inner Wheel spun hundreds of times — eighteen thousand drips of water — <strong>without touching that arrow once</strong>. And when, as happened nine times out of ten, the scout rode home safe in the third hour, one tug on his vermillion cord lifted the arrow straight out of the Outer Wheel: zero laps counted, zero sorting paid.</p>

  <p>Only when five full hours elapsed, the Outer Wheel’s hand clicked into slot five, and an arrow <strong>was still lying there</strong>, did the boy lift out the handful of surviving arrows in that slot, read what remained on each, and <strong>drop them down one wheel</strong> into slot twenty-three of the Middle Wheel; twenty-three quarters later, if still unclaimed, down into slot fourteen of the Inner Wheel; fourteen drips later — the bronze bell tolled.</p>

  <p>In the twelfth month a blizzard sealed the pass. All night only four scouts rode out, the earliest due back in five hours. Walking the hall at midnight, Pei Shu found the boy red-eyed and shivering by the brazier, still nudging the inner bronze hand <em>click, click, click</em> across empty slot after empty slot — eighteen thousand empty clicks before dawn.</p>

  <p>She handed him a wooden slate no bigger than her palm. “Don’t list the arrows,” she told him. <strong>“List only which slots aren’t empty.</strong> Even if ten thousand riders go out together, all three wheels hold only a hundred and ninety-two slots in total. Your slate will never carry more than a hundred and ninety-two lines.”</p>

  <p>The boy looked down. Tonight the whole slate held four lines, and the top line read <em>Outer Wheel, slot five</em>. He blew out the lamp, pulled his sheepskin coat over his shoulders, slept five straight hours, and woke when five measures of water filled the basin — stepping the pointer straight to Outer Wheel slot five, not a breath late.</p>

  <p>— By now you’ve probably recognised it: this is the <strong>Hierarchical Timing Wheel</strong>. The thirty-foot sorted arrow rack is a deadline-ordered <em>min-heap</em>; the sixty-slot wheel with vermillion cords is a single timing wheel giving $O(1)$ insertion and $O(1)$ cancellation via doubly-linked list nodes; chalking “three thousand laps” on an arrow is a <em>hashed timing wheel with rounds</em>; the concentric inner, middle, and outer rings dropping survivors inward is <strong>hierarchical cascading</strong>; and that palm-sized winter slate tracking only non-empty slots is the exact trick <em>Apache Kafka’s Purgatory</em> uses when it drives a hierarchical timing wheel with a <code class="language-plaintext highlighter-rouge">DelayQueue</code>.</p>

  <h3 id="what-it-is">What it is</h3>

  <p>George Varghese and Tony Lauck introduced timing wheels in their 1987 SOSP paper <em>“Hashed and Hierarchical Timing Wheels”</em> to break a bottleneck in network protocol stacks. In TCP retransmission, RPC deadlines, and heartbeat monitoring, a single host routinely holds hundreds of thousands or millions of active timers — yet those timers share a wildly skewed property: <strong>99.9% of them are cancelled early when an ACK or reply arrives</strong>, and fewer than one in a thousand ever expire.</p>

  <p>Manage those timers in a classic binary <em>min-heap</em> or red-black tree and every arm costs $O(\log N)$ while every early cancellation also costs $O(\log N)$. At $N = 10^6$ timers and hundreds of thousands of packets per second, CPU cache lines and cycles vanish into <strong>re-heapifying deadlines that will never fire</strong>.</p>

  <p>A timing wheel exploits one natural fact about timeouts: <strong>timeout resolution is bounded</strong> (say 1 ms per tick — nobody needs nanosecond precision on a 30-second socket timeout). Divide time into a circular array of fixed-width <em>ticks</em>, each bucket holding an unsorted doubly-linked list:</p>

  <ul>
    <li><strong>$O(1)$ insert</strong>: a timer due in $\Delta t$ ticks lands directly in bucket <code class="language-plaintext highlighter-rouge">(current + Δt) mod M</code>, prepended to the list with zero sorting.</li>
    <li><strong>$O(1)$ cancel</strong>: the request object holds a direct pointer to its list node (the vermillion cord); when the ACK arrives, unlinking two pointers removes it in constant time regardless of $N$.</li>
    <li><strong>Hierarchical cascade</strong>: to span both 1 ms precision and multi-day horizons without allocating hundreds of millions of empty buckets, stack multiple wheels of coarser granularity (e.g. <code class="language-plaintext highlighter-rouge">20 × 1ms</code>, <code class="language-plaintext highlighter-rouge">20 × 20ms</code>, <code class="language-plaintext highlighter-rouge">20 × 400ms</code>…). Distant timers sit untouched in outer wheels while the inner wheel spins; only when an outer hand steps onto a bucket are its surviving timers extracted and reinserted (<em>cascaded</em>) into the next finer wheel. Each timer cascades at most $L$ times (typically 3–4 levels), keeping amortized cost $O(1)$.</li>
  </ul>

  <h3 id="why-it-matters">Why it matters</h3>

  <p>Nearly every high-concurrency runtime turns this wheel under the hood, and two production evolutions show how deep the engineering goes:</p>

  <p>The first is <strong>Apache Kafka’s <em>DelayedOperationPurgatory</em></strong> (contrasted with Netty’s <code class="language-plaintext highlighter-rouge">HashedWheelTimer</code>). Netty uses a single wheel with <code class="language-plaintext highlighter-rouge">rounds</code> counters, which still walks colliding long-horizon timers on ticks; Kafka, managing hundreds of thousands of <code class="language-plaintext highlighter-rouge">acks=all</code> produce requests and long-polling fetch requests, pairs a <strong>Hierarchical Timing Wheel with a <code class="language-plaintext highlighter-rouge">DelayQueue</code></strong>. Ticking a background thread every 1 ms across empty buckets wastes CPU when a broker is quiet. Instead of putting <em>millions of individual timers</em> into a min-heap, Kafka puts <strong>only non-empty buckets</strong> into a <code class="language-plaintext highlighter-rouge">DelayQueue</code>. Even under a million pending requests, three or four wheels contain only a few dozen to a couple hundred buckets in total! The heap stays tiny and bounded by bucket count, worker threads block cleanly inside <code class="language-plaintext highlighter-rouge">poll()</code> until the next occupied bucket expires, and individual timers still arm and cancel in $O(1)$.</p>

  <p>The second is the <strong>Linux kernel timer wheel (<code class="language-plaintext highlighter-rouge">timer_list</code>)</strong>. Linux ran the classic cascading hierarchical wheel (<code class="language-plaintext highlighter-rouge">tvec_base</code>) from 1997 until 2016, when Thomas Gleixner’s Linux 4.8 redesign addressed a subtle tail-latency spike: whenever an outer wheel clicked forward every few seconds, cascading thousands of uncancelled timers all at once locked the wheel and injected millisecond-scale softirq jitter. Kernel engineers recognised a deeper domain truth — general timeouts (retransmits, socket keepalives) protect against failure and never care whether “30 seconds” fires at <code class="language-plaintext highlighter-rouge">30.0s</code> or <code class="language-plaintext highlighter-rouge">31.8s</code>, whereas true microsecond deadlines already belong in the separate red-black-tree <code class="language-plaintext highlighter-rouge">hrtimer</code> subsystem. So modern Linux <strong>eliminated cascading altogether</strong>: outer levels widen geometrically and accept ~6%–12.5% <em>timer slack</em>, firing expired timers directly from their outer bucket without ever moving them inward.</p>

  <p>It is one of systems engineering’s cleanest lessons: <strong>before choosing a data structure, ask which operation happens ten thousand times and which happens three.</strong> Paying $O(\log N)$ on nine thousand nine hundred and ninety-seven cancellations just to keep three expirations neatly sorted is solving the wrong half of the problem.</p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li><strong>Leaving an iron arrow at the pass and striking the rescue bell if overdue</strong> — arming a timeout timer (TCP retransmit, RPC deadline, heartbeat)</li>
    <li><strong>9,997 of 10,000 riders returning safe and pulling their arrows early</strong> — ~99.9% of production timers cancelled by normal ACKs/replies before expiry</li>
    <li><strong>The thirty-foot cedar rack sorted strictly by deadline</strong> — a <em>min-heap</em> or balanced tree paying $O(\log N)$ re-ordering on every insert and cancel</li>
    <li><strong>The sixty-slot wooden wheel and bronze water-clock drip</strong> — circular bucket array and fixed <em>tick</em> duration</li>
    <li><strong>Dropping an arrow into a slot without sorting against neighbours</strong> — unsorted doubly-linked bucket insertion in $O(1)$</li>
    <li><strong>The vermillion silk cord hung on each rider’s gate hook</strong> — direct list-node pointer held by the request for $O(1)$ unlink on cancellation</li>
    <li><strong>Chalking “ring after 3,000 laps” on long-range arrows</strong> — a single <em>hashed wheel with rounds</em>, forced to scan unexpired timers on every lap</li>
    <li><strong>“What is far off does not yet need a fine ruler”</strong> — coarse-to-fine time resolution across wheel levels</li>
    <li><strong>Concentric Inner (drip), Middle (quarter), and Outer (hour) wheels</strong> — multi-level <em>Hierarchical Timing Wheel</em> (<code class="language-plaintext highlighter-rouge">tickMs</code> scaling by <code class="language-plaintext highlighter-rouge">wheelSize</code> per level)</li>
    <li><strong>Lifting surviving arrows from an Outer slot and dropping them into the Middle wheel</strong> — <em>cascading</em> (demoting) timers inward as their horizon approaches</li>
    <li><strong>Clicking the hand eighteen thousand times across empty slots on a snowy night</strong> — idle CPU burn from empty-tick polling</li>
    <li><strong>The palm-sized slate listing only non-empty slots</strong> — Kafka Purgatory’s <code class="language-plaintext highlighter-rouge">DelayQueue</code> over <em>buckets</em> (bounded heap size + zero idle spinning + $O(1)$ per-timer ops)</li>
  </ul>
</section>]]></content><author><name></name></author><category term="distributed-systems" /><category term="linux" /><category term="performance" /><summary type="html"><![CDATA[朔方关外的风雪一年刮八个月，斥候堂的灯一夜也不许灭。]]></summary></entry><entry><title type="html">守潮人的簿子 · The Tide Keeper’s Ledger</title><link href="https://fables.day/2026/11/13/gorilla-tsdb-compression/" rel="alternate" type="text/html" title="守潮人的簿子 · The Tide Keeper’s Ledger" /><published>2026-11-13T00:00:00+00:00</published><updated>2026-11-13T00:00:00+00:00</updated><id>https://fables.day/2026/11/13/gorilla-tsdb-compression</id><content type="html" xml:base="https://fables.day/2026/11/13/gorilla-tsdb-compression/"><![CDATA[<section class="zh">
  <p>望潮塔立在海崖最东头，塔顶那口铜钟是给渔船报潮的。</p>

  <p>守塔的人叫阿潮。每隔十分钟，他要爬上塔顶，看一眼石壁上的水尺，把时辰和水位一并记进簿子里。水位要记到小数点后六位——这是京城海务司定的规矩，因为潮汐表就是照着这些簿子算出来的，差一分，渔船的锚就下错了地方。</p>

  <p>阿潮记得一丝不苟。一年下来，簿子堆满了整间屋子。</p>

  <p>麻烦在每年冬末。所有簿子要装船，绕过半个海疆送进京城。船就那么大，而簿子年年增厚。有一年装了三条船；再往后，海务司的司徒大人来信说：明年只能派一条船。信末还添了一句冰冷的话——<strong>一个字都不许省</strong>。</p>

  <p>阿潮试过省。他试过把六位小数砍成三位，收到回信挨了骂；他试过只记”大概水位”，海务司说那份潮汐表让三条渔船搁浅了。他要的不是”少记一点”，而是<strong>把同样的意思写得更短</strong>。</p>

  <p>于是他坐在灯下，从头翻自己的簿子。翻到第七本，他先看出了一件小事。</p>

  <p>每一行的时辰，从早到晚，写的都是同一个间隔：十分钟、十分钟、十分钟。他这本簿子里”十分钟”这三个字，一年写了一万遍。他把它改成<strong>只记间隔</strong>——第一行写”十分钟”，后面不必再写。</p>

  <p>再翻几本，他看出了第二件小事：<strong>间隔自己也很少变。</strong> 只有当他在塔顶耽搁了、或者换班的人晚到了一会儿，间隔才会变一点。于是他不再写间隔，只写<strong>间隔变了多少</strong>——而它几乎永远是零。</p>

  <p>他在簿子上添了新的记法：</p>

  <ul>
    <li>一个点 <code class="language-plaintext highlighter-rouge">·</code> —— 间隔一点没变（绝大多数情况，一个符号）</li>
    <li>一画 <code class="language-plaintext highlighter-rouge">—</code> 加个小数 —— 间隔变了一点点</li>
    <li>两画 <code class="language-plaintext highlighter-rouge">——</code> 加个中数 —— 变得多些</li>
    <li>三画 <code class="language-plaintext highlighter-rouge">———</code> 加个大数 —— 变了半刻钟以上</li>
    <li>四画 <code class="language-plaintext highlighter-rouge">————</code> —— 干脆把完整的间隔重新报一遍</li>
  </ul>

  <p>一万行时辰，压成了大半页点。</p>

  <p>接着他盯上了水位那一列。这一列才占簿子的九成。他把相邻两次水位都写成二进制，一行一行对齐着看，忽然发现：<strong>上下两行，左边的开头一模一样，右边的末尾也常常一模一样</strong>，真正不同的只是中间那一小截。</p>

  <p>理由其实很朴素：海水不会上一刻一米二、下一刻三百米。它总是在同一个量级里慢慢挪，所以前面那些位长期不动；而水尺刻到六位小数，末几位又常常是零。两行一比，中间那截短的只有几位。</p>

  <p>于是他改了水位的写法：先记<strong>左边从第几位起开始相同</strong>、<strong>右边到第几位为止相同</strong>，再把中间那截不同的一小截抄下来。一行水位，从六十多个符号缩成了十来个。风平浪静的夜里，前后两次水位一模一样，他连中间那截都不用写——只落一个点：<code class="language-plaintext highlighter-rouge">·</code>，跟上次一样。</p>

  <p>半年下来，他省掉了九成。</p>

  <p>真正的坎在九月。</p>

  <p>一场台风过境，海水在两个时辰里涨了两丈。水位从一米三跳到三丈一——量级变了，左边相同的位数从四十多位缩成了几位，右边也全乱了。他之前想了个偷懒的办法：把那两个”相同到第几位”的数字写在簿子第一页，之后一直沿用。台风一来，这个写法要么记不下真实的水位（那就等于偷偷省字，犯了司徒大人的忌讳），要么他得把幅度留得极宽，宽到平时每一个点都要多写十几个符号。</p>

  <p>他坐在灯下，看着被风刮歪的窗，明白了一件事：<strong>“两端相同到哪儿”，并不该是整本簿子的规矩，而该是每一行自己的决定。</strong></p>

  <p>他重新定了格式。每一行水位开头，先落一个小记号：</p>

  <ul>
    <li><code class="language-plaintext highlighter-rouge">··</code> —— 两个点：这一行沿用上一行的那对”相同到哪里”的数，只抄中间那截（平日的绝大多数）</li>
    <li><code class="language-plaintext highlighter-rouge">·|</code> —— 一个点一竖：这一行我自己重新报那对数，然后再抄中间那截（台风、换季、量级跳变时才用）</li>
  </ul>

  <p>一个字也没有多花。台风那几页，簿子确实厚了一些——但只厚了那几页。</p>

  <p>那年冬天，阿潮把十年的簿子装进一只布袋，交给了船。司徒大人拆开看了三天，把每一行都还原了出来，一分不差。他问阿潮用了什么法术。</p>

  <p>阿潮说：我什么都没省，我只是<strong>不再重复写那些从来没变过的东西</strong>。</p>

  <p>——到这儿你大概已经认出来了：这就是 <em>Gorilla 压缩</em>。阿潮只写”间隔变了多少”，是 <em>delta-of-delta</em> 时间戳编码；他把相邻两次水位异或、只抄中间那一小截，是 <em>XOR</em> 浮点编码；那个”沿用上一行的窗口还是重新报一个窗口”的两点记号，就是 <em>Gorilla</em> 里的两位控制位。</p>

  <h3 id="section">这是什么</h3>

  <p><em>Gorilla</em> 是 Facebook 在 2015 年提出的内存时序数据库压缩方案（VLDB 论文《Gorilla: A Fast, Scalable, In-Memory Time Series Database》）。它要解决的是监控数据的存储问题：一条指标每 10 秒一个点，一年 300 多万个点，每个点若按”8 字节时间戳 + 8 字节 float64”存，就是 16 字节、约 50MB。而指标数据绝大多数<strong>可预测</strong>——采样间隔固定，数值缓慢变化。Gorilla 就吃这两份可预测性，做到平均 <strong>1.37 字节一个点</strong>，约 12 倍压缩，且<strong>完全无损</strong>。</p>

  <p>时间戳：先存起始时间，之后每个点存”间隔”（<em>delta</em>），再之后每个点存”间隔的变化量”（<em>delta-of-delta</em>）。定时采集时 dod 恒为 0，只需 <strong>1 个 bit</strong>。其余情况按大小分桶：<code class="language-plaintext highlighter-rouge">10</code> + 7 bit（±63）、<code class="language-plaintext highlighter-rouge">110</code> + 9 bit、<code class="language-plaintext highlighter-rouge">1110</code> + 12 bit、<code class="language-plaintext highlighter-rouge">1111</code> + 32 bit。时间戳几乎白送。</p>

  <p>数值：把当前 float64 与上一个做 <em>XOR</em>。IEEE 754 的位布局让相邻且相近的数值共享指数位和高位尾数，异或结果前面一大片 0、后面也一大片 0。于是：</p>

  <ul>
    <li>XOR 为 0（值完全没变）→ 一个 bit <code class="language-plaintext highlighter-rouge">0</code></li>
    <li>XOR 非 0 → <code class="language-plaintext highlighter-rouge">1</code>，再看有效区间（<em>meaningful bits</em>）：若它的前导零数和尾随零数都落在上一行记下的窗口内，就用 <code class="language-plaintext highlighter-rouge">0</code> + 直接抄中间那截（<strong>复用窗口</strong>）；否则用 <code class="language-plaintext highlighter-rouge">1</code> + 5 bit 前导零数 + 6 bit 长度 + 中间那截（<strong>重报窗口</strong>）</li>
  </ul>

  <p>两个细节最见功夫：一是<strong>窗口按每个值决定</strong>，不做全局假设——所以数据剧烈跳变时只是那一段变胖，不会牺牲正确性，也不会拖累平时；二是<strong>解码只需要上一个值</strong>，状态是 O(1)，不用回看整块数据，所以它适合流式写入、也适合在读取时边解边算。</p>

  <h3 id="section-1">为什么重要</h3>

  <p>因为你每天用的监控系统几乎都在这条路上。Prometheus 的本地 TSDB 用 <em>delta-of-delta</em> 时间戳加 <em>XOR</em> 变体，InfluxDB 的 TSM、VictoriaMetrics、Gorilla 本身、AWS Timestream——同一套思路的不同变奏。一个集群的指标量动辄上亿条时间线，压缩比的差别不是”省点硬盘”，而是<strong>决定了单机能不能扛住、查询要不要扫过量数据、账单是一位数还是两位数</strong>。</p>

  <p>更深的一层是选型哲学。通用压缩（gzip、zstd）按块编码，压缩比可能更高，但<strong>解压必须从头来</strong>：想取第 100 万个点，得先把前面全解开，随机访问没了；CPU 代价也高。Gorilla 放弃了”榨干熵”，换取三件在存储引擎里更贵的东西：</p>

  <ul>
    <li><strong>无损</strong>——监控数据被有损压缩后，<code class="language-plaintext highlighter-rouge">p99</code> 就不对了，而 <code class="language-plaintext highlighter-rouge">p99</code> 正是你要看的东西</li>
    <li><strong>流式与 O(1) 状态</strong>——一个点只依赖前一个点，写入和解码都能边走边算</li>
    <li><strong>最坏情况有界</strong>——数据再乱，一个点也只多出几位，不会出现”这段数据压不动”的悬崖</li>
  </ul>

  <p>它教的是一种普适手法：<strong>要压缩的不是数据本身，而是数据里”没有变化”的那部分。</strong> 恒定间隔、缓慢漂移、稳定前缀——任何一条管道里长期重复的东西，都值得被识别出来、只写一次。</p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li><strong>阿潮的簿子</strong> —— 时序数据库里的一块时间线数据（time series）</li>
    <li><strong>“一个字都不许省”</strong> —— 无损压缩的硬约束，监控值不允许有损近似</li>
    <li><strong>每十分钟记一次</strong> —— 固定采样间隔（scrape interval）</li>
    <li><strong>先写”十分钟”，后面只记间隔</strong> —— <em>delta</em> 时间戳编码</li>
    <li><strong>只记”间隔变了多少”，几乎总是零</strong> —— <em>delta-of-delta</em> 编码，dod=0 时只花 1 bit</li>
    <li><strong>一个点 <code class="language-plaintext highlighter-rouge">·</code></strong> —— dod 为 0 的单比特标记</li>
    <li><strong>一画/两画/三画/四画加数</strong> —— 按 dod 大小分桶：<code class="language-plaintext highlighter-rouge">10</code>+7bit / <code class="language-plaintext highlighter-rouge">110</code>+9bit / <code class="language-plaintext highlighter-rouge">1110</code>+12bit / <code class="language-plaintext highlighter-rouge">1111</code>+32bit</li>
    <li><strong>相邻两行水位左边开头、右边末尾都一样</strong> —— IEEE 754 下相近数值共享高位，且定精度小数尾部位常为零</li>
    <li><strong>只抄中间那一小截不同的</strong> —— 对前值做 <em>XOR</em> 后只写 meaningful bits</li>
    <li><strong>水位一模一样时只落一个点</strong> —— XOR 为 0，编码成单个 <code class="language-plaintext highlighter-rouge">0</code> bit</li>
    <li><strong>记下”左边相同到第几位、右边到第几位”</strong> —— 前导零数（leading zeros）与尾随零数（trailing zeros）构成的窗口</li>
    <li><strong>把这对数写在第一页、之后一直沿用</strong> —— 全局固定窗口的错解：跳变时要么丢数据，要么给平时留出巨大冗余</li>
    <li><strong>“两端相同到哪儿该由每一行自己决定”</strong> —— 窗口按每个值判断，这是 Gorilla 的关键设计</li>
    <li><strong><code class="language-plaintext highlighter-rouge">··</code> 沿用上一行的窗口</strong> —— 控制位 <code class="language-plaintext highlighter-rouge">10</code>：复用窗口，只写中间那截</li>
    <li><strong><code class="language-plaintext highlighter-rouge">·|</code> 这一行重新报窗口</strong> —— 控制位 <code class="language-plaintext highlighter-rouge">11</code>：5 bit 前导零数 + 6 bit 长度，再写中间那截</li>
    <li><strong>台风那几页确实变厚，但只厚了那几页</strong> —— 最坏情况只有局部膨胀，不牺牲正确性、不拖累常态</li>
    <li><strong>十年簿子装进一只布袋</strong> —— 约 12 倍压缩，1.37 字节/点</li>
    <li><strong>司徒大人逐行还原，一分不差</strong> —— 无损、可流式解码，且只需前一个值作为状态</li>
  </ul>
</section>
<section class="en">
  <p>The Tide-Watching Tower stands at the far eastern edge of the sea cliff. The bronze bell at its top tells the fishing boats what the water is doing.</p>

  <p>The keeper is called A-Chao. Every ten minutes he climbs to the top, reads the water gauge cut into the stone wall, and writes the hour and the water level into his ledger. The level is recorded to six decimal places — a rule set by the Maritime Office in the capital, because the tide tables are computed from these ledgers. Get a digit wrong and a fishing boat drops anchor in the wrong place.</p>

  <p>A-Chao is meticulous. In a year, the ledgers fill a whole room.</p>

  <p>The trouble comes at the end of every winter. All the ledgers must be loaded onto ships and carried halfway around the coast to the capital. The ships are only so large, and the ledgers grow thicker every year. One year it took three ships; after that, Minister Situ wrote from the Maritime Office to say that next year there would be one. At the end of the letter was a cold sentence: <strong>not a single character may be omitted.</strong></p>

  <p>A-Chao tried omitting. He tried cutting six decimals to three and got a scolding by return mail. He tried writing “about this much” and the Maritime Office wrote back that three fishing boats had run aground on that tide table. What he needed was not to record less. He needed to <strong>write the same thing in fewer characters</strong>.</p>

  <p>So he sat down under the lamp and read his own ledgers from the beginning. In the seventh volume he noticed something small.</p>

  <p>Every row’s hour read the same interval: ten minutes, ten minutes, ten minutes. The words “ten minutes” appeared ten thousand times in one year. He changed the format to record <strong>only the interval</strong> — write it once at the top, then never again.</p>

  <p>A few volumes later he noticed a second small thing: <strong>the interval itself hardly ever changed.</strong> Only when he was late on the stairs, or a relief keeper arrived late, did it shift. So he stopped writing the interval and wrote <strong>how much the interval had changed</strong> — which was almost always zero.</p>

  <p>He added new marks to the ledger:</p>

  <ul>
    <li>A single dot <code class="language-plaintext highlighter-rouge">·</code> — the interval did not change at all (the overwhelming majority; one mark)</li>
    <li>One stroke <code class="language-plaintext highlighter-rouge">—</code> plus a small number — it shifted a little</li>
    <li>Two strokes <code class="language-plaintext highlighter-rouge">——</code> plus a middling number — more than a little</li>
    <li>Three strokes <code class="language-plaintext highlighter-rouge">———</code> plus a large number — more than a quarter hour</li>
    <li>Four strokes <code class="language-plaintext highlighter-rouge">————</code> — write the whole interval out again</li>
  </ul>

  <p>Ten thousand rows of hours collapsed into half a page of dots.</p>

  <p>Then he turned to the water-level column. That column was nine tenths of the book. He wrote consecutive levels in binary, lined them up row against row, and saw it at once: <strong>the left end of two rows was identical, and the right end was usually identical too.</strong> What actually differed was a short stretch in the middle.</p>

  <p>The reason was plain enough. The sea does not sit at 1.2 metres one moment and 300 metres the next. It drifts slowly inside the same order of magnitude, so the leading bits stay put for a long time; and since the gauge is cut to six decimals, the trailing bits are often zeros. Compare two rows and the middle is only a few bits wide.</p>

  <p>So he changed how he wrote the level: first, <strong>from which bit on the left the two rows agree</strong> and <strong>up to which bit on the right they agree</strong>, then copy out only the differing middle. A row of water level shrank from sixty-odd marks to a dozen. On a still night, when two readings were identical, he didn’t even need the middle — just a single dot: <code class="language-plaintext highlighter-rouge">·</code>, same as last time.</p>

  <p>In half a year he had saved nine tenths of the work.</p>

  <p>The real wall came in September.</p>

  <p>A typhoon crossed, and the sea rose six metres in four hours. The level jumped from 1.3 metres to 31 — the order of magnitude changed, the shared leading run collapsed from forty-odd bits to a handful, and the trailing run was a mess. He had taken a shortcut: write those two “where do they agree” numbers on the first page of the ledger and use them forever after. When the typhoon came, that scheme either could not hold the true level (which meant quietly omitting characters, exactly what the Minister forbade) or he had to leave the margin absurdly wide, costing a dozen extra marks on every ordinary point.</p>

  <p>He sat under the lamp watching the wind bend the window and understood one thing: <strong>“where the two ends agree” should not be a rule for the whole book. It should be each row’s own decision.</strong></p>

  <p>He rewrote the format. Every water-level row now opens with a small mark:</p>

  <ul>
    <li><code class="language-plaintext highlighter-rouge">··</code> — two dots: this row reuses the previous row’s pair of “agrees up to here” numbers and copies only the middle (nearly every ordinary row)</li>
    <li><code class="language-plaintext highlighter-rouge">·|</code> — a dot and a bar: this row announces the pair itself, then copies the middle (only for typhoons, season changes, jumps in magnitude)</li>
  </ul>

  <p>Not one character was spent unnecessarily. The typhoon pages really were thicker — but only those pages.</p>

  <p>That winter A-Chao packed ten years of ledgers into a single cloth sack and handed it to the boat. Minister Situ unpacked it, read for three days, and reconstructed every single row, to the last digit. He asked A-Chao what sorcery he had used.</p>

  <p>A-Chao said: I omitted nothing. I simply <strong>stopped writing down the things that never changed.</strong></p>

  <p>— By now you’ve probably recognised it: this is <em>Gorilla compression</em>. A-Chao writing only “how much the interval changed” is <em>delta-of-delta</em> timestamp encoding. Comparing two readings by XOR and copying only the differing middle is <em>XOR</em> floating-point encoding. And the two-dot mark that chooses between “reuse the last window” and “announce a new one” is the two control bits that carry Gorilla’s value encoding.</p>

  <h3 id="what-it-is">What it is</h3>

  <p><em>Gorilla</em> is the in-memory time-series compression scheme Facebook published in 2015 (“Gorilla: A Fast, Scalable, In-Memory Time Series Database”, VLDB). It attacks the storage problem of monitoring data: one metric sampled every 10 seconds is over 3 million points a year, and at “8-byte timestamp + 8-byte float64” per point that is 16 bytes and roughly 50 MB. Metric data is overwhelmingly <strong>predictable</strong> — fixed sampling intervals, slowly drifting values — and Gorilla eats exactly that predictability, landing at an average of <strong>1.37 bytes per point</strong>, about 12x, and <strong>completely losslessly</strong>.</p>

  <p>Timestamps: store the start time, then store the interval (<em>delta</em>) for each point, then store the change in the interval (<em>delta-of-delta</em>). On regular scraping dod is always 0 and costs <strong>one bit</strong>. Otherwise it buckets by magnitude: <code class="language-plaintext highlighter-rouge">10</code> + 7 bits (±63), <code class="language-plaintext highlighter-rouge">110</code> + 9 bits, <code class="language-plaintext highlighter-rouge">1110</code> + 12 bits, <code class="language-plaintext highlighter-rouge">1111</code> + 32 bits. Timestamps become nearly free.</p>

  <p>Values: <em>XOR</em> the current float64 with the previous one. IEEE 754’s bit layout means nearby values share exponent bits and high mantissa bits, so the XOR result is a long run of leading zeros and a long run of trailing zeros. Hence:</p>

  <ul>
    <li>XOR is 0 (the value did not change) → a single <code class="language-plaintext highlighter-rouge">0</code> bit</li>
    <li>XOR is non-zero → <code class="language-plaintext highlighter-rouge">1</code>, then look at the <em>meaningful bits</em>: if both the leading-zero count and the trailing-zero count fall inside the window recorded for the previous value, emit <code class="language-plaintext highlighter-rouge">0</code> and copy just the middle (<strong>reuse the window</strong>); otherwise emit <code class="language-plaintext highlighter-rouge">1</code> + 5 bits of leading zeros + 6 bits of length + the middle (<strong>announce a new window</strong>)</li>
  </ul>

  <p>Two details carry the design. First, <strong>the window is decided per value</strong>, not assumed globally — so a violent jump thickens only that stretch, never sacrificing correctness and never taxing the ordinary case. Second, <strong>decoding needs only the previous value</strong> — O(1) state, no rewinding to the start of a block — which is what makes it suitable for streaming writes and for decoding as you scan.</p>

  <h3 id="why-it-matters">Why it matters</h3>

  <p>Because almost every monitoring system you use is on this path. Prometheus’s local TSDB uses <em>delta-of-delta</em> timestamps plus a <em>XOR</em> variant; InfluxDB’s TSM, VictoriaMetrics, Gorilla itself, AWS Timestream — the same idea in different keys. With hundreds of millions of active series in a cluster, the compression ratio is not “saving some disk.” It decides whether a single node can hold the data, whether a query scans excessive bytes, and whether the bill has one digit or two.</p>

  <p>There is a deeper lesson in the choice. General-purpose compression (gzip, zstd) encodes in blocks and may reach a better ratio, but <strong>decompression must start from the beginning</strong>: to read point one-million you must first inflate everything before it, so random access is gone, and the CPU cost is real. Gorilla gives up on squeezing out every bit of entropy in exchange for three things a storage engine values more:</p>

  <ul>
    <li><strong>Losslessness</strong> — after lossy compression, <code class="language-plaintext highlighter-rouge">p99</code> is wrong, and <code class="language-plaintext highlighter-rouge">p99</code> is the whole point</li>
    <li><strong>Streaming, O(1) state</strong> — each point depends only on the previous one, so writes and decodes both proceed incrementally</li>
    <li><strong>A bounded worst case</strong> — however chaotic the data, a point costs only a few extra bits; there is no cliff where a stretch of data refuses to compress</li>
  </ul>

  <p>And it teaches a transferable move: <strong>what you compress is not the data, it is the part of the data that did not change.</strong> Fixed intervals, slow drift, stable prefixes — anything a pipeline repeats for a long time is worth identifying and writing exactly once.</p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li><strong>A-Chao’s ledger</strong> — one time series in a TSDB</li>
    <li><strong>“Not a single character may be omitted”</strong> — the lossless constraint; monitoring values may not be approximated</li>
    <li><strong>Recording every ten minutes</strong> — a fixed scrape interval</li>
    <li><strong>Write “ten minutes” once, then only the interval</strong> — <em>delta</em> timestamp encoding</li>
    <li><strong>Write only how much the interval changed — almost always zero</strong> — <em>delta-of-delta</em> encoding; dod = 0 costs one bit</li>
    <li><strong>The single dot <code class="language-plaintext highlighter-rouge">·</code></strong> — the one-bit marker for dod = 0</li>
    <li><strong>One/two/three/four strokes plus a number</strong> — the magnitude buckets: <code class="language-plaintext highlighter-rouge">10</code>+7 bits / <code class="language-plaintext highlighter-rouge">110</code>+9 bits / <code class="language-plaintext highlighter-rouge">1110</code>+12 bits / <code class="language-plaintext highlighter-rouge">1111</code>+32 bits</li>
    <li><strong>Two rows agreeing on the left end and the right end</strong> — under IEEE 754, nearby values share high bits, and fixed-precision decimals leave trailing zeros</li>
    <li><strong>Copying only the differing middle</strong> — <em>XOR</em> against the previous value, keeping only the meaningful bits</li>
    <li><strong>A single dot when the level did not move at all</strong> — XOR is 0, encoded as a lone <code class="language-plaintext highlighter-rouge">0</code> bit</li>
    <li><strong>Recording “agrees from bit N on the left, up to bit M on the right”</strong> — leading-zero and trailing-zero counts forming the window</li>
    <li><strong>Writing that pair on page one and reusing it forever</strong> — the wrong fix: a global fixed window either corrupts on a jump or bloats every ordinary point</li>
    <li><strong>“Where the ends agree should be each row’s own decision”</strong> — the window is judged per value; this is Gorilla’s key insight</li>
    <li><strong><code class="language-plaintext highlighter-rouge">··</code> reusing the previous row’s window</strong> — control bits <code class="language-plaintext highlighter-rouge">10</code>: reuse, then write only the middle</li>
    <li><strong><code class="language-plaintext highlighter-rouge">·|</code> announcing the window again</strong> — control bits <code class="language-plaintext highlighter-rouge">11</code>: 5 bits of leading zeros + 6 bits of length, then the middle</li>
    <li><strong>The typhoon pages are thicker, but only those pages</strong> — worst case is local inflation; correctness and the ordinary case are untouched</li>
    <li><strong>Ten years of ledgers in one cloth sack</strong> — about 12x compression, 1.37 bytes per point</li>
    <li><strong>The Minister reconstructing every row, digit for digit</strong> — lossless decoding with O(1) state: only the previous value is needed</li>
  </ul>
</section>]]></content><author><name></name></author><category term="databases" /><category term="storage" /><category term="observability" /><summary type="html"><![CDATA[望潮塔立在海崖最东头，塔顶那口铜钟是给渔船报潮的。]]></summary></entry><entry><title type="html">等回执的人 · The Two Who Waited for Each Other</title><link href="https://fables.day/2026/11/12/nagle-delayed-ack/" rel="alternate" type="text/html" title="等回执的人 · The Two Who Waited for Each Other" /><published>2026-11-12T00:00:00+00:00</published><updated>2026-11-12T00:00:00+00:00</updated><id>https://fables.day/2026/11/12/nagle-delayed-ack</id><content type="html" xml:base="https://fables.day/2026/11/12/nagle-delayed-ack/"><![CDATA[<section class="zh">
  <p>青溪渡口只隔着三十丈水，来回一趟船不过几口气的工夫。</p>

  <p>北岸管装船的是阿岚，南岸管收货的是老陈。两人素不相识，却各自守着一条祖上传下来的规矩。</p>

  <p>阿岚的规矩是<strong>不空跑</strong>：只要上一趟船的回执还没递回他手上，岸上这几箱零碎货就先攒着，攒够一整船再走。他觉得这道理再明白不过——你连上一船到了没有都不知道，还一箱一箱往外送，万一船沉了呢？万一南岸没人接呢？一条条小船白白荡过去，那是浪费。</p>

  <p>老陈的规矩是<strong>不单开票</strong>：来一箱货，他不急着写回执，等一等，看还有没有下一箱到了，两箱写一张票，省一张纸、省一趟人。人家问他一箱货的回执呢，他说：别急，兴许马上又来一箱。</p>

  <p>于是那天早上就出了这么一件怪事。</p>

  <p>北岸染坊的掌柜要传花样。第一箱花样递下去，阿岚手上没有未回执的船，立刻放行。船到南岸，老陈接了货，把箱子搬到廊下，笔搭在砚台上，心想：还有一箱吧？等等看。</p>

  <p>与此同时，第二箱花样也递到了阿岚手上。他抬头看了看手上的牌子——<strong>上一趟船还没回执</strong>。他把箱子往边上一搁：”等回执。”</p>

  <p>南岸的老陈在等第二箱，好凑一张票；北岸的阿岚在等第一箱的回执，好放第二箱。两个人各自完全按规矩办事，隔岸相望，谁也不看谁，井水不犯河水，可这条三十丈的水路，就这么空着。</p>

  <p>一直到老陈等得不耐烦了——他到底还有个底线：<strong>再没货来，也不能把回执压过四十口气</strong>。四十口气一到，他叹口气，单开一张票，让回程的小船带回去。</p>

  <p>阿岚接到回执，放了第二箱。老陈接了货，又等第二张回执的时机……</p>

  <p>一整个上午，染坊掌柜的每两箱花样，都要平白多等四十口气。他跑去渡口骂船慢、骂水急、骂艄公懒，把三十丈的水路量了三遍。水一点都不慢。慢的是两个都太会替对方省事的人。</p>

  <p>后来还是染坊掌柜自己想明白了，他给了三条路子。</p>

  <p>第一条，他让人给阿岚捎话：<strong>从今往后不许等，有一箱走一箱，回执没回来也照样走。</strong> 阿岚第一次听差点翻脸——这不是让他天天放空船吗？可水路通了。这就是后来人人挂在嘴上的那句”把等待关掉”。</p>

  <p>第二条，掌柜的不再一箱一箱递：<strong>两箱花样绑成一挑子，一次递下去。</strong> 一挑子就是满满一船，阿岚的规矩本来就不拦整船的货——他只管零碎。货一满，规矩自动失效。这比第一条省心得多。</p>

  <p>第三条，他托人跟老陈商量：<strong>收到货立刻开票，别攒。</strong> 老陈嘟囔了半天，说这样纸要费一半，但也就答应了。这一条治的是另一头。</p>

  <p>——到这儿你大概已经认出来了：阿岚那”不空跑”的规矩就是 <em>Nagle’s algorithm</em>，老陈那”不单开票”的规矩就是 <em>delayed ACK</em>，那”四十口气”的底线是延迟确认的定时器，而整条三十丈的水路空着不动的那个上午，就是<strong>写—写—读</strong>（<em>write-write-read</em>）场景下那个莫名其妙的延迟。</p>

  <h3 id="section">这是什么</h3>

  <p>TCP 上每一次发送，背后都有一笔”带宽税”要交：一个 40 字节的头部 + 一个必须被确认的序号。如果你的应用一次只写 1 字节，那这 1 字节的实际代价是 41 字节。</p>

  <p><em>Nagle’s algorithm</em>（RFC 896）就是针对这个的：<strong>发出去的小段，只要还有未被确认的数据在途，就先攒着；攒到一整段（<em>MSS</em>）再发。</strong> 注意它只拦小段——整段的数据随时可以走。这条规矩在 1984 年是纯赚：网络带宽比延迟贵得多，攒一攒是对的。</p>

  <p><em>Delayed ACK</em> 是同一条思路在接收端的镜像：收到一个段，不立刻回 ACK，而是<strong>等大约 40 毫秒，看能不能等到第二个段，让一个 ACK 覆盖两段</strong>（Linux 的上限就取在这个量级）。如果等不到，定时器到期，单独发一个 ACK。同样，它只对”小段单独到达”这种情况生效。</p>

  <p>两条各自合理的规矩叠在一起，就出现了著名的互锁：发送方拿着第二段等 ACK，接收方拿着 ACK 等第二段。谁都没错，谁都在等对方。破局只能靠接收方那个 40 毫秒的定时器——于是每一个”写—写—读”的往返，都被平白加上一个 40 毫秒。RPC、数据库驱动、某些 HTTP/1.1 客户端里，单次请求几十毫秒的莫名其妙的固定延迟，常常就是这么来的。它在局域网里最刺眼：传输本身只花 0.2 毫秒，你却量到 40。</p>

  <h3 id="section-1">为什么重要</h3>

  <p>因为它是一种<strong>看起来像”网络慢”的软件缺陷</strong>。延迟不出现在任何一层的监控里，不丢包、不重传、CPU 不忙、带宽不满，你量什么都是正常的，只有端到端的往返时间稳稳地多出几十毫秒。于是团队花几个月换网卡、调内核、加机器、升级交换机，而真正的原因是：应用写数据的方式，和两条 1980 年代的礼貌规矩。</p>

  <p>修法有三个方向，对应故事里那三条路：</p>

  <ul>
    <li><strong>关掉等待</strong>——<code class="language-plaintext highlighter-rouge">TCP_NODELAY</code>。代价是零碎小包满地跑，带宽效率下降；对交互式、请求-响应式的协议基本无脑开。</li>
    <li><strong>把货绑成整挑子</strong>——应用层合并写入（缓冲、<code class="language-plaintext highlighter-rouge">TCP_CORK</code>、批量成帧、协议里做 <em>pipelining</em>）。这是最优解：既没有 40 毫秒，也不浪费带宽。故事的教训在这里——<strong>能在一层解决的问题，不要留给对端去猜。</strong></li>
    <li><strong>让接收方立刻开票</strong>——<code class="language-plaintext highlighter-rouge">TCP_QUICKACK</code>。但它会被系统重置，不是稳定解，通常只适合在库内部临时用。</li>
  </ul>

  <p>还有一点反直觉的：延迟不是恒定的 40 毫秒，而是”最坏时正好一个 RTT”。当往返时间本身很短时，40 毫秒的定时器显得巨大；在长肥管道上反而被淹没了。所以同一个 bug 在开发机（本机回环，延迟为零）上永远复现不出来，一上线到同机房就现形。</p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li><strong>阿岚”不空跑”的规矩</strong> —— <em>Nagle’s algorithm</em></li>
    <li><strong>老陈”不单开票”的规矩</strong> —— <em>delayed ACK</em></li>
    <li><strong>“上一趟船还没回执，这箱先攒着”</strong> —— 有未确认数据在途时，小段被握在发送缓冲区</li>
    <li><strong>“攒够一整船再走”</strong> —— 满 <em>MSS</em> 的段不受 Nagle 限制，立即发送</li>
    <li><strong>“两箱写一张票”</strong> —— 一个 ACK 覆盖两个段</li>
    <li><strong>“四十口气”的底线</strong> —— 延迟确认的定时器（Linux 约 40 毫秒）</li>
    <li><strong>第一箱花样立刻放行</strong> —— 无在途数据时，第一个小段仍然立即发出</li>
    <li><strong>两人隔岸互相干等</strong> —— <em>write-write-read</em> 互锁</li>
    <li><strong>每两箱多等四十口气</strong> —— 每次往返被固定加上一个延迟确认周期</li>
    <li><strong>三十丈的水路空着</strong> —— 带宽完全空闲，却仍然慢</li>
    <li><strong>掌柜骂船慢、量水路</strong> —— 误判为网络问题，换网卡调内核</li>
    <li><strong>“从今往后不许等”</strong> —— <code class="language-plaintext highlighter-rouge">TCP_NODELAY</code></li>
    <li><strong>“两箱绑成一挑子”</strong> —— 应用层合并写入 / <code class="language-plaintext highlighter-rouge">TCP_CORK</code> / 协议 <em>pipelining</em>（正解）</li>
    <li><strong>“收到货立刻开票”</strong> —— <code class="language-plaintext highlighter-rouge">TCP_QUICKACK</code>（会被重置，不是稳定解）</li>
    <li><strong>本机回环上测不出来</strong> —— 延迟为零时定时器纹丝不动，问题只在真实网络里显形</li>
  </ul>
</section>
<section class="en">
  <p>The Green Creek ferry crossing is thirty yards of water. A boat makes the round trip in the time it takes to draw a few breaths.</p>

  <p>On the north bank, the man who loads the boats is called Lan. On the south bank, the man who receives the cargo is called Chen. They have never met, and each keeps one rule handed down from his own people.</p>

  <p>Lan’s rule is <strong>never sail half-empty</strong>: as long as the receipt from the previous crossing has not come back into his hand, small crates sit on the dock and wait until there is a full boatload. The logic seems unanswerable. You don’t even know whether the last boat arrived — why would you keep pushing crates out one at a time? What if it sank? What if nobody is on the south bank to take delivery? A string of little boats drifting across for nothing is pure waste.</p>

  <p>Chen’s rule is <strong>never write a single ticket</strong>: when a crate arrives he doesn’t rush to write a receipt. He waits a moment to see whether another crate is coming, then writes one ticket for two — saving a sheet of paper and a trip. Asked where the receipt for the first crate is, he says: easy, another one is probably right behind it.</p>

  <p>And so, that morning, something strange happened.</p>

  <p>A dyer on the north bank needed to send his patterns across. The first crate went down to the dock, and Lan had no unacknowledged crossing on his board, so he released it at once. The boat reached the south bank, Chen took the crate, set it under the eaves, and laid his brush on the inkstone. There’ll be another one, he thought. Let’s see.</p>

  <p>At that same moment, the second crate of patterns arrived at Lan’s dock. He glanced at the board — <strong>the last crossing has no receipt yet</strong>. He set the crate aside. “Wait for the receipt.”</p>

  <p>Chen on the south bank was waiting for a second crate so he could write one ticket for two. Lan on the north bank was waiting for the first crate’s receipt so he could release the second. Each was following his own rule perfectly. Thirty yards of water, two men facing each other, neither looking across, and the river lay completely empty.</p>

  <p>Eventually Chen ran out of patience — he did, after all, have a bottom line: <strong>a receipt must never be held longer than forty breaths</strong>. When forty breaths were up, he sighed, wrote a single ticket, and sent it back with the returning boat.</p>

  <p>Lan took the receipt and released the second crate. Chen received it and began waiting again for the right moment to write the receipt…</p>

  <p>All morning long, every two crates of the dyer’s patterns cost him an extra forty breaths for nothing. He went down to the ferry and cursed the boats for being slow, the current for being fast, the boatmen for being lazy. He measured the thirty yards of water three times. The water was not slow. What was slow was two men each trying a little too hard to save the other some trouble.</p>

  <p>In the end the dyer worked it out himself, and he offered three fixes.</p>

  <p>The first: he sent word to Lan — <strong>from now on, no waiting. One crate, one boat, receipt or no receipt.</strong> Lan nearly took offence. Was he to sail empty boats all day? But the river started moving. This is the fix everyone remembers: turn the waiting off.</p>

  <p>The second: the dyer stopped handing crates over one at a time. <strong>He bound two crates into one yoke and sent them down together.</strong> A yoke is a full boatload, and Lan’s rule never blocked full boats — it only governed odds and ends. Fill the boat and the rule disables itself. Far less trouble than the first fix.</p>

  <p>The third: he had someone talk to Chen. <strong>Write the ticket the moment the crate lands. Don’t save up.</strong> Chen grumbled about spending twice the paper, but he agreed. This one treats the other end.</p>

  <p>— By now you’ve probably recognised it: Lan’s “never sail half-empty” is <em>Nagle’s algorithm</em>, Chen’s “never write a single ticket” is <em>delayed ACK</em>, the “forty breaths” bottom line is the delayed-ACK timer, and that whole morning with thirty yards of empty water is the mysterious delay in the <strong>write-write-read</strong> pattern.</p>

  <h3 id="what-it-is">What it is</h3>

  <p>Every send on TCP pays a bandwidth tax: a 40-byte header plus a sequence number that has to be acknowledged. If your application writes one byte at a time, one byte of payload really costs 41 bytes on the wire.</p>

  <p><em>Nagle’s algorithm</em> (RFC 896) exists to fix that: <strong>a small outgoing segment is held back whenever there is still unacknowledged data in flight; it goes out once a full segment (<em>MSS</em>) has accumulated.</strong> Note that it only holds small segments — a full-sized segment always leaves immediately. In 1984 this was a straight win, because bandwidth was far more expensive than latency, and batching was right.</p>

  <p><em>Delayed ACK</em> is the same idea mirrored at the receiver: on receiving a segment, don’t send an ACK at once. <strong>Wait roughly 40 milliseconds to see whether a second segment arrives, so one ACK can cover two</strong> (that’s the order of magnitude Linux uses). If nothing comes, the timer expires and a lone ACK goes out. Again, it only applies to small segments arriving alone.</p>

  <p>Stack two individually sensible rules and you get the famous interlock: the sender holds the second segment waiting for an ACK, the receiver holds the ACK waiting for a second segment. Neither is wrong. Each is waiting for the other. The only way out is the receiver’s 40-millisecond timer — so every write-write-read round trip pays a flat 40 milliseconds for nothing. The odd fixed delay of tens of milliseconds in RPC calls, database drivers, and some HTTP/1.1 clients usually comes from exactly this. It is most glaring on a LAN: the transfer itself takes 0.2 milliseconds and you measure 40.</p>

  <h3 id="why-it-matters">Why it matters</h3>

  <p>Because it is a <strong>software defect that looks like a slow network</strong>. The latency shows up in no layer’s monitoring: no loss, no retransmits, idle CPU, empty bandwidth. Everything you measure looks fine, yet end-to-end round-trip time is steadily dozens of milliseconds longer than it should be. Teams then spend months swapping NICs, tuning kernels, adding machines, upgrading switches — when the real cause is how the application writes its data, plus two polite rules from the 1980s.</p>

  <p>There are three directions of fix, matching the story’s three paths:</p>

  <ul>
    <li><strong>Turn the waiting off</strong> — <code class="language-plaintext highlighter-rouge">TCP_NODELAY</code>. The cost is small packets everywhere and worse bandwidth efficiency; for interactive, request-response protocols it’s essentially a reflex.</li>
    <li><strong>Bind the crates into one yoke</strong> — coalesce writes in the application (buffering, <code class="language-plaintext highlighter-rouge">TCP_CORK</code>, batch framing, protocol <em>pipelining</em>). This is the best answer: no 40 milliseconds and no wasted bandwidth. The story’s real lesson is here — <strong>a problem you can fix at your own layer shouldn’t be left for the peer to guess about.</strong></li>
    <li><strong>Make the receiver write its ticket immediately</strong> — <code class="language-plaintext highlighter-rouge">TCP_QUICKACK</code>. But the stack resets it, so it isn’t a stable solution and generally belongs inside a library, used temporarily.</li>
  </ul>

  <p>One counter-intuitive point: the delay is not a constant 40 milliseconds, it’s “at worst one full RTT.” On a very short path the 40-millisecond timer is enormous; on a long fat pipe it gets drowned out. Which is why the same bug never reproduces on a developer machine over loopback — zero latency, timer never fires — and appears the moment you deploy to a real network in the same datacentre.</p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li><strong>Lan’s “never sail half-empty” rule</strong> — <em>Nagle’s algorithm</em></li>
    <li><strong>Chen’s “never write a single ticket” rule</strong> — <em>delayed ACK</em></li>
    <li><strong>“The last crossing has no receipt, so hold this crate”</strong> — small segments held in the send buffer while unacked data is in flight</li>
    <li><strong>“Wait until the boat is full, then sail”</strong> — full-<em>MSS</em> segments bypass Nagle and leave immediately</li>
    <li><strong>“One ticket for two crates”</strong> — a single ACK covering two segments</li>
    <li><strong>The “forty breaths” bottom line</strong> — the delayed-ACK timer (roughly 40 ms on Linux)</li>
    <li><strong>The first crate released at once</strong> — with nothing in flight, the first small segment still goes out immediately</li>
    <li><strong>Two men on opposite banks waiting each other out</strong> — the <em>write-write-read</em> interlock</li>
    <li><strong>Every two crates costing an extra forty breaths</strong> — a flat delayed-ACK period added to each round trip</li>
    <li><strong>Thirty yards of empty water</strong> — bandwidth completely idle, yet still slow</li>
    <li><strong>The dyer cursing the boats and measuring the river</strong> — misdiagnosing it as a network problem, swapping NICs and tuning kernels</li>
    <li><strong>“From now on, no waiting”</strong> — <code class="language-plaintext highlighter-rouge">TCP_NODELAY</code></li>
    <li><strong>Binding two crates into one yoke</strong> — application-level write coalescing / <code class="language-plaintext highlighter-rouge">TCP_CORK</code> / protocol <em>pipelining</em> (the right fix)</li>
    <li><strong>“Write the ticket the moment the crate lands”</strong> — <code class="language-plaintext highlighter-rouge">TCP_QUICKACK</code> (gets reset, not a stable fix)</li>
    <li><strong>It never reproduces over loopback</strong> — at zero latency the timer never fires, so the problem only shows on a real network</li>
  </ul>
</section>]]></content><author><name></name></author><category term="networking" /><category term="performance" /><summary type="html"><![CDATA[青溪渡口只隔着三十丈水，来回一趟船不过几口气的工夫。]]></summary></entry><entry><title type="html">红布条下的湖 · The Lake Beneath the Red Cloth</title><link href="https://fables.day/2026/11/11/spanning-tree-protocol/" rel="alternate" type="text/html" title="红布条下的湖 · The Lake Beneath the Red Cloth" /><published>2026-11-11T00:00:00+00:00</published><updated>2026-11-11T00:00:00+00:00</updated><id>https://fables.day/2026/11/11/spanning-tree-protocol</id><content type="html" xml:base="https://fables.day/2026/11/11/spanning-tree-protocol/"><![CDATA[<section class="zh">
  <p>柳湾九村环着一片湖。九村之间的路不是水路，是石堤——宽的石堤能过一辆牛车，窄的只容两人错身。</p>

  <p>石堤上跑的是<strong>信童</strong>。柳湾的规矩很老，也很简洁：<strong>一个村子收到一封通告，抄一份，凡是来路以外的每条堤，都送出去。</strong> 那封通告抄得清清楚楚，还在信封背面画了村子收到它的时辰。</p>

  <p>早年的堤少——一条主堤串起五村，两条支堤各通两村，形状就是一棵树，从根到梢，一步一条，没有第二条路。这个规矩用了六十年，没出过事。</p>

  <p>后来九村凑钱，修了环湖堤，又从中间横插了三道捷径，为的是赶集不用绕。路从一棵树，变成了一张网。修堤那年，谁也没多看这个变化一眼。</p>

  <p>那年谷雨，第一封通告是”开市”。第一天没事。第二天有事：通告沿着环湖堤跑了一整圈，回到最初发出它的那个村口。村口的老门房接过抄件，按六十年的规矩办——抄一份，来路以外的每条堤都送出去。他没认出这是自己昨天早上送走的那封。</p>

  <p><strong>同一封通告，现在有两份在路上跑。</strong></p>

  <p>第三圈四份。第四圈八份。到第四十圈，十九条堤上双向全是信童，怀里抱着同一张纸，迎面撞上，谁也不让。村子越多，抄得越慢；抄得越慢，一份抄件占路越久；占路越久，路上的抄件越多。</p>

  <p>桥头的老周后来总结了一句话，说得比谁都准：</p>

  <p><strong>问题不在信太多，问题在信封上没有”走过九村就作废”这一行字。</strong></p>

  <p>水路文书都有这一行——商号的札子上明写着”过三驿不达则毁”，船家一看就知道该烧了。石堤上的信没有。柳湾的规矩是”送到就算”，不是”送到为止”。<strong>一份抄件可以永远在环上转，而且每转一圈就多一倍。</strong></p>

  <p>第七天，湖东的粮船靠不了岸。不是堤堵了，是<strong>码头承认不了货单</strong>——账房要等”开市”通告落定才敢收粮，而通告永远在路上。九村的米价一天涨三回。柳湾的灯语也传不出去，因为灯语也是信童传的。</p>

  <p><strong>第一个办法是限速。</strong></p>

  <p>族长下令：每条堤上最多同时跑四个信童，多的一律拦住。</p>

  <p>结果很怪：湖面安静了半刻钟，然后更糟。因为跑得最凶的那几封正好被放进了堤里，而正经的货单排在堤外，一动不能动。<strong>限速只保证了路不塌，它一个字都没说谁该上路。</strong></p>

  <p><strong>第二个办法，是老族长拿着一根刻了刻度的竹尺，坐在湖西老槐树下拿出来的。</strong></p>

  <p><strong>一、定一个宗主。</strong> 九村各在门楣上刻一个<strong>名号</strong>：先比门楣上那个数字，数字一样再比名号本身（比如”槐树村”排在”柳树村”前头，因为槐的顺序在前）。<strong>名号最小的那个村子是宗主。</strong> 它未必是最强的，它只是名字最小。这条规矩妙在<strong>不需要开会</strong>：你只要报得出你的名号和邻居的名号，你自己就能算出谁是宗主。</p>

  <p><strong>二、量距离，但只跟自己邻居讲话。</strong> 竹尺给每段堤一个<strong>脚价</strong>：堤越窄越绕，价越高；堤越宽越直，价越低。一个村子到宗主的”路程价” = 最近那条通往宗主方向的上游堤的价 + 上游村子报出来的价。<strong>每个村子只需要报一个数给自己的邻居，也只需要听邻居报的数。</strong> 没有人去把全湖走一遍。</p>

  <p><strong>三、每个村子只留一条归路。</strong> 从通往宗主的那些路里，挑<strong>脚价最低</strong>的一条，敞着走人。价一样时比上游村子的名号，再一样时比那一段堤的门牌号。一个村子<strong>只有一条</strong>归路。</p>

  <p><strong>四、两村之间的那段堤，只认一个当值的村子。</strong> 谁离宗主更近（路程价更低，一样则比名号），谁负责把这段堤上的信往外送，它的堤口敞着；<strong>另一头的堤口，挂红布。</strong></p>

  <p><strong>五、挂了红布的堤，人不能走，耳朵要留着。</strong> 红布后面照旧站一个听差的，一刻不停地听宗主那边传过来的心跳木牌——宗主每两拍敲一次。他不送信。但他知道那条堤是死的还是活的。<strong>柳湾九村的命，就悬在这件看起来最没用的事上。</strong></p>

  <p><strong>六、等二十拍才认死，认死了还要先听、再认。</strong> 听差连着二十拍听不见心跳，才判上游断了，那条归路作废；然后从挂了红布的路里挑最好的一条，改作归路。改之前还有两道：先”听”十五拍——什么都别送，只听两边的名号和价；再”认”十五拍——开始记路，先别送信；十五拍之后才放信童上堤。看着啰嗦，这是为了让全湖换路的时候<strong>不会一边把旧账翻掉、一边又把新账用错</strong>。</p>

  <p>十九条堤，敞着九条，挂红布十条。一封信从湖西到湖东，只走一条不重复的路；抄件的数量从”每圈翻一倍”变成”每村一份”。</p>

  <p><strong>第三年春天，出了第二件事，比第一件更值得记。</strong></p>

  <p>湖东三村嫌归路绕远——明明有一条直堤，走过去三里，绕归路要十一里。他们说服了老族长，试着把那条红布撤开半天。</p>

  <p><strong>半刻钟。</strong> 湖上的信童翻了三倍，两个村口的门房当场晕了。红布当天挂回去，从此没人再提。</p>

  <p>九村后来懂了那句谁也没说出口的话：<strong>红布不是浪费，红布是湖还活着的唯一条件。</strong></p>

  <p><strong>第四年夏汛，湖心那道主堤塌了。</strong></p>

  <p>半个湖接不上宗主。而更麻烦的事在同一刻发生：各村的”认路账”全错了——账上写着”寄湖东渔行的信走北堤”，可北堤那头已经没人了。信童抱着信跑到断口，折回，再跑，路上还照旧被别的信童拦住问路。</p>

  <p>宗主的应对很省力：派出<strong>催烧</strong>的信童一路快跑，让每个村子把旧账烧得比平时快得多——平时账记三个月才擦，这几天二十天就擦。误投的信在半刻钟内就停了。</p>

  <p>而这一次，全湖重新走通信只在<strong>一拍</strong>之内，不是以前的二十拍加十五拍加十五拍。</p>

  <p>为什么？因为挂了红布那十条堤后面的听差，早就在跟对面的村子<strong>对着话</strong>——两边都留着上一轮的名号和价。归路一断，听差直接喊：”我接。”对面立刻答：”你接。”不用等”听十五拍、认十五拍”，也不用等二十拍判死。旧规矩是<strong>死等</strong>，新规矩是<strong>先约好</strong>。柳湾人后来把这套改法叫”对答改路”。</p>

  <p><strong>第五年，出了两桩怪事，被记进了族规。</strong></p>

  <p>第一桩：湖东新搬来一户大户，在自家门楣上刻了个比现任宗主还小的数字，想让全湖的信都绕到他家过一遍，好抽成。族规添了一条：<strong>从村外大路进来的方向，凡报出的名号比现任宗主还小的，一律不理，并且把这个口封掉。</strong> 宗主是你算出来的，不是你自己喊出来的。</p>

  <p>第二桩：湖边一户人家院里自己绕成了一个圈——他买了两个院门，两边都通到正屋，正屋的抄件往两个门都送。院门口一旦收到心跳木牌，说明这个口本来不该有木牌。族规又添一条：<strong>通到人家院子的门口不算堤；院门口收到木牌，立刻插门并喊人来查。</strong> 别让一户人家把全湖拖下水。</p>

  <p>到这儿，你大概已经认出来了：这就是<strong>生成树协议</strong>（<em>Spanning Tree Protocol</em>，<em>STP</em>）。</p>

  <h3 id="section">这是什么</h3>

  <p>先说清它要挡的那个东西是什么。</p>

  <p>以太网的转发是<strong>看门牌就送</strong>：交换机收到一帧，查目的 MAC 在哪个口，从那个口送出去；查不到，就从<strong>除了来路以外的所有口</strong>送出去（<em>unknown unicast flooding</em>）。广播帧更是无条件往所有口送。</p>

  <p>这套机制的全部安全感，建立在”路是一棵树”的假设上。而二层<strong>没有 TTL</strong>——IP 包头里有”跳数用完就丢”的字段，以太网帧里没有。<strong>一个帧进了环，就永远不会死。</strong> 它只会绕着环越跑越多：每个交换机对一个广播帧只复制一份，但十个交换机首尾相连，就是十份在绕；而未知单播的洪泛再叠上去，一圈之内就能把链路跑到线速。</p>

  <p>三条后果同时发生：<strong>带宽被吃光</strong>；<strong>每个交换机的 MAC 表都在疯狂翻转</strong>（同一个源 MAC 从两个口交替出现），于是连本该精确转发的单播也被洪泛；<strong>交换机的 CPU 被中断打满</strong>，连管理面都连不上——你连上去改配置的通道，也是那条被打死的链路。</p>

  <p><strong>STP（IEEE 802.1D）做的事极其干脆：它把有环的网，算成一棵树，然后把不在树上的口关掉。</strong></p>

  <p><strong>一、选举根桥。</strong> 每个桥有个 <strong>8 字节桥 ID</strong>：2 字节优先级（默认 <code class="language-plaintext highlighter-rouge">32768</code>，必须是 <code class="language-plaintext highlighter-rouge">4096</code> 的倍数）+ 6 字节 MAC。<strong>优先级最低、再 MAC 最小者为根桥。</strong> 之所以用优先级而不是”谁最强”，是为了让人能手工指定——你把它调成 <code class="language-plaintext highlighter-rouge">4096</code>，你就是根。Cisco 的 PVST+ 会把 VLAN ID 加到优先级低 12 位上（<em>sys-id-ext</em>），所以同一台物理交换机在不同 VLAN 里可以是不同的根。</p>

  <p><strong>二、BPDU：心跳木牌。</strong> 根桥每 <strong>2 秒</strong>（Hello）发一个配置 BPDU，目的 MAC 是 <code class="language-plaintext highlighter-rouge">01:80:C2:00:00:00</code>（交换机不会转发这个地址，所以心跳木牌不占信路）。BPDU 里四个关键字段：<strong>根桥 ID</strong>、<strong>根路径开销</strong>、<strong>发送方桥 ID</strong>、<strong>发送方端口 ID</strong>。非根桥收到后改写”根路径开销”和”发送方”再往下传——这就是”只跟邻居讲话，只报一个数”。802.1D 里只有根桥<strong>产生</strong> BPDU，其余桥转发；RSTP 里每个桥每 2 秒都自己发。</p>

  <p><strong>三、端口角色与阻塞。</strong> 每台非根桥选<strong>一个根端口</strong>（<em>root port</em>）：到根的路径开销最低，平手比发送方桥 ID，再平手比发送方端口 ID。每条链路（网段）上选<strong>一个指定端口</strong>（<em>designated port</em>）：这段上离根最近的那台桥负责向外送，平手比桥 ID。<strong>剩下的端口就是阻塞端口</strong>（<em>alternate</em> / <em>backup</em>）——它们<strong>不转发数据帧，但照常收、照常解析 BPDU</strong>。这一点是整套东西的支点：<strong>阻塞不等于断电，阻塞是”闭嘴听”</strong>。听差必须一直知道那条堤是活的。</p>

  <p>开销默认值（802.1t 长整型）：<code class="language-plaintext highlighter-rouge">20,000,000 / 速率(Mbps)</code> —— 10M 是 2,000,000，100M 是 200,000，1G 是 20,000，10G 是 2,000。老 802.1D-1998 的短整型是 10M=100、100M=19、1G=4、10G=2，小到 10G 和 100G 都是 2，<strong>根本分不出快慢</strong>——这就是为什么现代网络必须用长开销。</p>

  <p><strong>四、状态与计时器。</strong> 802.1D 有五个状态：<code class="language-plaintext highlighter-rouge">Disabled</code>、<code class="language-plaintext highlighter-rouge">Blocking</code>、<code class="language-plaintext highlighter-rouge">Listening</code>（15 秒）、<code class="language-plaintext highlighter-rouge">Learning</code>（15 秒）、<code class="language-plaintext highlighter-rouge">Forwarding</code>。一个端口从”决定要开”到”真的转数据”，要经过听、学两段，各 <strong>Forward Delay = 15 秒</strong>；判定上游死了要等 <strong>Max Age = 20 秒</strong>。所以最坏收敛 <strong>20 + 15 + 15 = 50 秒</strong>（常说的 30–50 秒就是这么来的）。听和学的区别在于：听的时候连 MAC 都不学（因为拓扑还没定，学了也是错的），学的时候学但不转发。</p>

  <p><strong>五、拓扑变了，全湖赶紧烧旧账。</strong> 一台桥发现链路变化，就往根的方向发 <strong>TCN BPDU</strong>；根收到后置 <strong>TC 标志</strong>，并在 <code class="language-plaintext highlighter-rouge">Max Age + Forward Delay</code>（35 秒）内让全部桥把 MAC 表老化时间从默认 <strong>300 秒</strong>缩到 <strong>15 秒</strong>。这就是”催烧”：不烧，流量会一直往那个已经不存在的方向送，送到 TTL 之外（二层没有 TTL，所以更糟——它会一直送到有人把它冲掉）。</p>

  <p><strong>六、RSTP：把”死等”改成”先约好”。</strong> IEEE 802.1w（已并入 802.1D-2004）把五个状态压成三个（<code class="language-plaintext highlighter-rouge">Discarding</code> / <code class="language-plaintext highlighter-rouge">Learning</code> / <code class="language-plaintext highlighter-rouge">Forwarding</code>），角色扩成四个（根、指定、<strong>alternate</strong>、<strong>backup</strong>），并加了<strong>提议/同意</strong>（<em>proposal / agreement</em>）握手：一个端口想立刻转发，就向下游发一个带 proposal 位的 BPDU；下游把自己的非边缘口全部同步到 discarding，回一个 agreement，这边立刻进转发。<strong>不必再等两个 15 秒</strong>，典型收敛在<strong>亚秒到几秒</strong>。它还把”边缘口”（<em>edge port</em>，接终端的口）直接置为转发，并把 alternate 端口预先留在身边——根口一断，alternate 立刻顶上。</p>

  <p><strong>七、MSTP：一棵树不够用。</strong> IEEE 802.1s 允许把 VLAN 映射到多个<strong>实例</strong>，每个实例一棵独立的树。于是可以让 VLAN 10 堵这条路、VLAN 20 堵那条路，<strong>同一条被 STP 判死的链路对另一半流量是活的</strong>。这是解决”红布浪费带宽”怨言的第一个正式答案。</p>

  <h3 id="section-1">为什么重要</h3>

  <p><strong>一、二层环不是慢，是秒级致命，而且看起来像别的问题。</strong> IP 路由环会被 TTL 挡住，最坏也就是丢几十个包。以太网环没有 TTL，一个帧可以在里面转到天亮。症状是<strong>广播包速率瞬间打到线速、交换机 CPU 100%、MAC 表在两个口之间跳（MAC flapping）、全网丢包</strong>——而你连 SSH 进去看的那条链路，用的正是同一个被打死的广播域。<strong>在二层，你可能是自己关掉了自己唯一的救援通道。</strong></p>

  <p><strong>二、收敛时间不是一个性能数字，它决定你敢不敢做双上行。</strong> 802.1D 的 30–50 秒足够让一个 OSPF/BGP 邻居掉光、让一次数据库主从切换完成、让几千个 TCP 连接超时重连。这就是为什么现网几乎都在用 RSTP/MSTP，也是为什么在 L2 上做”双活上行”总带着一点心虚。</p>

  <p><strong>三、STP 最大的问题不是它算得慢，是它必须堵掉一半链路。</strong> 一棵树用不到环里的全部带宽，这不是实现缺陷，这是”树”的定义。所以现代数据中心的做法不是把 STP 调快，而是<strong>根本不让二层成环</strong>：把 L2 域压到机柜内（”L2 到接入交换机为止”），上行全部跑 L3 路由，用 <strong>ECMP</strong> 把每条链路都用满（路由有 TTL，成环不会杀死你）；或者用 <strong>MLAG / vPC / EVPN 多归属</strong> 让两条上行同时转发；或者干脆用 <strong>EVPN-VXLAN</strong> 把二层语义搬到三层网络之上，让底层靠路由收敛。<strong>从”堵掉一半”到”全都用上”，从来不是把 STP 调快能达到的。</strong></p>

  <p><strong>四、绝大多数 STP 事故是误配，不是协议缺陷。</strong> 三种最高频：<strong>根桥被随便一台接入交换机抢走</strong>（有人优先级没配、或者买了一台默认 <code class="language-plaintext highlighter-rouge">32768</code> 但 MAC 更小的新设备）→ 全网流量重新绕行一条 1G 的接入链路。<strong>边缘口接了个小交换机、小交换机自己绕成环</strong> → 广播风暴从机柜里长出来。<strong>单向链路（光纤只断了一个方向）导致阻塞口收不到 BPDU，误以为上游死了，把自己打开</strong> → 环回来了，而这次没人知道为什么。对应三道防线：<strong>Root Guard</strong>（在朝外的口上，凡收到优于现任根的 BPDU 就把它挡在 discarding，不允许别人篡位）、<strong>BPDU Guard</strong>（在边缘口上，一旦收到 BPDU 说明这里不该是交换机，立刻 err-disable）、<strong>Loop Guard / UDLD</strong>（阻塞口一直收不到 BPDU 时，把它锁死在阻塞态，而不是打开）。</p>

  <p><strong>五、它的设计哲学值得抄：用”最小的号”而不是”最强的号”选主。</strong> 根桥不是选举出来的，是<strong>算出来</strong>的——只要每个桥知道自己的 ID 和邻居的 ID，全网必然算出同一个根，不需要任何共识协议、不需要投票、不需要多数派。这是分布式系统里最省事的一类选举：<strong>把决策压成一个全序，然后让每个节点独立算同一个 argmin。</strong> 它的代价也很清楚：<strong>没有防欺骗</strong>——任何人报一个更小的号就能篡位。STP 用 Root Guard 在边界上补这一刀，而不是在协议里修。</p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li>一封信被抄给”来路以外的每条堤”，路成环则同一封信无限翻倍 → 交换机对未知单播与广播帧向除入端口外的所有口洪泛，二层成环即广播风暴</li>
    <li>信封上没有”走过九村就作废”这一行 → 以太网帧没有 TTL，环里的帧永远不会自己死掉</li>
    <li>四条堤上双向都是信童，迎面撞上，谁也不让 → 环上的帧在线速上自我复制，链路饱和</li>
    <li>村子越多抄得越慢，抄得越慢占路越久 → 洪泛随拓扑规模放大，交换机 CPU 与中断被广播吃掉</li>
    <li>各村的”认路账”记着某个方向 → 交换机 MAC 地址表（CAM 表）</li>
    <li>信童跑到断口，折回，再跑 → 陈旧 MAC 表项把流量送往已不存在的方向</li>
    <li>门楣上刻的号：先比数字，数字一样比名号 → 8 字节桥 ID：2 字节优先级（默认 32768，4096 的倍数）+ 6 字节 MAC，优先级低者优先，再比 MAC 小者</li>
    <li>号最小的村子是宗主，不用开会，自己算得出 → 根桥选举：全网独立算同一个最小桥 ID，不需要任何共识协议</li>
    <li>竹尺给每段堤一个脚价：越窄越绕越贵 → 端口路径开销，802.1t 为 <code class="language-plaintext highlighter-rouge">20,000,000 / 速率(Mbps)</code>（1G = 20,000）</li>
    <li>每个村子只报一个数给邻居，只听邻居的数 → BPDU 里的根路径开销，非根桥收到后加上本地开销再向下游宣告</li>
    <li>心跳木牌每两拍敲一次，宗主发 → 配置 BPDU：Hello 2 秒，由根桥产生</li>
    <li>木牌走的是专门的传令牌道，不占信路 → BPDU 目的 MAC <code class="language-plaintext highlighter-rouge">01:80:C2:00:00:00</code>，交换机不转发</li>
    <li>每个村子只留一条最低价的归路 → 每台非根桥只有一个根端口（<em>root port</em>）</li>
    <li>两村之间那段堤只认离宗主更近的那个村当值 → 每条网段上只有一个指定端口（<em>designated port</em>），由离根最近（平手比桥 ID）的桥承担</li>
    <li>另一头的堤口挂红布 → 阻塞端口（<em>alternate</em> / <em>backup</em>），不转发数据</li>
    <li>挂红布的堤后面照旧站一个听差的 → 阻塞口仍然接收并处理 BPDU，这正是它能在故障时立刻接管的理由</li>
    <li>连二十拍听不见心跳才认死 → Max Age 20 秒</li>
    <li>认死之后先”听”十五拍，再”认”十五拍，才放信童 → Listening 15 秒 + Learning 15 秒（Forward Delay），学习期学 MAC 但不转发</li>
    <li>最坏要等 20 + 15 + 15 拍 → 802.1D 最坏收敛约 50 秒（常说的 30–50 秒）</li>
    <li>催烧信童让各村把旧账烧得比平时快：三个月改二十天 → TCN BPDU 触发 TC 标志，MAC 表老化时间从 300 秒缩到 Forward Delay（15 秒），持续 <code class="language-plaintext highlighter-rouge">Max Age + Forward Delay</code>（35 秒）</li>
    <li>挂红布的堤后面那位听差早就跟对面”对过话”，归路一断直接喊”我接” → RSTP 的 proposal / agreement 握手，把收敛从”等计时器”变成”换手一次”</li>
    <li>恢复只要一拍，不是二十加十五加十五 → RSTP 典型收敛亚秒到几秒，而不是 802.1D 的数十秒</li>
    <li>预先留在身边的那条挂红布的路 → alternate 端口，根端口失效时立刻顶上</li>
    <li>挂红布的堤人不能走但耳朵留着，所以知道它是死是活 → 阻塞态仍收 BPDU；这也是 Loop Guard 的依据</li>
    <li>湖东三村嫌绕远，撤了半天红布，半刻钟信童翻三倍 → 打开一个被 STP 阻塞的冗余链路即立刻成环；”堵掉一半”不是可选项</li>
    <li>同一批 VLAN 让不同的堤挂红布 → MSTP（802.1s）把 VLAN 映射到多个实例，每个实例一棵树，用阻塞链路做负载分担</li>
    <li>村外大路进来却报出比现任宗主更小的号，一律不理并封口 → Root Guard：在朝外端口上收到优于现任根的 BPDU 时置为 discarding，防止根桥被抢</li>
    <li>通到院子的门口收到心跳木牌，立刻插门喊人 → BPDU Guard：边缘口收到 BPDU 说明下面接了交换机，置 err-disable</li>
    <li>单向断了的堤：听差收不到木牌，以为上游死了，把红布撤了 → 单向链路导致阻塞口收不到 BPDU 而转入转发，Loop Guard / UDLD 把它锁死在阻塞态</li>
    <li>石堤上赶集的路越修越多，路从树变成网 → L2 域扩张、冗余链路增加，环的概率从”不可能”变成”迟早”</li>
  </ul>
</section>
<section class="en">
  <p>Nine villages ring a lake called Willow Bay. The roads between them are not water but stone causeways — the wide ones take an ox cart, the narrow ones barely let two people pass.</p>

  <p>The causeways carry <strong>letter boys</strong>. Willow Bay’s rule is old and very simple: <strong>a village that receives a notice copies it once and sends it out along every causeway except the one it came in on.</strong> Each copy is made neatly, and on the back of the envelope is written the hour the village received it.</p>

  <p>In the early years there were few causeways — one main causeway strung through five villages, two branch causeways reaching two villages each. The shape was a tree: one step to the next, from root to tip, with no second path. The rule served for sixty years without incident.</p>

  <p>Then the nine villages pooled their money, built a causeway around the whole lake, and cut three shortcuts straight across the middle, so nobody would have to walk the long way to market. The roads went from a tree to a net. Nobody looked twice at that change the year it was built.</p>

  <p>That spring, at the Grain Rain festival, the first notice was “market opens.” The first day was fine. On the second day the notice ran the entire ring causeway and came back to the village gate that had first sent it out. The old gatekeeper took the copy and did what sixty years of rule said to do — copy it once, send it along every causeway but the one it came in on. He did not recognize the notice he had sent out the previous morning.</p>

  <p><strong>There were now two copies of the same notice on the roads.</strong></p>

  <p>Third lap, four. Fourth lap, eight. By the fortieth lap both directions of all nineteen causeways were solid with letter boys carrying the same sheet of paper, meeting head-on, nobody giving way. The more villages, the slower the copying; the slower the copying, the longer each copy held the road; the longer it held the road, the more copies were on it.</p>

  <p>Old Zhou at the bridgehead later put it better than anyone:</p>

  <p><strong>The problem was not too many letters. The problem was that the envelope had no line reading “destroy after nine villages.”</strong></p>

  <p>Water-borne documents all have that line — the merchants’ pouches say plainly <em>destroy if undelivered after three post stations</em>, and the boatmen know to burn them. The letters on the causeways had nothing. Willow Bay’s rule was “delivered and done,” not “delivered or dead.” <strong>A copy could circle the ring forever, and double at every turn.</strong></p>

  <p>On the seventh day the grain boats could not tie up on the east shore. Not because the causeway was jammed, but because <strong>the quay could not accept a bill of lading</strong> — the clerk would not take grain until the “market opens” notice had settled, and the notice was permanently on the road. Rice prices in the nine villages rose three times a day. The lantern signals failed too, because the lantern signals were also carried by letter boys.</p>

  <p><strong>The first remedy was a speed limit.</strong></p>

  <p>The headman ordered: no more than four letter boys on any causeway at once, everyone else held back.</p>

  <p>The result was strange. The lake went quiet for a quarter hour, and then it got worse. The letters running hardest were exactly the ones that got through, while legitimate bills of lading stood outside the causeway and did not move at all. <strong>A speed limit guarantees the road will not collapse. It says not one word about who should be on it.</strong></p>

  <p><strong>The second remedy came from the old headman, sitting under the scholar tree on the west shore with a bamboo rule marked in graduations.</strong></p>

  <p><strong>One: name a sovereign.</strong> Each of the nine villages carves an <strong>appellation</strong> above its gate — first compare the number carved there, and if the numbers tie, compare the appellations themselves (say, Scholar Tree Village ahead of Willow Village, because scholar comes first). <strong>The village with the smallest appellation is the sovereign.</strong> It is not necessarily the strongest. It merely has the smallest name. The beauty of the rule is that <strong>no meeting is required</strong>: given your own appellation and your neighbor’s, you can compute the sovereign yourself.</p>

  <p><strong>Two: measure distance, but speak only to your neighbors.</strong> The bamboo rule gives every causeway a <strong>toll</strong>: the narrower and more winding, the higher; the wider and straighter, the cheaper. A village’s “journey toll” to the sovereign = the toll of its cheapest upstream causeway toward the sovereign + the toll its upstream neighbor announced. <strong>Each village announces exactly one number to its neighbors and listens to exactly one number from them.</strong> Nobody walks the whole lake.</p>

  <p><strong>Three: each village keeps only one way home.</strong> Of the roads leading toward the sovereign, keep the <strong>cheapest</strong> one open for traffic. Ties go to the upstream village with the smaller appellation, and if that ties, to the smaller gate number on that stretch of causeway. A village has <strong>exactly one</strong> way home.</p>

  <p><strong>Four: on any stretch between two villages, only one village is on duty.</strong> Whichever is closer to the sovereign (lower journey toll, and on a tie, the smaller appellation) is responsible for carrying letters out along that stretch, and its gate stands open; <strong>the gate at the other end hangs a red cloth.</strong></p>

  <p><strong>Five: a causeway under red cloth carries no people, but keeps its ears.</strong> A listener still stands behind the cloth, hearing the heartbeat tokens relayed from the sovereign — the sovereign strikes one every two beats. He carries no letters. But he knows whether that causeway is alive or dead. <strong>The life of the nine villages hangs on this least useful-looking job in the whole town.</strong></p>

  <p><strong>Six: wait twenty beats before calling it dead, and after that, listen first and learn second.</strong> Only after twenty consecutive beats with no heartbeat does the listener rule the upstream dead and void the way home; then he picks the best of the red-clothed causeways and makes it the new way home. Two steps before that: first <strong>listen</strong> for fifteen beats — send nothing, only hear the names and tolls on both sides; then <strong>learn</strong> for fifteen beats — start recording the roads, still send nothing; only after those fifteen may letter boys onto the causeway. It sounds fussy. It is what keeps the whole lake from <strong>burning the old ledger and misusing the new one in the same instant</strong>.</p>

  <p>Nineteen causeways: nine open, ten under red cloth. A letter from the west shore to the east shore now travels one non-repeating path; the number of copies drops from “double every lap” to “one per village.”</p>

  <p><strong>The third spring brought a second event, more worth recording than the first.</strong></p>

  <p>Three eastern villages complained that the way home was a long way round — there was a straight causeway, three li, while the way home was eleven. They talked the old headman into pulling the red cloth off that causeway for half a day.</p>

  <p><strong>A quarter hour.</strong> Letters on the lake tripled, and two gatekeepers fainted outright. The red cloth went back that same day, and nobody raised it again.</p>

  <p>Afterward the nine villages understood a sentence nobody had said aloud: <strong>the red cloth is not waste. The red cloth is the single condition under which the lake is still alive.</strong></p>

  <p><strong>In the fourth summer’s floods, the main causeway at the lake’s heart collapsed.</strong></p>

  <p>Half the lake lost the sovereign. And something worse happened in the same instant: every village’s <strong>route ledger was wrong</strong> — the ledger said <em>letters to the east fishery go by the north causeway</em>, but there was nobody at the north causeway anymore. Letter boys ran to the break, turned back, ran again, and were still stopped along the way by other letter boys asking directions.</p>

  <p>The sovereign’s response was effortless: he sent <strong>burn-order</strong> letter boys at a run, telling every village to burn its old ledger far faster than usual — where an entry normally lasted three months, for these days it lasted twenty. Misdelivered letters stopped within a quarter hour.</p>

  <p>But this time the whole lake was talking again <strong>within one beat</strong> — not twenty beats plus fifteen plus fifteen.</p>

  <p>Why? Because the listeners behind those ten red cloths had been <strong>talking to the village opposite them all along</strong> — both sides held each other’s names and tolls from the previous round. The moment the way home broke, the listener simply called out “I take it,” and the far side answered at once “you take it.” No waiting twenty beats to declare death, no fifteen to listen, no fifteen to learn. The old rule was <strong>wait until you are certain</strong>. The new rule was <strong>agree in advance</strong>. The villages later called this revision “answering the road.”</p>

  <p><strong>In the fifth year, two odd incidents were written into the clan rules.</strong></p>

  <p>The first: a wealthy house newly moved to the east shore carved a number above its gate smaller than the reigning sovereign’s, hoping all the lake’s letters would detour through his courtyard so he could take a cut. A rule was added: <strong>coming in from the outer highway, any appellation smaller than the reigning sovereign’s is ignored outright, and that gate is sealed.</strong> The sovereign is something you compute, not something you declare.</p>

  <p>The second: a lakeside household had looped itself — it had bought two courtyard gates, both leading to the main hall, and the hall copied outgoing notices to both. A heartbeat token arriving at a courtyard gate means no token should ever have been there. Another rule: <strong>a gate into someone’s courtyard is not a causeway; if a token arrives at one, bar the gate at once and shout for someone to come look.</strong> Do not let one household drag the whole lake down.</p>

  <p>By now you have probably recognized it: this is the <strong>Spanning Tree Protocol</strong> (<em>STP</em>).</p>

  <h3 id="what-it-is">What it is</h3>

  <p>Start with what it has to hold back.</p>

  <p>Ethernet forwarding is <strong>read the house number and deliver</strong>: a switch receives a frame, looks up which port holds the destination MAC, sends it there; if it does not know, it sends it out <strong>every port except the one it arrived on</strong> (unknown unicast flooding). Broadcast frames go out every port unconditionally.</p>

  <p>The entire safety of that scheme rests on the assumption that the roads form a tree. And layer 2 <strong>has no TTL</strong> — an IP header carries a hop counter that discards the packet when exhausted; an Ethernet frame carries nothing of the kind. <strong>A frame that enters a loop never dies.</strong> It only multiplies around the loop: each switch makes one copy of a broadcast frame, but ten switches in a ring means ten copies circling; add unknown-unicast flooding on top and a single lap saturates the link.</p>

  <p>Three consequences arrive together: <strong>bandwidth is consumed entirely</strong>; <strong>every switch’s MAC table flips wildly</strong> (the same source MAC alternating between two ports), so even unicast that should be forwarded precisely gets flooded; and <strong>switch CPUs are consumed by interrupts</strong>, so you cannot even reach the management plane — the channel you would use to fix the config runs over the same dead broadcast domain.</p>

  <p><strong>What STP (IEEE 802.1D) does is brutally simple: compute a tree out of a looped network, then shut the ports that are not on the tree.</strong></p>

  <p><strong>One: elect a root bridge.</strong> Every bridge has an <strong>8-byte bridge ID</strong>: a 2-byte priority (default <code class="language-plaintext highlighter-rouge">32768</code>, must be a multiple of <code class="language-plaintext highlighter-rouge">4096</code>) plus a 6-byte MAC. <strong>Lowest priority, then lowest MAC, becomes the root.</strong> Priority rather than “who is strongest” exists so a human can pin it — set it to <code class="language-plaintext highlighter-rouge">4096</code> and you are the root. Cisco’s PVST+ adds the VLAN ID into the low 12 bits of the priority (<em>sys-id-ext</em>), so one physical switch can be root of one VLAN and not another.</p>

  <p><strong>Two: BPDU, the heartbeat token.</strong> The root sends a configuration BPDU every <strong>2 seconds</strong> (Hello), to destination MAC <code class="language-plaintext highlighter-rouge">01:80:C2:00:00:00</code> (switches never forward that address, so heartbeat tokens do not consume the letter roads). Four fields matter: <strong>root bridge ID</strong>, <strong>root path cost</strong>, <strong>sender bridge ID</strong>, <strong>sender port ID</strong>. A non-root bridge rewrites the cost and the sender, then relays it downstream — that is “speak only to your neighbors, announce one number.” In 802.1D only the root <strong>originates</strong> BPDUs and everyone else relays; in RSTP every bridge sends its own every 2 seconds.</p>

  <p><strong>Three: port roles and blocking.</strong> Every non-root bridge elects <strong>one root port</strong>: lowest path cost to the root, ties to lowest sender bridge ID, then lowest sender port ID. On every link (segment) <strong>one designated port</strong> is chosen: the bridge on that segment closest to the root handles outbound traffic, ties to lowest bridge ID. <strong>Whatever is left is a blocked port</strong> (<em>alternate</em> / <em>backup</em>) — it <strong>forwards no data frames, but still receives and parses BPDUs</strong>. That is the fulcrum of the whole design: <strong>blocked does not mean powered off. Blocked means shut up and listen.</strong> The listener must keep knowing whether that causeway is alive.</p>

  <p>Default costs (802.1t long form): <code class="language-plaintext highlighter-rouge">20,000,000 / speed in Mbps</code> — 10M is 2,000,000; 100M is 200,000; 1G is 20,000; 10G is 2,000. The old 802.1D-1998 short form was 10M=100, 100M=19, 1G=4, 10G=2 — so coarse that 10G and 100G are both 2 and <strong>cannot be told apart</strong>, which is why modern networks must use long costs.</p>

  <p><strong>Four: states and timers.</strong> 802.1D has five states: <code class="language-plaintext highlighter-rouge">Disabled</code>, <code class="language-plaintext highlighter-rouge">Blocking</code>, <code class="language-plaintext highlighter-rouge">Listening</code> (15 s), <code class="language-plaintext highlighter-rouge">Learning</code> (15 s), <code class="language-plaintext highlighter-rouge">Forwarding</code>. A port that has decided to open still spends <strong>Forward Delay = 15 seconds</strong> listening and 15 learning before it carries data; declaring the upstream dead takes <strong>Max Age = 20 seconds</strong>. Worst-case convergence is therefore <strong>20 + 15 + 15 = 50 seconds</strong> (the familiar 30–50 s figure). The difference between listening and learning: while listening, not even MAC addresses are learned, because the topology is not settled and anything learned would be wrong; while learning, addresses are learned but nothing is forwarded.</p>

  <p><strong>Five: when the topology changes, the whole lake burns its old ledger.</strong> A bridge detecting a link change sends a <strong>TCN BPDU</strong> toward the root; the root sets the <strong>TC flag</strong>, and for <code class="language-plaintext highlighter-rouge">Max Age + Forward Delay</code> (35 seconds) every bridge drops its MAC aging time from the default <strong>300 seconds</strong> to <strong>15 seconds</strong>. That is the burn order. Without it, traffic keeps being sent toward a direction that no longer exists — and at layer 2 it is worse than an IP loop, because there is no TTL to stop it; it continues until something flushes it.</p>

  <p><strong>Six: RSTP turns “wait until certain” into “agree in advance.”</strong> IEEE 802.1w (folded into 802.1D-2004) collapses five states into three (<code class="language-plaintext highlighter-rouge">Discarding</code> / <code class="language-plaintext highlighter-rouge">Learning</code> / <code class="language-plaintext highlighter-rouge">Forwarding</code>), expands roles to four (root, designated, <strong>alternate</strong>, <strong>backup</strong>), and adds a <strong>proposal/agreement</strong> handshake: a port wanting to forward immediately sends a BPDU with the proposal bit set; the downstream bridge synchronizes all its non-edge ports to discarding and answers with agreement, and this port goes straight to forwarding. <strong>No two 15-second waits</strong>, so typical convergence is <strong>sub-second to a few seconds</strong>. It also puts <strong>edge ports</strong> (those facing end hosts) into forwarding immediately and keeps an alternate port warm by its side — when the root port dies, the alternate takes over at once.</p>

  <p><strong>Seven: MSTP, because one tree is not enough.</strong> IEEE 802.1s maps VLANs onto multiple <strong>instances</strong>, each with an independent tree. So VLAN 10 can block this link while VLAN 20 blocks that one — <strong>the link STP condemned is alive for half your traffic.</strong> This is the first formal answer to the complaint that red cloth wastes bandwidth.</p>

  <h3 id="why-it-matters">Why it matters</h3>

  <p><strong>One, a layer-2 loop is not slow, it is lethal within seconds, and it looks like something else.</strong> An IP routing loop is stopped by TTL; worst case you lose a few dozen packets. An Ethernet loop has no TTL, and a frame can circle until morning. The symptoms are <strong>broadcast rate instantly at line rate, switch CPU at 100%, MAC table flapping between two ports, loss across the entire domain</strong> — and the SSH session you would use to inspect it runs over that same dead broadcast domain. <strong>At layer 2, you may have closed your only rescue route yourself.</strong></p>

  <p><strong>Two, convergence time is not a performance number; it decides whether you dare build dual uplinks.</strong> The 30–50 seconds of 802.1D is long enough to drop an OSPF/BGP adjacency entirely, to complete a database failover, and to time out thousands of TCP connections. That is why production runs RSTP/MSTP almost everywhere, and why “active/active uplinks” at layer 2 always carries a certain nervousness.</p>

  <p><strong>Three, STP’s real problem is not that it converges slowly; it is that it must block half the links.</strong> A tree cannot use the bandwidth of a ring. That is not an implementation defect, it is the definition of a tree. So modern data centers do not tune STP faster, they <strong>stop letting layer 2 form loops at all</strong>: shrink the L2 domain to the cabinet (“L2 no further than the access switch”), run every uplink as layer 3, and use <strong>ECMP</strong> to spend all links at once (routing has TTL, so a loop cannot kill you); or use <strong>MLAG / vPC / EVPN multihoming</strong> so both uplinks forward simultaneously; or put layer-2 semantics on top of a layer-3 fabric with <strong>EVPN-VXLAN</strong> and let the underlay converge by routing. <strong>Getting from “block half” to “use all” has never been achievable by making STP faster.</strong></p>

  <p><strong>Four, the overwhelming majority of STP incidents are misconfiguration, not protocol defect.</strong> Three highest-frequency causes: <strong>the root gets stolen by some random access switch</strong> (nobody set a priority, and a new box shipped at <code class="language-plaintext highlighter-rouge">32768</code> with a lower MAC) → the whole network re-routes through one 1G access link. <strong>An edge port gets a small switch plugged into it, and the small switch loops itself</strong> → a broadcast storm grows out of one cabinet. <strong>A unidirectional link</strong> (fibre broken in one direction only) leaves a blocked port receiving no BPDUs, so it concludes the upstream is dead and opens itself → the loop is back, and this time nobody knows why. Hence three defenses: <strong>Root Guard</strong> (on outward-facing ports, a superior BPDU puts the port in discarding — nobody gets to usurp the root), <strong>BPDU Guard</strong> (on edge ports, any BPDU means a switch is not supposed to be there — err-disable immediately), and <strong>Loop Guard / UDLD</strong> (a blocked port that stops receiving BPDUs is pinned in blocking instead of being opened).</p>

  <p><strong>Five, the design philosophy is worth copying: choose a leader by the smallest name, not the strongest.</strong> The root is not elected, it is <strong>computed</strong> — as long as each bridge knows its own ID and its neighbors’, the whole network necessarily computes the same root, with no consensus protocol, no voting, no majority. This is the cheapest kind of election in distributed systems: <strong>collapse the decision into a total order, then let every node independently take the same argmin.</strong> Its cost is equally clear: <strong>there is no defense against lying</strong> — anyone announcing a smaller number usurps the root. STP patches that with Root Guard at the boundary rather than in the protocol.</p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li>A notice copied to every causeway but the one it arrived on; with a ring, the same notice doubles forever → switches flood unknown unicast and broadcast out every port but the ingress; a layer-2 loop means a broadcast storm</li>
    <li>The envelope has no “destroy after nine villages” line → Ethernet frames have no TTL, so a frame in a loop never dies on its own</li>
    <li>Four causeways solid in both directions, letter boys meeting head-on, nobody giving way → frames replicate themselves at line rate around the loop, saturating the links</li>
    <li>The more villages, the slower the copying; the slower the copying, the longer each copy holds the road → flooding scales with topology size and eats switch CPU and interrupts</li>
    <li>Each village’s route ledger names a direction → the switch MAC address table (CAM table)</li>
    <li>Letter boys run to the break, turn back, run again → stale MAC entries send traffic toward a direction that no longer exists</li>
    <li>The carved number above the gate: compare numbers, then names → the 8-byte bridge ID: 2-byte priority (default 32768, multiple of 4096) plus 6-byte MAC; lower priority wins, then lower MAC</li>
    <li>The smallest appellation is sovereign, no meeting needed, computable by anyone → root bridge election: every node independently computes the same minimum bridge ID; no consensus protocol required</li>
    <li>The bamboo rule’s toll: narrower and more winding costs more → port path cost, 802.1t <code class="language-plaintext highlighter-rouge">20,000,000 / Mbps</code> (1G = 20,000)</li>
    <li>Each village announces one number to its neighbors and hears one back → the root path cost field in the BPDU; non-root bridges add local cost and re-announce</li>
    <li>Heartbeat tokens every two beats, struck by the sovereign → configuration BPDUs, Hello 2 s, originated by the root</li>
    <li>Tokens travel a reserved channel that does not consume the letter roads → BPDU destination MAC <code class="language-plaintext highlighter-rouge">01:80:C2:00:00:00</code>, never forwarded by switches</li>
    <li>Each village keeps only its cheapest way home → exactly one root port per non-root bridge</li>
    <li>On any stretch, the village closer to the sovereign is on duty → one designated port per segment, held by the bridge closest to the root (tie: lower bridge ID)</li>
    <li>The gate at the other end hangs red cloth → a blocked port (<em>alternate</em> / <em>backup</em>) forwards no data</li>
    <li>A listener still stands behind the red cloth → blocked ports still receive and process BPDUs; that is exactly why they can take over instantly</li>
    <li>Twenty beats with no heartbeat before calling it dead → Max Age, 20 seconds</li>
    <li>After death, listen fifteen beats, then learn fifteen, then send → Listening 15 s + Learning 15 s (Forward Delay); learning populates the MAC table without forwarding</li>
    <li>Worst case is 20 + 15 + 15 beats → 802.1D worst-case convergence near 50 seconds (the familiar 30–50 s)</li>
    <li>Burn-order boys shorten the ledger’s life from three months to twenty days → TCN BPDU sets the TC flag; MAC aging drops from 300 s to Forward Delay (15 s) for <code class="language-plaintext highlighter-rouge">Max Age + Forward Delay</code> (35 s)</li>
    <li>Listeners behind the red cloth had already “talked” to the far side and call “I take it” the instant the way home breaks → the RSTP proposal/agreement handshake turns convergence from waiting on timers into one exchange</li>
    <li>Recovery in one beat rather than twenty plus fifteen plus fifteen → RSTP converges in sub-second to a few seconds instead of tens of seconds</li>
    <li>The red-clothed road kept warm at one’s side → the alternate port, ready to take over the moment the root port fails</li>
    <li>Red cloth stops people but keeps listening, so its health is known → blocked ports keep receiving BPDUs; this is also what Loop Guard relies on</li>
    <li>Three eastern villages pull the red cloth for half a day and letters triple in a quarter hour → unblocking one STP-blocked redundant link creates a loop immediately; “block half” is not optional</li>
    <li>Different causeways under red cloth for different batches of villages → MSTP (802.1s) maps VLANs to instances, one tree each, so blocked links carry the other half of the traffic</li>
    <li>An appellation smaller than the reigning sovereign’s arriving from the outer highway is ignored and the gate sealed → Root Guard puts an outward port in discarding on a superior BPDU, preventing root usurpation</li>
    <li>A heartbeat token at a courtyard gate means bar the gate and shout → BPDU Guard err-disables an edge port that receives a BPDU, because a switch does not belong there</li>
    <li>A causeway broken in one direction only: the listener hears nothing and pulls the red cloth → a unidirectional link starves a blocked port of BPDUs until it forwards; Loop Guard / UDLD pins it in blocking</li>
    <li>More and more market roads until the shape goes from tree to net → L2 domain growth and added redundant links turn loops from impossible into inevitable</li>
  </ul>
</section>]]></content><author><name></name></author><category term="networking" /><category term="distributed-systems" /><summary type="html"><![CDATA[柳湾九村环着一片湖。九村之间的路不是水路，是石堤——宽的石堤能过一辆牛车，窄的只容两人错身。]]></summary></entry><entry><title type="html">正本斋的三支笔 · Three Brushes in the Scriptorium</title><link href="https://fables.day/2026/11/10/kafka-isr-high-watermark-leader-epoch/" rel="alternate" type="text/html" title="正本斋的三支笔 · Three Brushes in the Scriptorium" /><published>2026-11-10T00:00:00+00:00</published><updated>2026-11-10T00:00:00+00:00</updated><id>https://fables.day/2026/11/10/kafka-isr-high-watermark-leader-epoch</id><content type="html" xml:base="https://fables.day/2026/11/10/kafka-isr-high-watermark-leader-epoch/"><![CDATA[<section class="zh">
  <p>盐课司后院有一间没有窗的屋子，叫正本斋。</p>

  <p>屋里只有一卷长纸，叫<strong>正本</strong>。纸从东墙的轴子上放出来，铺过三张长案，最后垂进西墙的铜槽。盐引、拨兑、罚没、补录，全都只在这卷纸上写，从尾巴往上写，一笔接一笔。不能插队，不能改前面的字。</p>

  <p>写字的人只有一个，姓韩，人叫韩先生。他一天写四万笔。</p>

  <p>案子后面隔一道纱屏，屏后坐着三个誊录手：陈五、阿泰，和一个不到二十的吕七。他们不写新账，只做一件事——<strong>顺着正本，一笔一笔往自己的抄本上誊。</strong> 每人一支笔、一盏油灯、一枚小铜镇纸。镇纸压在”我已经誊到哪儿了”的那一行上。那一行，斋里的人叫<strong>笔头</strong>。</p>

  <p>查账的人不进这间屋子。他们在前院柜台上查，但柜台上有一条铁规矩：<strong>只能查到封印线以上。</strong></p>

  <p>封印线是许掌印手里那枚章盖的位置。许掌印不誊账，不写账，一辈子只做一件事：盖印。规矩是他父亲传下来的——<strong>印，盖在三个誊录手笔头里最低的那一行的下一行。</strong></p>

  <p>为什么盖在那儿？</p>

  <p>因为笔头以下是三个人手里都确实有的账；笔头以上，只有韩先生一个人有。韩先生若今夜暴毙，斋里要从屏后挑一个人来接他的笔，而接的人只接得住自己抄到的地方。印盖在最低的那个笔头下面，等于对天下人宣布：<strong>这行以上的账，换谁接笔都还在。</strong></p>

  <p>盐商交引要回执。回执也照这条线开：韩先生写完不算数，三个誊录手的笔头都过了那一行，才盖印、才开回执。</p>

  <p>那年八月，吕七的油灯漏了。</p>

  <p>他不肯喊人来换，怕耽误抄账，就着半盏灯抄。灯不亮，抄得慢。韩先生写十行，他抄六行。三天下来，他的笔头落后了四千行。</p>

  <p>封印线立刻不动了。</p>

  <p>不是许掌印不肯盖，是那条规矩把印死死按住——最低的那个笔头就是吕七的笔头。韩先生照写，纸照长，可印一行也上不去。前院柜台上，六百个盐商排着队等回执，队伍从柜台一直排到大门外的槐树下。</p>

  <p>许掌印没有先换灯。他先做的是另一件事：<strong>把吕七的名字，从屏后那块”在位名册”上划掉了。</strong></p>

  <p>名册上的规矩更冷：一个誊录手，若<strong>一炷香之内笔头没往前挪过一行</strong>，就不算在位。吕七的灯暗了三天，早就不算。名册上剩下陈五和阿泰，封印线随即跳到阿泰的笔头，印重新动起来，柜台重开。</p>

  <p>被划出名册的人不许走。吕七换了新灯，人还坐在屏后，接着抄。他抄得比以前还整齐，一路追到韩先生的笔尾，然后一笔一笔跟着往下抄。可许掌印的印，不等他。</p>

  <p>他抄下的那些行，<strong>印一次也没盖过</strong>。</p>

  <p>第二年开春，韩先生死在案上，笔还压在最后一行。</p>

  <p>许掌印当场要挑一个接笔的人。规矩第三条：<strong>接笔的人只能从在位名册里挑，挑笔头最长的那一个。</strong></p>

  <p>名册上是陈五和阿泰。陈五的笔头长，陈五接了笔。</p>

  <p>这条规矩不只为”最长”两个字。名册里的人，笔头一定在封印线以上。所以不管挑谁，<strong>封印线都不会往后退一行</strong>，前一天开出去的回执也不会第二天变成假账。</p>

  <p>新韩先生上任头一件事，是把正本卷首那行小字改了。原来写着”第六任，起于第八千一百一十二行”，他添成”第七任，起于第八千一百一十二行”。字稍大，用朱砂。</p>

  <p>这是许家唯一一条被外人笑话过的规矩：<strong>每一个接笔的人，都要在任期开头留一道任签</strong>，写清自己是第几任、从哪一行起。老许掌印的原话是：”将来总有人拿着尾巴来问你，那时候你别跟他算印，让他自己看签。”</p>

  <p>吕七就是那年三月回来的。抄本誊到了第九千四百行——比新韩先生的笔尾长出二百八十行。</p>

  <p>那二百八十行是老韩先生临死前写的：盐引三十七张，罚没两笔。印从没盖过。陈五和阿泰的抄本上没有。新韩先生的笔尾，就停在那二百八十行之前。</p>

  <p>吕七抱着抄本去找许掌印：”这二百八十行是真的。字是老韩先生写的，我亲眼看他落笔。”</p>

  <p>许掌印说：”是真的，也不算。印没盖过，名册上没有第二个人有这二百八十行。老韩先生写完那三十七张引，当场就去了——那三十七张引的盐，一担都没出过库。留在册上，就是让后人对着一笔从未发生的兑付查账。”</p>

  <p>于是他让吕七做一件事：<strong>把抄本上多出来的部分，撕掉。</strong></p>

  <p>撕到哪儿？</p>

  <p>第一个办法，是照自己抄本上的封印线撕。这个办法会把好账撕坏。</p>

  <p>因为印是许掌印盖的，传到吕七手里有时差。他被划出名册那三天，印已经往前跳了一大截；而他抄本上记着的印，还停在老地方。照它撕，就撕得不够——剩下的那截账跟新正本对不上，明天还得再撕一次。若碰上印传得过快，又会撕得太多：一笔已经开过回执的账被他亲手撕了，柜台查得到，抄本上没了。</p>

  <p>那半个月，正本斋一天来回撕抄十几遍，越撕越乱。陈五说，屋子里像有两个人各拿一把刀，谁也不肯先停手。</p>

  <p>第二个办法，是许掌印后来才立成文的一条：<strong>不比印，比任签。</strong></p>

  <p>两个人的抄本上都抄着每卷开头那道朱砂签。吕七翻到自己抄本上最后一处和新正本对得上的签——”第六任，起于第八千一百一十二行”。新韩先生身上也有这一签，位置一样。</p>

  <p>那就什么也不用算了。<strong>这一签之后的所有行，全撕。</strong></p>

  <p>撕完正正好好停在新韩先生笔尾的前一行。一行不多，一行不少。撕一次，永不回头。</p>

  <p>这套办法能一次了结，是因为任签是<strong>接笔人自己写的、只能往上加的号</strong>，不是谁的观察记录。号是几，就从第几任之后撕，答案只有一个。</p>

  <p>那年冬天，还出过一桩事。</p>

  <p>连下半月雨，库房漏水，屏后三张案全泡了。名册上的人一个都誊不了。柜台上压着六百张回执，商会的人在外面砸门。</p>

  <p>许掌印等了一夜，等到天亮，三个笔头一个也没动。</p>

  <p>他开了祖上传下来的一条口子——<strong>从已经划出名册的人里，挑一个来接笔。</strong></p>

  <p>他挑的是当时笔头最长的那一个，恰恰是当年被划出去的人。理由是急：隔着屏的人已经十天没抄过账，可他有几百行没盖过印的账，全须全尾地揣在怀里。</p>

  <p>新正本一立，封印线往后退了。</p>

  <p>第二天，前院那些领了回执的盐商来查账，自己那一笔没了。不是被人涂掉，是从没在官册上存在过。盐已经出了库，回执上是朱红印，官册上是一片空白。</p>

  <p>档册里管这件事叫”三百张回执案”。许掌印后来把一个铜楔钉进了那条口子的门框上，谁都拔不下来。他的说法很短：</p>

  <p><strong>宁可柜台停三天，不要官册后退一行。</strong></p>

  <p>到这里你大概已经认出来了：这就是 <strong>Kafka 的副本复制——ISR、high watermark 和 leader epoch</strong>。</p>

  <h3 id="section">这是什么</h3>

  <p>Kafka 的一个 partition 就是那卷正本：<strong>只能追加，不能改写，每一笔有一个严格递增的行号（offset）。</strong></p>

  <p>同一时刻只有一个副本接受写，叫 <em>leader</em>（韩先生）。其余副本叫 <em>follower</em>（陈五、阿泰、吕七），它们发 <code class="language-plaintext highlighter-rouge">Fetch</code> 请求顺着 leader 的 log 往前拉，<strong>只读不写</strong>。</p>

  <ul>
    <li><strong>LEO（log end offset）</strong>：每个副本自己写到/抄到哪一行，即笔头。</li>
    <li><strong>HW（high watermark）</strong>：ISR 中所有副本 LEO 的最小值，即封印线。消费者只能读到 HW 之前的消息。</li>
    <li><strong>ISR（in-sync replicas）</strong>：跟得上的副本集合。判据是 <code class="language-plaintext highlighter-rouge">replica.lag.time.max.ms</code>（默认 30000 ms）——follower 在这个窗口内没有把 fetch 推进到 leader 的 LEO，就被踢出 ISR。注意它是<strong>按时间</strong>判的，不是按落后多少条：吞吐不同的 partition 之间，”落后多少条”没有可比性。</li>
    <li><strong>leader epoch</strong>：leader 每换届一次，新任 leader 在自己的 log 里落一个 <code class="language-plaintext highlighter-rouge">(epoch, startOffset)</code> 标记。这就是朱砂任签。KIP-101 之后，follower 重连时不再用自己本地的 HW 去截断，而是发 <code class="language-plaintext highlighter-rouge">OffsetForLeaderEpoch</code>，双方找出<strong>最后一段 epoch 相同的区段</strong>，一次性 <code class="language-plaintext highlighter-rouge">truncate</code> 到那个区段的边界。</li>
  </ul>

  <p><strong>为什么消费者只能读到 HW。</strong> 因为 leader 随时会死，而可见的消息必须满足”换 leader 也不会消失”。HW 恰好是这个下界：所有 ISR 成员都追过了的位置。所以 leader 只能从 ISR 里选（<em>clean leader election</em>），新 leader 的 LEO ≥ HW，HW 只增不减，已开的回执不会变成假账。</p>

  <p><strong>ISR 的收缩与扩张不对称。</strong> 收缩让 HW 继续前进（这是拿一致性换可用性，也是唯一让柜台重开的办法）；扩张要求 follower 先追上 HW——不是追上 LEO——所以一个副本重新回到 ISR，不会把 HW 拖回去。</p>

  <p><strong>三个把这条机制接到业务上的旋钮</strong>：</p>
  <ul>
    <li><code class="language-plaintext highlighter-rouge">acks=all</code> + <code class="language-plaintext highlighter-rouge">min.insync.replicas=N</code>：producer 只有在至少 N 个 ISR 副本追加成功后才收到 ack；ISR 数小于 N 时<strong>拒写</strong>，报 <code class="language-plaintext highlighter-rouge">NotEnoughReplicas</code>。这是”名册少于两人，韩先生停笔”。</li>
    <li><code class="language-plaintext highlighter-rouge">unclean.leader.election.enable</code>：默认 <code class="language-plaintext highlighter-rouge">false</code>。设成 <code class="language-plaintext highlighter-rouge">true</code>，允许从 ISR 之外选 leader——封印线后退，已 ack 的消息消失，就是”三百张回执案”。</li>
    <li><code class="language-plaintext highlighter-rouge">isolation.level=read_committed</code>：消费者只读到 HW；<code class="language-plaintext highlighter-rouge">read_uncommitted</code> 读到 leader 的 LEO，就会看见那些明天可能被撕掉的行。</li>
  </ul>

  <h3 id="section-1">为什么重要</h3>

  <p><strong>一、”提交”不是抽象概念，是一个可以写下来的式子。</strong> 在 Kafka 里它就是 <code class="language-plaintext highlighter-rouge">committed = min(LEO of every ISR member)</code>。任何多副本系统都得回答同一个问题：一条记录在什么时刻才算数？答案从来不是”写成功了”，而是”<strong>哪些副本上已经有了，且换谁当家都不会没</strong>“。把这句话变成公式，你才看得出成本在哪：HW 的前进速度等于<strong>最慢的那个 ISR 成员</strong>的前进速度。一块慢盘可以把一整个 partition 的读可见性按在地上——这就是 ISR 收缩在运维上属于”必要的残忍”的原因。</p>

  <p><strong>二、分歧的裁定标准必须来自权威的单调序列，不能来自各方的本地观察。</strong> 这一条超出 Kafka：</p>

  <ul>
    <li>Raft 用 <em>term</em>，Paxos 用 <em>ballot</em>，Kafka 用 <em>leader epoch</em>，etcd 用 <em>revision</em>。</li>
    <li>让 follower 拿自己本地的 HW 去决定截多少（Kafka 0.11 之前的做法），会得到震荡：截少了不收敛，截多了撕掉已提交的数据，于是截断—重拉—再截断，即 <em>truncation storm</em>。</li>
    <li>用 leader 的任期去决定，则一次截断、确定收敛。吕七撕一次就不再回头，原因不在他更聪明，在于他问的是一个<strong>只能往上加的号</strong>。</li>
  </ul>

  <p>这条规律在别处一样有效：Kubernetes 的 <code class="language-plaintext highlighter-rouge">resourceVersion</code>、Terraform 的 state serial、数据库的 LSN，都是”用权威的单调数判定分歧，而不是用观察者的猜测判定分歧”。</p>

  <p><strong>三、可用性与正确性在这里是一个显式的开关，而且它会静默失败。</strong> <code class="language-plaintext highlighter-rouge">unclean.leader.election.enable=true</code> 不会报错，不会告警，它只是让历史被改写。反过来说，<code class="language-plaintext highlighter-rouge">min.insync.replicas=2</code> + <code class="language-plaintext highlighter-rouge">acks=all</code> 的代价是：只要 ISR 掉到 1，写入就会<strong>停下来</strong>。选哪一个不是技术问题，是业务问题——但你必须知道自己在选，因为默认值替你选了一次。</p>

  <p><strong>四、该盯的指标很具体，而且能被翻译成一句话。</strong></p>

  <ul>
    <li><code class="language-plaintext highlighter-rouge">UnderReplicatedPartitions</code> &gt; 0：有副本被划出名册了。</li>
    <li><code class="language-plaintext highlighter-rouge">UnderMinIsrPartitions</code> &gt; 0：名册人数不够，<strong>此刻写入正在被拒</strong>。这个值持续不为 0，等于你的可用性是假的——客户端拿不到 ack。</li>
    <li><code class="language-plaintext highlighter-rouge">MaxLag</code> 与 <code class="language-plaintext highlighter-rouge">isr-shrink</code>/<code class="language-plaintext highlighter-rouge">isr-expand</code> 事件率：一个副本反复进出 ISR，说明它在临界点上抖，而每一次收缩都在放大 HW 的延迟。</li>
    <li>消费者侧看 <code class="language-plaintext highlighter-rouge">records-lag-max</code> 与 <code class="language-plaintext highlighter-rouge">log-end-offset - high-watermark</code>：这个差就是”已经写进 Kafka、但还看不到”的部分，它由最慢的副本决定，不由你决定。</li>
  </ul>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li>正本斋只有一卷长纸，从尾巴往上写，不能插队、不能改前面的字 → 一个 partition 是一条只能追加的 log，消息按 offset 严格有序</li>
    <li>一行账 = 一笔盐引 → 一条消息 + 一个 offset</li>
    <li>韩先生一个人写新账 → 同一时刻只有一个 <em>leader</em> 副本接受写</li>
    <li>屏后的陈五、阿泰、吕七只顺着正本抄，从不写新账 → <em>follower</em> 副本通过 <code class="language-plaintext highlighter-rouge">Fetch</code> 从 leader 拉取，只读不写</li>
    <li>铜镇纸压着”我已经誊到哪儿了”的那一行 → 每个副本的 <strong>LEO</strong>（log end offset）</li>
    <li>封印线 = 三个笔头里最低的那一行的下一行 → <strong>HW</strong> = ISR 中所有副本 LEO 的最小值</li>
    <li>柜台只能查到封印线以上 → 消费者只能读到 HW 之前的消息</li>
    <li>韩先生写完不算数，三个笔头都过了才开回执 → <code class="language-plaintext highlighter-rouge">acks=all</code>：所有 ISR 副本都追加后才向 producer 确认</li>
    <li>印一旦盖下就不再收回，已开的回执不会变假账 → 只要 leader 从 ISR 中产生，HW 单调不减，已提交的消息不会消失</li>
    <li>一炷香之内笔头没挪过一行，就划出在位名册 → <code class="language-plaintext highlighter-rouge">replica.lag.time.max.ms</code>（默认 30s）为判据的 ISR 收缩</li>
    <li>判据是”一炷香”而不是”落后多少行” → ISR 判据基于时间而非消息条数，吞吐不同的 partition 之间才有可比性</li>
    <li>吕七的灯漏了，三天抄得比人慢，封印线就此不动，柜台排到槐树下 → 一个慢副本把整个 partition 的 HW 拖住，读可见性随最慢的 ISR 成员前进</li>
    <li>划掉吕七之后印立刻往前跳，柜台重开 → ISR 收缩解除 HW 的阻塞，用一致性换回可用性</li>
    <li>被划掉的人可以回来接着抄，但印不等他 → 掉出 ISR 的副本仍在复制，但它的进度不影响 HW</li>
    <li>重新回到名册要先追到封印线，而不是追到笔尾 → 一个副本重新进入 ISR 需先追上 HW，因此不会把 HW 拖回去</li>
    <li>接笔的人只能从在位名册里挑 → <em>clean leader election</em>，新 leader 只从 ISR 产生</li>
    <li>挑名册里笔头最长的那一个 → 新 leader 取 ISR 中 LEO 最靠前的副本，保证 LEO ≥ HW</li>
    <li>卷首那道朱砂任签：”第七任，起于第八千一百一十二行” → <strong>leader epoch</strong>，<code class="language-plaintext highlighter-rouge">(epoch, startOffset)</code> 标记，随每次换届单调递增</li>
    <li>吕七抄本上比新正本多出二百八十行没盖过印的账 → follower 的 log 尾部超出 HW 的未提交消息</li>
    <li>撕掉多出来的部分 → <strong>log truncation</strong></li>
    <li>照自己抄本上的封印线撕，撕少了不收敛、撕多了撕掉已提交的账，一天来回十几遍 → 旧协议按本地 HW 截断导致的 <em>truncation storm</em></li>
    <li>不比印，比任签：找到最后一处相同的朱砂签，之后全撕 → <code class="language-plaintext highlighter-rouge">OffsetForLeaderEpoch</code>：以双方共享的最大 epoch 边界做截断点</li>
    <li>撕一次就永不回头 → epoch 单调递增，截断点唯一且幂等，一次收敛</li>
    <li>任签由接笔的人自己写、只能往上加 → epoch 由 leader 的任期决定，不由 follower 的本地观察决定</li>
    <li>库里漏水，名册上无人可誊，许掌印从划掉的人里挑一个接笔 → <code class="language-plaintext highlighter-rouge">unclean.leader.election.enable=true</code></li>
    <li>新正本一立，封印线往后退，三百张回执变成空白 → HW 后退，已 ack 的消息丢失，历史被静默改写</li>
    <li>名册少于两人，韩先生停笔，门上挂”暂不收引” → <code class="language-plaintext highlighter-rouge">min.insync.replicas</code> 未满足时拒写，报 <code class="language-plaintext highlighter-rouge">NotEnoughReplicas</code></li>
    <li>柜台有两种查法：只查到印，或查到韩先生笔尾 → <code class="language-plaintext highlighter-rouge">isolation.level=read_committed</code> 与 <code class="language-plaintext highlighter-rouge">read_uncommitted</code></li>
    <li>急性子查到的账第二天可能没了 → 读到 HW 之后的消息，就是读到了可能被截断的部分</li>
    <li>门框上钉死的铜楔：”宁可柜台停三天，不要官册后退一行” → 默认 <code class="language-plaintext highlighter-rouge">unclean.leader.election.enable=false</code>，正确性优先于可用性</li>
  </ul>
</section>
<section class="en">
  <p>Behind the salt bureau there is a windowless room called the Master Scriptorium.</p>

  <p>In it there is exactly one long scroll, called <strong>the Master</strong>. The paper runs off a roller on the east wall, across three tables, and down into a bronze trough in the west wall. Salt warrants, transfers, fines, supplementary entries — all of it is written on that one scroll, from the tail upward, entry after entry. Nobody jumps the line. Nobody alters a character already written.</p>

  <p>Only one man writes: his surname is Han, and everyone calls him Master Han. He writes forty thousand entries a day.</p>

  <p>Behind a gauze screen sit three copyists: Chen the Fifth, a boy called Tai, and Lü the Seventh, not yet twenty. They write no new entries. They do one thing — <strong>they follow the Master and transcribe it, entry by entry, onto their own copies.</strong> Each has a brush, an oil lamp, and a small bronze paperweight. The weight rests on the line that says <em>this is how far I have copied</em>. In that room, that line is called the <strong>brush-head</strong>.</p>

  <p>Auditors do not enter the room. They work at the counter in the front courtyard, from the copies. But the counter has one iron rule: <strong>you may read only up to the seal line.</strong></p>

  <p>The seal line is wherever Seal-Keeper Xu has pressed his stamp. Xu copies nothing and writes nothing; all his life he has done one thing: press the seal. The rule came from his father — <strong>the seal goes on the line just below the lowest brush-head of the three copyists.</strong></p>

  <p>Why there?</p>

  <p>Because below the brush-heads, all three men genuinely hold the entry. Above them, only Master Han holds it. If Master Han dies tonight, the room must pick someone from behind the screen to take up his brush, and that person can only reach as far as he has copied. Pressing the seal below the lowest brush-head declares to the world: <strong>every entry above this line survives whoever takes the brush.</strong></p>

  <p>A salt merchant who presents a warrant needs a receipt. The receipt follows the same line: Master Han’s writing does not count by itself. Only when all three brush-heads have passed that line is the seal pressed and the receipt issued.</p>

  <p>In the eighth month of that year, Lü the Seventh’s lamp sprang a leak.</p>

  <p>He would not call anyone to change it — he was afraid of losing copying time — so he worked by half a flame. Half a flame is slow. For every ten lines Master Han wrote, Lü copied six. Within three days his brush-head had fallen four thousand lines behind.</p>

  <p>The seal line stopped dead.</p>

  <p>Not because Xu refused to press. The rule held the seal down: the lowest brush-head was Lü’s. Master Han kept writing, the paper kept growing, and the seal would not move a single line. At the front counter, six hundred salt merchants queued for receipts, from the counter out the gate and under the locust tree.</p>

  <p>Xu did not change the lamp first. He did something else: <strong>he struck Lü’s name off the roster of those in place</strong> behind the screen.</p>

  <p>The roster’s rule is colder still: a copyist whose brush-head has not advanced by one line <strong>within the time it takes to burn one stick of incense</strong> is no longer in place. Lü’s lamp had been dim for three days, so he had long since ceased to be. The roster now held Chen the Fifth and Tai. The seal line jumped immediately to Tai’s brush-head, the seal moved again, and the counter reopened.</p>

  <p>A man struck off the roster is not sent away. Lü changed his lamp and stayed behind the screen, copying. He copied more neatly than before, caught up to Master Han’s tail, and then followed it down line by line. But Xu’s seal did not wait for him.</p>

  <p>Not one of the lines Lü copied ever carried a seal.</p>

  <p>The following spring, Master Han died at his table, his brush still weighing down the last line.</p>

  <p>Xu had to choose a successor that same hour. The third rule: <strong>the successor must come from the roster of those in place, and must be the one with the longest brush-head.</strong></p>

  <p>The roster held Chen the Fifth and Tai. Chen’s brush-head was longer, so Chen took the brush.</p>

  <p>The rule was not only about length. Everyone on the roster has a brush-head above the seal line, so whoever is chosen, <strong>the seal line cannot move backward by one line</strong>, and yesterday’s receipts cannot become tomorrow’s forgeries.</p>

  <p>The new Master Han’s first act was to amend the small line at the head of the scroll. It read <em>Sixth tenure, beginning at line eight thousand one hundred twelve</em>; he made it <em>Seventh tenure, beginning at line eight thousand one hundred twelve</em>. Slightly larger characters, in vermillion.</p>

  <p>It is the one Xu family rule that outsiders have always laughed at: <strong>every man who takes up the brush leaves a tenure mark at the start of his term</strong>, stating which tenure he is and which line he begins at. Old Xu’s own words: “Someday a man will come to you holding a tail and asking questions. Do not argue with him about the seal. Let him read the marks.”</p>

  <p>Lü came back in the third month of that year. His copy had reached line nine thousand four hundred — two hundred eighty lines beyond the new Master Han’s tail.</p>

  <p>Those two hundred eighty lines were written by the old Master Han in the hour before he died: thirty-seven salt warrants, two fines. No seal had ever touched them. Chen’s copy did not have them. Tai’s did not. The new Master Han’s tail stopped just short of them.</p>

  <p>Lü carried his copy to Xu: “These two hundred eighty lines are real. The old Master Han wrote them. I watched his brush move.”</p>

  <p>Xu said: “Real, and still not counted. No seal touched them, and no second name on the roster holds them. The old Master Han wrote those thirty-seven warrants and died where he sat — not one load of that salt ever left the granary. Keep them in the book and you make every future auditor reconcile against a payment that never happened.”</p>

  <p>So he gave Lü one task: <strong>tear out the surplus from your copy.</strong></p>

  <p>Tear back to where?</p>

  <p>The first method was to tear back to the seal line recorded in his own copy. That method destroys good entries.</p>

  <p>The seal is pressed by Xu, and word of it reaches Lü late. In the three days Lü had been off the roster, the seal had already jumped a long way forward, while the seal recorded on Lü’s copy still sat at the old place. Tear there and you have not torn enough: the remainder will not match the new Master, and tomorrow you tear again. And when word of the seal arrives too fast, you tear too much — you tear out an entry whose receipt is already in a merchant’s hand. The counter can prove it existed. The copy no longer can.</p>

  <p>For a fortnight the Scriptorium tore and recopied a dozen times a day, and each round was worse than the last. Chen said the room felt like two men with knives, neither willing to set his down first.</p>

  <p>The second method is the one Xu later wrote down: <strong>do not compare seals. Compare tenure marks.</strong></p>

  <p>Both copies carry the vermillion mark at the head of every span of the Master. Lü turned to the last mark in his own copy that matched the new Master — <em>Sixth tenure, beginning at line eight thousand one hundred twelve</em>. The new Master Han carried the same mark, in the same place.</p>

  <p>Then there is nothing to calculate. <strong>Everything after that mark gets torn out.</strong></p>

  <p>What remains stops exactly one line short of the new Master Han’s tail. Not a line too many, not a line too few. One tear, and no going back.</p>

  <p>This method settles in a single pass because a tenure mark is <strong>written by the man who takes the brush, and can only count upward.</strong> It is not anyone’s observation record. If the mark says sixth, you tear everything after the sixth. There is only one answer.</p>

  <p>That winter, one more thing happened.</p>

  <p>Rain fell for half a month, the storeroom leaked, and all three tables behind the screen were soaked. Not one man on the roster could copy. Six hundred receipts sat at the counter and the guild was hammering on the gate.</p>

  <p>Xu waited one night. At dawn, not one of the three brush-heads had moved.</p>

  <p>He opened a door his family had left in the rules: <strong>choose a successor from among those already struck off the roster.</strong></p>

  <p>He chose the man with the longest brush-head at that moment — the very man who had been struck off years before. The reason was urgency: the man behind the screen had not copied in ten days, but he carried several hundred unsealed lines, complete and intact.</p>

  <p>The new Master was installed, and the seal line moved backward.</p>

  <p>The next day, merchants who came to the counter with receipts in hand found their entry gone. Not struck out. Never present in the official book. The salt had left the granary, the receipt bore a vermillion seal, and the register was blank.</p>

  <p>The archive calls it the Case of the Three Hundred Receipts. Xu drove a bronze wedge into the frame of that door afterward, and no one has been able to pull it out. His explanation was short:</p>

  <p><strong>Better the counter closes for three days than the register moves back one line.</strong></p>

  <p>By now you have probably recognized it: this is <strong>Kafka replication — ISR, the high watermark, and leader epoch</strong>.</p>

  <h3 id="what-it-is">What it is</h3>

  <p>A Kafka partition is that scroll: <strong>append-only, no rewriting, every entry carrying a strictly increasing line number (offset).</strong></p>

  <p>At any moment exactly one replica accepts writes, the <em>leader</em> (Master Han). The others are <em>followers</em> (Chen, Tai, Lü). They issue <code class="language-plaintext highlighter-rouge">Fetch</code> requests and pull forward along the leader’s log, <strong>read-only, never writing</strong>.</p>

  <ul>
    <li><strong>LEO (log end offset)</strong> — how far each replica has written or copied: the brush-head.</li>
    <li><strong>HW (high watermark)</strong> — the minimum LEO across the ISR: the seal line. Consumers can read only up to the HW.</li>
    <li><strong>ISR (in-sync replicas)</strong> — the set of replicas that keep up. The test is <code class="language-plaintext highlighter-rouge">replica.lag.time.max.ms</code> (default 30000 ms): a follower that has not advanced its fetch to the leader’s LEO within that window is removed from the ISR. Note that it is <strong>time-based</strong>, not count-based: “how many records behind” is not comparable across partitions with different throughput.</li>
    <li><strong>leader epoch</strong> — every time leadership changes, the new leader writes an <code class="language-plaintext highlighter-rouge">(epoch, startOffset)</code> marker into its log. That is the vermillion tenure mark. Since KIP-101, a reconnecting follower no longer truncates using its own local HW; it sends <code class="language-plaintext highlighter-rouge">OffsetForLeaderEpoch</code>, the two sides find <strong>the last span whose epoch matches</strong>, and the follower truncates once to that boundary.</li>
  </ul>

  <p><strong>Why consumers stop at the HW.</strong> Because the leader can die at any moment, and a visible message must satisfy “it survives a leader change.” The HW is exactly that lower bound: the position every ISR member has passed. So a leader is chosen only from the ISR (<em>clean leader election</em>), the new leader’s LEO is ≥ HW, the HW never decreases, and receipts already issued never become forgeries.</p>

  <p><strong>Shrinking and expanding the ISR are not symmetric.</strong> Shrinking lets the HW move again — trading consistency for availability, and the only way to reopen the counter. Expanding requires the follower to catch up to the HW, not to the LEO, so a replica rejoining the ISR cannot drag the HW backward.</p>

  <p><strong>Three knobs that connect this mechanism to your business:</strong></p>
  <ul>
    <li><code class="language-plaintext highlighter-rouge">acks=all</code> + <code class="language-plaintext highlighter-rouge">min.insync.replicas=N</code> — the producer is acknowledged only after at least N ISR replicas have appended; when the ISR falls below N, writes are <strong>rejected</strong> with <code class="language-plaintext highlighter-rouge">NotEnoughReplicas</code>. This is “fewer than two names on the roster, Master Han stops writing.”</li>
    <li><code class="language-plaintext highlighter-rouge">unclean.leader.election.enable</code> — <code class="language-plaintext highlighter-rouge">false</code> by default. Set it to <code class="language-plaintext highlighter-rouge">true</code> and you allow a leader from outside the ISR: the seal line moves backward, acknowledged messages vanish, and you get the Case of the Three Hundred Receipts.</li>
    <li><code class="language-plaintext highlighter-rouge">isolation.level=read_committed</code> — consumers read only to the HW. <code class="language-plaintext highlighter-rouge">read_uncommitted</code> reads to the leader’s LEO, which means reading lines that may be torn out tomorrow.</li>
  </ul>

  <h3 id="why-it-matters">Why it matters</h3>

  <p><strong>One. “Committed” is not an abstraction; it is a formula you can write down.</strong> In Kafka it is <code class="language-plaintext highlighter-rouge">committed = min(LEO of every ISR member)</code>. Every replicated system must answer the same question: at what instant does a record count? The answer is never “the write succeeded.” It is “<strong>which replicas hold it, and would it still hold if the wrong man took the brush?</strong>” Put that into a formula and the cost becomes visible: the HW advances at the speed of <strong>the slowest ISR member</strong>. One slow disk can pin an entire partition’s read visibility to the floor. That is why ISR shrinking counts as necessary cruelty in operations.</p>

  <p><strong>Two. The arbiter of divergence must be an authoritative monotonic sequence, never a local observation.</strong> This rule reaches well beyond Kafka:</p>

  <ul>
    <li>Raft uses a <em>term</em>, Paxos a <em>ballot</em>, Kafka a <em>leader epoch</em>, etcd a <em>revision</em>.</li>
    <li>Let a follower truncate based on its own local HW — Kafka before 0.11 — and you get oscillation: truncate too little and it never converges, truncate too much and you delete committed data, so it truncates, refetches, and truncates again. A <em>truncation storm</em>.</li>
    <li>Decide with the leader’s tenure and you truncate once, deterministically. Lü tears once and never looks back — not because he is cleverer, but because he asked about <strong>a number that can only count upward</strong>.</li>
  </ul>

  <p>The same rule works elsewhere: Kubernetes <code class="language-plaintext highlighter-rouge">resourceVersion</code>, a Terraform state serial, a database LSN. All of them settle divergence with an authoritative monotonic number instead of an observer’s guess.</p>

  <p><strong>Three. Availability versus correctness is an explicit switch here, and it fails silently.</strong> <code class="language-plaintext highlighter-rouge">unclean.leader.election.enable=true</code> throws no error and raises no alarm; it simply lets history be rewritten. Conversely, <code class="language-plaintext highlighter-rouge">min.insync.replicas=2</code> with <code class="language-plaintext highlighter-rouge">acks=all</code> means that the moment the ISR drops to one, writes <strong>stop</strong>. Choosing between them is a business decision, not a technical one — but you should know you are choosing, because a default already chose for you.</p>

  <p><strong>Four. The metrics to watch are specific, and each translates into one sentence.</strong></p>

  <ul>
    <li><code class="language-plaintext highlighter-rouge">UnderReplicatedPartitions</code> &gt; 0 — some replica has been struck off the roster.</li>
    <li><code class="language-plaintext highlighter-rouge">UnderMinIsrPartitions</code> &gt; 0 — the roster is too thin and <strong>writes are being rejected right now</strong>. A sustained non-zero value means your availability is fictional: clients are not getting acks.</li>
    <li><code class="language-plaintext highlighter-rouge">MaxLag</code>, plus the rate of <code class="language-plaintext highlighter-rouge">isr-shrink</code> / <code class="language-plaintext highlighter-rouge">isr-expand</code> — a replica bouncing in and out of the ISR is oscillating at the threshold, and every shrink lengthens the HW’s delay.</li>
    <li>On the consumer side, <code class="language-plaintext highlighter-rouge">records-lag-max</code> and <code class="language-plaintext highlighter-rouge">log-end-offset - high-watermark</code>: that difference is what is already in Kafka but not yet visible, and it is set by the slowest replica, not by you.</li>
  </ul>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li>One long scroll, written from the tail upward, no jumping the line, no altering earlier characters → a partition is an append-only log; records are strictly ordered by offset</li>
    <li>One line of the ledger = one salt warrant → one message plus one offset</li>
    <li>Master Han alone writes new entries → exactly one <em>leader</em> replica accepts writes at a time</li>
    <li>Chen, Tai, and Lü only follow the Master and never write new entries → <em>followers</em> pull from the leader via <code class="language-plaintext highlighter-rouge">Fetch</code>, read-only</li>
    <li>The bronze weight rests on “how far I have copied” → each replica’s <strong>LEO</strong> (log end offset)</li>
    <li>The seal line sits just below the lowest of the three brush-heads → <strong>HW</strong> = minimum LEO across the ISR</li>
    <li>The counter may read only up to the seal line → consumers can read only up to the HW</li>
    <li>Master Han’s writing does not count; all three brush-heads must pass the line before a receipt is issued → <code class="language-plaintext highlighter-rouge">acks=all</code>: confirm to the producer only after every ISR replica has appended</li>
    <li>A pressed seal is never retracted, and issued receipts never become forgeries → with leadership confined to the ISR, the HW is monotonic and committed messages never disappear</li>
    <li>A brush-head that does not advance within one stick of incense is struck off the roster → ISR shrink governed by <code class="language-plaintext highlighter-rouge">replica.lag.time.max.ms</code> (default 30s)</li>
    <li>The test is “one stick of incense,” not “how many lines behind” → ISR membership is time-based rather than count-based, so it is comparable across partitions of different throughput</li>
    <li>Lü’s leaking lamp makes him copy slowly for three days; the seal freezes and the queue reaches the locust tree → one slow replica pins the whole partition’s HW; read visibility advances at the speed of the slowest ISR member</li>
    <li>Striking Lü off lets the seal jump forward and the counter reopen → ISR shrink unblocks the HW, trading consistency for availability</li>
    <li>The struck-off man may return and keep copying, but the seal does not wait for him → a replica outside the ISR keeps replicating, but its progress does not affect the HW</li>
    <li>Rejoining the roster requires catching up to the seal line, not to the tail → a replica re-enters the ISR only after reaching the HW, so it cannot drag the HW back</li>
    <li>The successor must come from the roster of those in place → <em>clean leader election</em>: only an ISR member can become leader</li>
    <li>Choose the roster member with the longest brush-head → the new leader is the ISR replica with the highest LEO, guaranteeing LEO ≥ HW</li>
    <li>The vermillion mark at the head: “Seventh tenure, beginning at line eight thousand one hundred twelve” → <strong>leader epoch</strong>, an <code class="language-plaintext highlighter-rouge">(epoch, startOffset)</code> marker that increments monotonically with each leadership change</li>
    <li>Lü’s copy holds two hundred eighty unsealed lines beyond the new Master → a follower’s log tail holding uncommitted messages past the HW</li>
    <li>Tearing out the surplus → <strong>log truncation</strong></li>
    <li>Tearing back to the seal recorded in one’s own copy under- or over-tears, a dozen rounds a day → the <em>truncation storm</em> caused by the old local-HW truncation protocol</li>
    <li>Do not compare seals; compare tenure marks — find the last shared vermillion mark and tear everything after it → <code class="language-plaintext highlighter-rouge">OffsetForLeaderEpoch</code>: truncate at the boundary of the greatest epoch the two sides share</li>
    <li>One tear, never revisited → epochs increase monotonically, making the truncation point unique, idempotent, and convergent in one pass</li>
    <li>The tenure mark is written by the incoming man and can only count upward → the epoch is set by the leader’s tenure, not by a follower’s local observation</li>
    <li>The storeroom floods, nobody on the roster can copy, and Xu appoints a struck-off man → <code class="language-plaintext highlighter-rouge">unclean.leader.election.enable=true</code></li>
    <li>The new Master is installed, the seal line retreats, three hundred receipts turn blank → the HW moves backward, acknowledged messages are lost, and history is silently rewritten</li>
    <li>Fewer than two names on the roster and Master Han stops writing, hanging out a “not accepting warrants” sign → writes rejected with <code class="language-plaintext highlighter-rouge">NotEnoughReplicas</code> when <code class="language-plaintext highlighter-rouge">min.insync.replicas</code> is unmet</li>
    <li>Two ways to read at the counter: up to the seal, or up to Master Han’s tail → <code class="language-plaintext highlighter-rouge">isolation.level=read_committed</code> versus <code class="language-plaintext highlighter-rouge">read_uncommitted</code></li>
    <li>What the impatient reader sees today may be gone tomorrow → reading past the HW means reading content that may yet be truncated</li>
    <li>The bronze wedge nailed into the door frame: “better the counter closes for three days than the register moves back one line” → the default <code class="language-plaintext highlighter-rouge">unclean.leader.election.enable=false</code>; correctness before availability</li>
  </ul>
</section>]]></content><author><name></name></author><category term="distributed-systems" /><category term="databases" /><summary type="html"><![CDATA[盐课司后院有一间没有窗的屋子，叫正本斋。]]></summary></entry><entry><title type="html">六百个渡位 · Six Hundred Seats</title><link href="https://fables.day/2026/11/09/api-priority-and-fairness/" rel="alternate" type="text/html" title="六百个渡位 · Six Hundred Seats" /><published>2026-11-09T00:00:00+00:00</published><updated>2026-11-09T00:00:00+00:00</updated><id>https://fables.day/2026/11/09/api-priority-and-fairness</id><content type="html" xml:base="https://fables.day/2026/11/09/api-priority-and-fairness/"><![CDATA[<section class="zh">
  <p>青石渡在沉沙河西岸。对岸只有一座埠头，叫独石埠。</p>

  <p>独石埠窄。一次只容一船靠，一船靠了要半刻钟卸完，卸完还得由官仓的账房在簿子上记一笔——<strong>记完这一笔，才轮到下一船。</strong> 所以这条河上真正的瓶颈从来不是水，也不是船，是那座独石埠。</p>

  <p>渡口司在河里备着船，也备着”位”。一个过河的人占一个位，一担货占一个位，从离岸到在东岸卸完、簿子上记完，位才放开。早年规矩很省事：<strong>四百个位给人，两百个位给货，先到先渡，候渡棚不设上限。</strong></p>

  <p>一千多号人靠这条渡口过日子。更夫老陈每半个时辰要过一趟，替西岸报一声平安；商会要往东岸递行文，选一个”值年”，谁先递到谁当家；七家帮口各有各的货；官府有札子；镇上还有一堆闲着没事、一天要过三趟河看热闹的看客。各行其是，谁也不管谁，倒也太平了十几年。</p>

  <p>出事那年秋天，渡口上新开了一家叫<strong>顺风号</strong>的货栈。</p>

  <p>顺风号雇了个伙计叫周二，人勤快，就是没听懂东家的话。东家说的是”每半个时辰看一遍货单”，周二听成了”一直看”。他从早到晚往渡口跑，一天跑一万二千趟，每趟都带着一张写满”我要知道东岸仓里现在有多少箱”的货单。</p>

  <p>候渡棚原来能站八百人。第三天站了三千，第五天站了三万。</p>

  <p>然后独石埠开始堵。</p>

  <p>堵不是慢慢堵，是<strong>越堵越慢</strong>。因为每卸一笔货，账房除了记账，还要把这张货单抄一份给所有在东岸蹲着看账的人。蹲着看账的人越多，抄一份要越久；抄得越久，一笔货占埠头的时间越长；占得越久，外海面上挤的船越多；挤得越多，账房被打断的次数越多。</p>

  <p><strong>渡口一刻钟原本过两百担，那年秋天掉到六十担。</strong> 而棚里的人从八百涨到三万。</p>

  <p>更夫老陈那天在棚外排了四个时辰，一动没动。他要报的那句话只有两个字：<strong>平安。</strong></p>

  <p>东岸的官仓等不到平安。等不到，就按老规矩判了：西岸失联。青石渡当天从”在册”里被划掉，漕粮停了，官府还预备派人来清点人口、把流民迁走。</p>

  <p>老陈后来跟人说，那天他不是被谁挤死的，他是<strong>排在一条不动的队里，慢慢被”等”死的</strong>。</p>

  <p><strong>第一个办法是封顶。</strong></p>

  <p>渡口司下令：水面上同时只许六百人。多的人一律不许下船。</p>

  <p>这一条立竿见影。独石埠不再堵成一团，抄账不再被打断，渡口一刻钟又回到了两百担。渡口司松了口气。</p>

  <p>第二天，顺风号的伙计占满了六百个位里头的五百八十个。老陈的筏子照旧在棚外，照旧过不去。</p>

  <p><strong>封顶只保证了渡口不倒，它一个字都没说谁该先过。</strong></p>

  <p><strong>第二个办法，是渡口司最后拿出来的七条。</strong></p>

  <p><strong>第一条，分等级。</strong> 报平安的一等，商会行文的一等，官府差事的一等，帮口日常的一等，寻常看客一等，剩下说不清的一等。最后那一等份额极小，且<strong>不设候渡道</strong>——来了当场回绝。另外单设一路，叫<strong>钦差</strong>：持钦差火牌的（整个系统的主子），不占位、不排队、直接过。理由很冷，但渡口司写在了告示上：<strong>万一册子本身写坏了，得有人能进去改册子。</strong></p>

  <p><strong>第二条，立名册。</strong> 每个请求按三样东西分等级——<strong>谁派来的</strong>、<strong>办什么事</strong>、<strong>去哪座仓</strong>。册子上标着序号，从序号最小的开始查，<strong>第一个对上的算数</strong>，后面的不再看。所以序号不是排序，是<strong>优先匹配的次序</strong>：越要紧的规则，序号越小，先给它机会。</p>

  <p><strong>第三条，分份子。</strong> 六百个位不按人头分，按”份”分。报平安那一等份子最大——因为它是整条河的脉搏。份子是比例不是死数：哪天渡口把总位从六百改成一千二，各等按同一个比例一起涨，不用重写册子。</p>

  <p><strong>第四条，借位。</strong> 闲着的等级可以把位借出去，但<strong>最多只借出自己份子的三成</strong>。这个数不是抠门，是为了能收回来：自己的货一到，得立刻把位拿回去，不能等别人卸完。每隔一小会儿重算一次——先把借出去且自己又要用的收回来，再按近来谁最挤，匀一匀。借来的位是<strong>借</strong>，不是<strong>抢</strong>：<strong>已经上了船的人，不赶下船。</strong> 渡口司的原话是：”赶人下船，比让人等更贵。”</p>

  <p><strong>第五条，分道。</strong> 每个等级里划一百二十八条候渡道。一个帮口不排所有的道——按自己名字的号，它只落在其中<strong>六条</strong>道上。六条满了，它也占不了另外一百二十二条。这一条是整份告示里最巧的一格：<strong>它不靠讲道理防人挤人，它靠说不上话的号码，让挤的人挤不到别人头上。</strong></p>

  <p><strong>第六条，公平取号。</strong> 每条道记一个”虚拟完工时刻”。每放一趟船，从<strong>虚拟时刻最落后</strong>的那条道取人，取完把那道往前推一点。于是嗓门大没有用，本钱厚也没有用，只有”这条道走得少”才占先。同一个帮口里的两个伙计，各走各的道，谁也饿不死谁。</p>

  <p><strong>第七条，队满即拒。</strong> 道满了，当场回绝，回绝时附一句”X 刻后再来”。这看着最不客气，其实最慈悲：<strong>一个从不回绝的队，最后一定变成一个拖到超时的队</strong>，而超时的人还占着位、还在等、还不敢重试。</p>

  <p>告示贴出去那天，老陈的筏子在第一等里排到了第九位，一炷香没过河。</p>

  <p>同一时刻，顺风号的周二被卡在自己的六条道里，进不去，也被回绝得明明白白。他蹲在棚外数了一夜的回绝条子，第二天早晨忽然想通了一件事：<strong>不是渡口不给他过，是他自己一天跑一万二千趟这件事本身，才是那三万人的原因。</strong></p>

  <p>他把趟数改回半个时辰一趟，六条道立刻空了五条。</p>

  <p>那年冬天，渡口司在告示旁边又添了一行小字，写得很不客气：</p>

  <p><strong>六百个位不是渡口的上限。六百个位是独石埠的上限。多放船进去，不是多运货，是多堵。</strong></p>

  <p>到这儿，你大概已经认出来了：这就是 <strong>Kubernetes 的 API 优先级与公平性</strong>（<em>API Priority and Fairness</em>，<em>APF</em>）。</p>

  <h3 id="section">这是什么</h3>

  <p>先说它要挡的东西。</p>

  <p><code class="language-plaintext highlighter-rouge">kube-apiserver</code> 前面原本只有一把大刀：<code class="language-plaintext highlighter-rouge">--max-requests-inflight</code>（默认 400）和 <code class="language-plaintext highlighter-rouge">--max-mutating-requests-inflight</code>（默认 200）。两个数是<strong>全体共用</strong>的：不管来的是 kubelet 的心跳，还是某个控制器里的死循环，都从同一个池子里舀。</p>

  <p>于是有了那个经典的正反馈：一个写坏的控制器热循环打满 apiserver → <strong>kubelet 的心跳请求也被挤掉</strong> → 节点被判 <code class="language-plaintext highlighter-rouge">NotReady</code> → 触发驱逐 → 新 Pod 起来 → 请求更多 → 更挤。apiserver 没倒，它只是<strong>安静地杀掉了一部分节点</strong>。这就是老陈那天报不出去的”平安”。</p>

  <p><strong>APF</strong> 做的事，是把那把大刀换成一整套渡口调度。</p>

  <p><strong>一、两个数加起来，然后按”份”分。</strong> APF 打开后，<code class="language-plaintext highlighter-rouge">--max-requests-inflight</code> 与 <code class="language-plaintext highlighter-rouge">--max-mutating-requests-inflight</code> 相加（默认 600），成为全服务器的<strong>总并发</strong>，即座位（<em>seat</em>）总数——APF 的官方用词就是”座位”，借的是火车与飞机上”一个乘客占一个座”的意象。它<strong>不再区分读写</strong>；要区分，就自己写两条 FlowSchema 分别匹配动词。</p>

  <p><strong>二、座位不是均匀的。</strong> <code class="language-plaintext highlighter-rouge">list</code> 请求若预估会返回大量对象，会一次占<strong>多个</strong>座位，数量与预估对象数成正比——这是周二那张”我要知道东岸现在有多少箱”的货单。写请求在写完<strong>之后</strong>还要多占一段时间、多占几个座位，因为它要给所有 <code class="language-plaintext highlighter-rouge">watch</code> 流扇出通知，扇出的量也参与估算。<code class="language-plaintext highlighter-rouge">watch</code> 请求只在<strong>首轮存量对象的通知发完</strong>之前占座位，之后蹲着就不占了。而 <code class="language-plaintext highlighter-rouge">exec</code>、日志 <code class="language-plaintext highlighter-rouge">tail</code> 这类长连接请求<strong>根本不进 APF</strong>——它们不占位、不排队。</p>

  <p><strong>三、等级（PriorityLevelConfiguration）。</strong> 每个等级有自己的并发上限。默认配置给出几个：<code class="language-plaintext highlighter-rouge">node-high</code>（节点健康上报）、<code class="language-plaintext highlighter-rouge">system</code>（<code class="language-plaintext highlighter-rouge">system:nodes</code> 组的非健康请求，即 kubelet）、<code class="language-plaintext highlighter-rouge">leader-election</code>（内置控制器抢 <code class="language-plaintext highlighter-rouge">leases</code>/<code class="language-plaintext highlighter-rouge">endpoints</code>）、<code class="language-plaintext highlighter-rouge">workload-high</code>（内置控制器的其他请求）、<code class="language-plaintext highlighter-rouge">workload-low</code>（任何其他 ServiceAccount，通常是 Pod 里的控制器）、<code class="language-plaintext highlighter-rouge">global-default</code>（其余，比如人敲的 <code class="language-plaintext highlighter-rouge">kubectl</code>）。另外两个是强制的：<code class="language-plaintext highlighter-rouge">catch-all</code>（份额极小、<strong>不排队</strong>）、<code class="language-plaintext highlighter-rouge">exempt</code>（完全不受限，<code class="language-plaintext highlighter-rouge">system:masters</code> 组默认落在这里）。</p>

  <p><code class="language-plaintext highlighter-rouge">leader-election</code> 单独一级不是讲究：选主失败会让控制器重启，重启的控制器回来要全量 sync informer，那笔流量比选主本身贵几个数量级。</p>

  <p><strong>四、借与还。</strong> 等级的上限写成 <code class="language-plaintext highlighter-rouge">nominalConcurrencyShares</code>（份子），不是绝对座位数，所以调总并发时所有等级按同一比例涨落。每秒重算一次动态上限：先收回自己借出去且当下又需要的座位，再按近来的需求公平分配。上下界由 <code class="language-plaintext highlighter-rouge">lendingPercent</code> / <code class="language-plaintext highlighter-rouge">borrowingPercent</code> 卡住——一边最多借出份子的百分比，另一边最多借入的百分比。<strong>借不是抢占</strong>：已经开始的请求不会被取消。在 apiserver 里抢占意味着半提交的写，加上客户端重试，比让人排队糟糕得多。</p>

  <p><strong>五、排队、抓阄、公平。</strong> 等级内部分成若干队列（<code class="language-plaintext highlighter-rouge">queues</code>），请求按<strong>流</strong>（<em>flow</em>）分配：流 = 命中的 FlowSchema 名 + 一个<strong>区分符</strong>（<em>distinguisher</em>，可以是请求的用户、或目标资源的 namespace，或者不分）。分配用 <strong>shuffle sharding（洗牌分片）</strong>：每个流从 <code class="language-plaintext highlighter-rouge">queues</code> 条队列里哈希出 <code class="language-plaintext highlighter-rouge">handSize</code> 条，再从这 <code class="language-plaintext highlighter-rouge">handSize</code> 条里挑最短的那条排。队列之间用<strong>公平排队</strong>（<em>fair queuing</em>）派发——按虚拟完工时刻取最落后的那条。</p>

  <p>两个数字的取舍很具体：<code class="language-plaintext highlighter-rouge">handSize</code> 大，两个流撞在一起的概率低（谁也别想饿死谁），但少数大流量更容易吃满整个服务器；<strong>单个流最多能排 <code class="language-plaintext highlighter-rouge">handSize × queueLengthLimit</code> 个请求</strong>。<code class="language-plaintext highlighter-rouge">queues</code> 大，碰撞率低，内存高；<code class="language-plaintext highlighter-rouge">queues = 1</code> 等于关掉公平逻辑，只剩排队。</p>

  <p><strong>六、多出来的请求怎么办。</strong> 由等级的 <code class="language-plaintext highlighter-rouge">limitResponse</code> 决定：<code class="language-plaintext highlighter-rouge">Reject</code> 直接回 429；<code class="language-plaintext highlighter-rouge">Queue</code> 则排队，排不下或等太久，同样以 429 结束，并带 <code class="language-plaintext highlighter-rouge">Retry-After</code>。<code class="language-plaintext highlighter-rouge">apiserver_flowcontrol_rejected_requests_total</code> 的 <code class="language-plaintext highlighter-rouge">reason</code> 会把原因分开写清：<code class="language-plaintext highlighter-rouge">queue-full</code>、<code class="language-plaintext highlighter-rouge">concurrency-limit</code>、<code class="language-plaintext highlighter-rouge">time-out</code>、<code class="language-plaintext highlighter-rouge">cancelled</code>。</p>

  <h3 id="section-1">为什么重要</h3>

  <p><strong>一、并发不是吞吐。</strong> 独石埠是串行的：etcd 的写要过 leader 逐条提交。超过某个拐点后，更多在途请求不会提高完成率，只会抬高每次提交的延迟，然后完成率<strong>下降</strong>——这是排队网络里的拥塞崩溃，不是过载的比喻，是它的定义。用 <em>Little’s law</em> 说更直白：<code class="language-plaintext highlighter-rouge">L = λW</code>。位子数 <code class="language-plaintext highlighter-rouge">L</code> 是定的，你硬把 <code class="language-plaintext highlighter-rouge">λ</code> 提上去，<code class="language-plaintext highlighter-rouge">W</code> 就按比例涨；而 <code class="language-plaintext highlighter-rouge">W</code> 会反过来吃掉 <code class="language-plaintext highlighter-rouge">λ</code>。APF 的核心动作因此是<strong>限额加排队</strong>，不是加并发。这是它和”把 <code class="language-plaintext highlighter-rouge">--max-requests-inflight</code> 调大”最根本的分歧。</p>

  <p><strong>二、一个坏客户端能拖垮一整个集群，而那件事看起来像别的问题。</strong> 热循环的控制器打满 apiserver，症状不出现 apiserver 上，出现在<strong>节点的 <code class="language-plaintext highlighter-rouge">NotReady</code></strong> 上、在驱逐上、在 Pod 起不来上。运维看到的是节点抖动，根因是某个 namespace 里一个写了 <code class="language-plaintext highlighter-rouge">for {}</code> 的 reconcile。这是 Kubernetes 事故里最常见的一类误导。APF 把这件事变得<strong>可归因</strong>：<code class="language-plaintext highlighter-rouge">apiserver_flowcontrol_*</code> 指标按 <code class="language-plaintext highlighter-rouge">flow_schema</code> 与 <code class="language-plaintext highlighter-rouge">priority_level</code> 打标签，谁在打满服务器，一眼看得见——前提是流的区分符选对了（<code class="language-plaintext highlighter-rouge">ByUser</code> 而不是不分）。文档里有一句毫不含糊的提醒：<strong>控制器实例多，就该用不同的用户名认证</strong>，否则它们全被算成一个流，公平排队对它们失效。</p>

  <p><strong>三、默认配置是可改的，而且大概率该改。</strong> 官方给的是”合理”的一组，不是”适合你”的一组。三个具体的坑：默认配置<strong>不</strong>给 kubelet 的本地健康检查（<code class="language-plaintext highlighter-rouge">/healthz</code>、<code class="language-plaintext highlighter-rouge">/livez</code>、<code class="language-plaintext highlighter-rouge">/readyz</code>，走安全端口但无凭据）任何优待，它们落在 <code class="language-plaintext highlighter-rouge">global-default</code> 里被人挤掉；<code class="language-plaintext highlighter-rouge">catch-all</code> 不排队，一律 429；而 <code class="language-plaintext highlighter-rouge">--max-requests-inflight</code> 在 APF 下的含义已经变了——它不再是天花板，是分母。</p>

  <p><strong>四、它是一个”分层失败”的设计，而不是”不失败”的设计。</strong> 五百八十个位被一个帮口占满时，渡口没有倒，它只是让老陈过不去；<code class="language-plaintext highlighter-rouge">exempt</code> 存在的意义就是——万一告示写坏了，还有人能进去改告示。承认自己会被配错，然后为配错留一条活路，比假装配不错更可靠。</p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li>沉沙河对岸只有一座独石埠，一船一卸，记完一笔才轮到下一船 → etcd 的写由 leader 串行提交，是 apiserver 真正的瓶颈</li>
    <li>一个位 = 一个从离岸到记完账的名额 → 一个 <em>seat</em> = 一个在途请求占用的一个并发单位</li>
    <li>早年四百个位给人、两百个位给货，先到先渡，棚不设上限 → <code class="language-plaintext highlighter-rouge">--max-requests-inflight</code> 400 与 <code class="language-plaintext highlighter-rouge">--max-mutating-requests-inflight</code> 200，全局共享，无排队</li>
    <li>顺风号的周二把”每半个时辰看一遍”听成”一直看”，一天一万二千趟 → 控制器热循环（tight reconcile / list-watch loop）打满 apiserver</li>
    <li>每卸一笔要把货单抄给所有蹲守看账的人，人越多抄得越慢 → 写请求要为 <code class="language-plaintext highlighter-rouge">watch</code> 扇出通知，扇出量与在途 watcher 数成正比，也计入座位与占位时长</li>
    <li>蹲守看账的人只在头一遍抄旧账时占位，之后蹲着不占 → <code class="language-plaintext highlighter-rouge">watch</code> 请求在首轮存量对象通知发完后即释放座位</li>
    <li>沿河拉纤、盯着人干活这类长差事不占位不排队 → <code class="language-plaintext highlighter-rouge">exec</code>、日志 <code class="language-plaintext highlighter-rouge">tail</code> 等长连接请求不经过 APF</li>
    <li>一张”东岸现在有多少箱”的货单一次要占很多位 → <code class="language-plaintext highlighter-rouge">list</code> 请求按预估对象数占用多个座位</li>
    <li>越堵越慢，渡口从一刻二百担掉到六十担 → 拥塞崩溃：超过拐点后并发上升、吞吐下降（<code class="language-plaintext highlighter-rouge">L = λW</code>）</li>
    <li>老陈四个时辰报不出”平安”，西岸被判失联、停漕粮 → kubelet 心跳被挤掉 → 节点 <code class="language-plaintext highlighter-rouge">NotReady</code> → 驱逐 → 更多请求的正反馈</li>
    <li>封顶六百人：渡口不倒，但老陈照旧过不去 → 单一全局 max-inflight 保护了服务器，却保护不了任何具体请求</li>
    <li>报平安 / 商会行文 / 官府差事 / 帮口日常 / 看客 / 说不清的 → <code class="language-plaintext highlighter-rouge">node-high</code>/<code class="language-plaintext highlighter-rouge">system</code>、<code class="language-plaintext highlighter-rouge">leader-election</code>、<code class="language-plaintext highlighter-rouge">workload-high</code>、<code class="language-plaintext highlighter-rouge">workload-low</code>、<code class="language-plaintext highlighter-rouge">global-default</code>、<code class="language-plaintext highlighter-rouge">catch-all</code></li>
    <li>最后那一等份额极小且不设候渡道，来了当场回绝 → <code class="language-plaintext highlighter-rouge">catch-all</code> 份额极小且不排队，直接 429</li>
    <li>持钦差火牌者不占位不排队，理由”万一册子写坏了得有人能改册子” → <code class="language-plaintext highlighter-rouge">exempt</code> 等级，<code class="language-plaintext highlighter-rouge">system:masters</code> 默认在内，防止错误的流控配置把 apiserver 彻底锁死</li>
    <li>名册按”谁派来的、办什么事、去哪座仓”分等级 → FlowSchema 的 <code class="language-plaintext highlighter-rouge">subjects</code>、<code class="language-plaintext highlighter-rouge">resourceRules</code>/<code class="language-plaintext highlighter-rouge">nonResourceRules</code>、<code class="language-plaintext highlighter-rouge">namespaces</code>/<code class="language-plaintext highlighter-rouge">nonResourceURLs</code></li>
    <li>册子标序号，从序号最小的查起，第一个对上的算数 → <code class="language-plaintext highlighter-rouge">matchingPrecedence</code> 数值最小者优先匹配，首个命中生效</li>
    <li>六百个位不按人头分，按份分；份是比例不是死数 → <code class="language-plaintext highlighter-rouge">nominalConcurrencyShares</code> 按比例分配总并发，总并发变化时同比例涨落</li>
    <li>报平安那一等份子最大 → <code class="language-plaintext highlighter-rouge">system</code>（kubelet）在默认配置里拿最大份之一，因为它是集群脉搏</li>
    <li>闲着的等级可以借位，但最多借出三成，为的是能马上收回来 → 借出/借入按份子百分比设界，动态上限每秒重算，先收回再匀</li>
    <li>已经上船的人不赶下船，赶人比让人等更贵 → 不做抢占；取消在途写会留下半提交状态，比排队更糟</li>
    <li>一百二十八条候渡道，一个帮口按名字的号只落在其中六条 → <code class="language-plaintext highlighter-rouge">queues</code> 与 <code class="language-plaintext highlighter-rouge">handSize</code> 的 shuffle sharding，把单个流的破坏面钉在 <code class="language-plaintext highlighter-rouge">handSize</code> 条队列内</li>
    <li>六条道满了也占不了另外一百二十二条 → 一个流最多排队 <code class="language-plaintext highlighter-rouge">handSize × queueLengthLimit</code> 个请求</li>
    <li>每趟从虚拟完工时刻最落后的那条道取人 → 公平排队（fair queuing）按虚拟时间派发，声量与本钱都不占先</li>
    <li>同帮口的两个伙计各走各的道，谁也饿不死谁 → 流的区分符（<code class="language-plaintext highlighter-rouge">ByUser</code> / <code class="language-plaintext highlighter-rouge">ByNamespace</code>）决定公平的粒度</li>
    <li>队满了当场回绝，附一句”X 刻后再来” → 429 加 <code class="language-plaintext highlighter-rouge">Retry-After</code>；<code class="language-plaintext highlighter-rouge">rejected_requests_total</code> 的 <code class="language-plaintext highlighter-rouge">reason</code> 分 <code class="language-plaintext highlighter-rouge">queue-full</code>/<code class="language-plaintext highlighter-rouge">concurrency-limit</code>/<code class="language-plaintext highlighter-rouge">time-out</code>/<code class="language-plaintext highlighter-rouge">cancelled</code></li>
    <li>从不回绝的队最后变成拖到超时的队，而超时的人还占着位 → 拒绝优于堆积：排队超时既占座位又触发客户端重试风暴</li>
    <li>周二被卡在自己的六条道里，被回绝得明明白白，第二天把趟数改回来 → 429 + <code class="language-plaintext highlighter-rouge">Retry-After</code> 让客户端能自愈；指标让责任可归因</li>
    <li>告示旁的小字：六百个位是独石埠的上限，不是渡口的上限 → 总并发是瓶颈定的，不是愿望定的；调大 max-inflight 只是把拥塞挪个位置</li>
  </ul>
</section>
<section class="en">
  <p>Bluestone Ford sits on the west bank of the Sinking Sands River. Across the water there is exactly one landing, called the Single Stone Quay.</p>

  <p>It is narrow. One boat at a time. A boat takes a quarter hour to unload, and then the granary’s clerk has to enter the shipment in the ledger — <strong>the entry must be finished before the next boat ties up.</strong> So the true bottleneck on this river was never the water and never the boats. It was that one quay.</p>

  <p>The ford office kept boats in the water, and it kept <strong>seats</strong>. A person crossing takes a seat; a load of goods takes a seat; the seat is released when the goods are unloaded on the east bank and the ledger entry is made. The old rule was simple: <strong>four hundred seats for people, two hundred for goods, first come first served, and the waiting shed has no limit.</strong></p>

  <p>Over a thousand people lived off that ford. The night watchman, Old Chen, crossed every half hour to report that the west bank was well; the merchants’ guild sent its documents east to elect a factor for the year, whoever arrived first holding the post; seven trade houses moved their goods; the magistrate sent pouches; and a good number of idle townsfolk crossed three times a day just to look. Each went about its own business, nobody governed anybody, and for a dozen years it held.</p>

  <p>The autumn it broke, a freight house called <strong>Fairwind</strong> opened at the ford.</p>

  <p>Fairwind hired a hand named Zhou the Second — industrious, but he had not understood his employer. The instruction was “look at the manifest once every half hour.” Zhou heard “keep looking.” He ran to the ford from dawn to dark, twelve thousand trips a day, each carrying a slip reading: <em>I need to know how many crates are in the east granary right now.</em></p>

  <p>The waiting shed held eight hundred. On the third day it held three thousand. On the fifth, thirty thousand.</p>

  <p>Then the Single Stone Quay began to jam.</p>

  <p>Not gradually. <strong>The more it jammed, the slower it got.</strong> Every time a shipment was unloaded, the clerk not only entered it — he had to copy the slip for every person squatting on the east bank watching the ledger. The more watchers, the longer each copy took; the longer the copy, the longer each boat held the quay; the longer the quay was held, the more boats pressed in; the more boats pressed in, the more often the clerk was interrupted.</p>

  <p><strong>The ford had moved two hundred loads a quarter hour. That autumn it moved sixty.</strong> And the shed went from eight hundred to thirty thousand.</p>

  <p>Old Chen stood outside that shed for four hours without moving. The message he was carrying was two characters long: <strong>all is well.</strong></p>

  <p>The east granary never received it. Never receiving it, it applied the old rule: the west bank is out of contact. Bluestone Ford was struck off the register that same day, the grain shipments stopped, and the magistrate began preparing to send men to count the population and relocate the refugees.</p>

  <p>Old Chen later told people he had not been crushed by anyone. He had <strong>stood in a queue that did not move, and been slowly waited to death.</strong></p>

  <p><strong>The first remedy was a cap.</strong></p>

  <p>The ford office ordered: never more than six hundred people on the water at once. Nobody else may board.</p>

  <p>It worked immediately. The quay stopped balling up, the copying stopped being interrupted, and the ford returned to two hundred loads a quarter hour. The office breathed out.</p>

  <p>The next day, Fairwind’s hands occupied five hundred and eighty of the six hundred seats. Old Chen’s raft sat outside the shed, exactly as before, unable to cross.</p>

  <p><strong>A cap guarantees the ford will not collapse. It says not one word about who should go first.</strong></p>

  <p><strong>The second remedy was seven articles, and it was the last one the ford office produced.</strong></p>

  <p><strong>One: rank them.</strong> A grade for the peace report, a grade for guild documents, a grade for magistrate business, a grade for the houses’ daily goods, a grade for onlookers, and a grade for everything unclassifiable. That last grade has a very small share and <strong>no waiting lanes at all</strong> — it is refused on the spot. And one more lane is set apart, called <strong>by imperial warrant</strong>: whoever holds the warrant — the master of the whole system — takes no seat, joins no queue, and crosses at once. The reason is cold, and the office wrote it on the notice: <strong>if the register itself is ever written wrong, somebody has to be able to get in and fix the register.</strong></p>

  <p><strong>Two: keep a register.</strong> Every request is graded on three things — <strong>who sent it</strong>, <strong>what business it is</strong>, <strong>which granary it targets</strong>. The register is numbered, and it is read from the lowest number upward, <strong>first match wins</strong>, nothing after it is consulted. The number is therefore not a ranking but an <strong>order of precedence</strong>: the rule that matters most gets the smallest number, and first claim.</p>

  <p><strong>Three: allocate by share.</strong> The six hundred seats are not divided per head but per <strong>share</strong>. The peace-report grade holds the largest share, because it is the pulse of the whole river. A share is a proportion, not a number: raise the ford from six hundred seats to twelve hundred and every grade grows by the same fraction, with no rewriting of the register.</p>

  <p><strong>Four: lend seats.</strong> An idle grade may lend its seats out, but <strong>never more than three tenths of its own share.</strong> That number is not stinginess; it is what makes recall possible — when its own goods arrive it must take the seats back at once, not wait for someone else’s unload to finish. The office recomputes every little while: first reclaim what it lent and now needs, then even out the rest by who has been crowded lately. A borrowed seat is <strong>borrowed</strong>, not <strong>seized</strong>: <strong>nobody already aboard is thrown off.</strong> The office’s wording: “Throwing a man off the boat costs more than making him wait.”</p>

  <p><strong>Five: split the lanes.</strong> Inside each grade, a hundred and twenty-eight waiting lanes are marked out. A trade house does not queue in all of them — by the number of its own name it lands in just <strong>six</strong>. When those six are full it still cannot take a single one of the other hundred and twenty-two. This is the cleverest cell in the whole notice: <strong>it does not argue people out of crowding each other. It uses a number nobody can talk to, so that a crowd cannot crowd anyone else.</strong></p>

  <p><strong>Six: call the numbers fairly.</strong> Each lane keeps a <em>virtual completion time</em>. Each departure is taken from the lane <strong>furthest behind</strong> on that clock, and then that lane’s clock is pushed forward. Loudness buys nothing. Capital buys nothing. Only “this lane has moved less” gets you ahead. Two hands of the same house walk their own lanes, and neither can starve the other.</p>

  <p><strong>Seven: refuse a full lane at once.</strong> When a lane is full, refuse on the spot, and refuse with a note: <em>come back in so many minutes.</em> This looks like the rudest article and is in fact the kindest: <strong>a queue that never refuses always becomes a queue that times out</strong> — and the ones timing out are still holding seats, still waiting, and afraid to retry.</p>

  <p>The day the notice went up, Old Chen’s raft stood ninth in the first grade and was across inside the time it takes to burn a stick of incense.</p>

  <p>At that same hour, Zhou the Second was stuck in his own six lanes, refused with perfect clarity. He squatted outside the shed all night counting refusal slips, and by morning something had landed: <strong>the ford had not been refusing to let him cross. His twelve thousand trips a day were themselves the reason for the thirty thousand people.</strong></p>

  <p>He put the trips back to one every half hour. Five of his six lanes emptied immediately.</p>

  <p>That winter the ford office added a line beside the notice, and it was not polite:</p>

  <p><strong>Six hundred seats are not the ford’s limit. Six hundred seats are the quay’s limit. Putting more boats on the water does not move more goods. It jams more.</strong></p>

  <p>By now you have probably recognized it: this is Kubernetes’ <strong>API Priority and Fairness</strong> (<em>APF</em>).</p>

  <h3 id="what-it-is">What it is</h3>

  <p>Start with what it has to hold back.</p>

  <p>In front of <code class="language-plaintext highlighter-rouge">kube-apiserver</code> there used to be a single blunt instrument: <code class="language-plaintext highlighter-rouge">--max-requests-inflight</code> (default 400) and <code class="language-plaintext highlighter-rouge">--max-mutating-requests-inflight</code> (default 200). Both numbers are <strong>shared by everyone</strong>: whether the request is a kubelet heartbeat or a hot loop inside some controller, it scoops from the same pool.</p>

  <p>Hence the classic positive feedback: a broken controller hot-loops the apiserver → <strong>the kubelet’s heartbeat is squeezed out too</strong> → the node is judged <code class="language-plaintext highlighter-rouge">NotReady</code> → eviction fires → new pods start → more requests → more pressure. The apiserver does not go down. It <strong>quietly kills some of your nodes.</strong> That is Old Chen’s undeliverable peace report.</p>

  <p><strong>APF</strong> replaces the blunt instrument with an entire ferry schedule.</p>

  <p><strong>One: add the two numbers, then divide by share.</strong> With APF enabled, the two flags are summed (600 by default) into the server’s total concurrency — its total number of seats (<em>seat</em> is literally the APF term, borrowed from a passenger taking one of a fixed supply of seats on a train). It <strong>no longer distinguishes reads from writes</strong>; to distinguish them, write two FlowSchemas matching different verbs.</p>

  <p><strong>Two: seats are not uniform.</strong> A <code class="language-plaintext highlighter-rouge">list</code> request estimated to return many objects occupies <strong>several</strong> seats, proportional to the estimated object count — that is Zhou’s slip asking how many crates are in the granary. A write request keeps occupying seats, and more of them, for some time <strong>after</strong> the write completes, because it must fan notifications out to every <code class="language-plaintext highlighter-rouge">watch</code> stream, and the volume of that fan-out participates in the estimate. A <code class="language-plaintext highlighter-rouge">watch</code> request occupies its seat only until <strong>the initial burst of notifications for pre-existing objects</strong> is delivered, then sits without a seat. And long-running requests — <code class="language-plaintext highlighter-rouge">exec</code>, log <code class="language-plaintext highlighter-rouge">tail</code> — <strong>never enter APF at all</strong>: no seat, no queue.</p>

  <p><strong>Three: priority levels (<code class="language-plaintext highlighter-rouge">PriorityLevelConfiguration</code>).</strong> Each level has its own concurrency ceiling. The default configuration ships several: <code class="language-plaintext highlighter-rouge">node-high</code> (node health updates), <code class="language-plaintext highlighter-rouge">system</code> (non-health requests from the <code class="language-plaintext highlighter-rouge">system:nodes</code> group — kubelets), <code class="language-plaintext highlighter-rouge">leader-election</code> (built-in controllers competing for <code class="language-plaintext highlighter-rouge">leases</code>/<code class="language-plaintext highlighter-rouge">endpoints</code>), <code class="language-plaintext highlighter-rouge">workload-high</code> (other built-in controller requests), <code class="language-plaintext highlighter-rouge">workload-low</code> (any other ServiceAccount, typically controllers inside Pods), <code class="language-plaintext highlighter-rouge">global-default</code> (everything else, such as a human typing <code class="language-plaintext highlighter-rouge">kubectl</code>). Two more are mandatory: <code class="language-plaintext highlighter-rouge">catch-all</code> (a very small share, and it <strong>does not queue</strong>) and <code class="language-plaintext highlighter-rouge">exempt</code> (entirely unlimited; the <code class="language-plaintext highlighter-rouge">system:masters</code> group lands here by default).</p>

  <p>Giving <code class="language-plaintext highlighter-rouge">leader-election</code> its own level is not fussiness: a failed election restarts the controller, and a restarted controller comes back and re-syncs its informers, which costs orders of magnitude more traffic than the election itself.</p>

  <p><strong>Four: lending and reclaiming.</strong> A level’s ceiling is written as <code class="language-plaintext highlighter-rouge">nominalConcurrencyShares</code>, not as an absolute seat count, so raising total concurrency raises every level by the same proportion. Dynamic limits are recomputed periodically: each level first reclaims seats it lent and now needs, then the rest is apportioned by recent demand. Bounds are set by lending and borrowing percentages of the nominal limit. <strong>Borrowing is not preemption</strong>: a request already executing is never cancelled. In an apiserver, preemption means half-committed writes plus client retries, which is far worse than a wait.</p>

  <p><strong>Five: queueing, shuffling, fairness.</strong> Within a level, requests are distributed across queues (<code class="language-plaintext highlighter-rouge">queues</code>) by <strong>flow</strong> — flow = the matching FlowSchema’s name plus a <strong>distinguisher</strong> (the requesting user, the target resource’s namespace, or nothing). Distribution uses <strong>shuffle sharding</strong>: each flow hashes out <code class="language-plaintext highlighter-rouge">handSize</code> queues from the <code class="language-plaintext highlighter-rouge">queues</code> available and joins the shortest of those. Between queues, dispatch is <strong>fair queuing</strong> — take the queue furthest behind in virtual finish time.</p>

  <p>The two numbers trade off concretely: a larger <code class="language-plaintext highlighter-rouge">handSize</code> makes two flows less likely to collide (so neither can starve the other) but makes it likelier that a few flows can dominate the server; <strong>a single flow can queue at most <code class="language-plaintext highlighter-rouge">handSize × queueLengthLimit</code> requests</strong>. A larger <code class="language-plaintext highlighter-rouge">queues</code> lowers collision rate at the cost of memory; <code class="language-plaintext highlighter-rouge">queues = 1</code> effectively disables fair queuing and leaves only queueing.</p>

  <p><strong>Six: what happens to the excess.</strong> The level’s <code class="language-plaintext highlighter-rouge">limitResponse</code> decides: <code class="language-plaintext highlighter-rouge">Reject</code> returns 429 immediately; <code class="language-plaintext highlighter-rouge">Queue</code> queues, and if the queue is full or the wait too long, also ends in 429 with <code class="language-plaintext highlighter-rouge">Retry-After</code>. The <code class="language-plaintext highlighter-rouge">reason</code> label on <code class="language-plaintext highlighter-rouge">apiserver_flowcontrol_rejected_requests_total</code> separates the causes: <code class="language-plaintext highlighter-rouge">queue-full</code>, <code class="language-plaintext highlighter-rouge">concurrency-limit</code>, <code class="language-plaintext highlighter-rouge">time-out</code>, <code class="language-plaintext highlighter-rouge">cancelled</code>.</p>

  <h3 id="why-it-matters">Why it matters</h3>

  <p><strong>One, concurrency is not throughput.</strong> The quay is serial: an etcd write goes through the leader one commit at a time. Past a knee, more in-flight requests do not raise the completion rate; they raise per-commit latency, and then the completion rate <strong>falls</strong>. That is congestion collapse in a queueing network — not a metaphor for overload, its definition. <em>Little’s law</em> puts it plainly: <code class="language-plaintext highlighter-rouge">L = λW</code>. The seat count <code class="language-plaintext highlighter-rouge">L</code> is fixed; force <code class="language-plaintext highlighter-rouge">λ</code> up and <code class="language-plaintext highlighter-rouge">W</code> rises proportionally, and <code class="language-plaintext highlighter-rouge">W</code> then eats <code class="language-plaintext highlighter-rouge">λ</code>. APF’s core move is therefore <strong>limit plus queue</strong>, not more concurrency. That is its fundamental disagreement with “just raise <code class="language-plaintext highlighter-rouge">--max-requests-inflight</code>.”</p>

  <p><strong>Two, one bad client can take down a whole cluster, and it will look like something else.</strong> A hot-looping controller saturates the apiserver, and the symptom appears not on the apiserver but on <strong>nodes going <code class="language-plaintext highlighter-rouge">NotReady</code></strong>, on evictions, on pods failing to start. Operations sees node flapping; the root cause is a <code class="language-plaintext highlighter-rouge">for {}</code> in somebody’s reconcile loop. This is among the most common misdirections in Kubernetes incidents. APF makes it <strong>attributable</strong>: <code class="language-plaintext highlighter-rouge">apiserver_flowcontrol_*</code> metrics carry <code class="language-plaintext highlighter-rouge">flow_schema</code> and <code class="language-plaintext highlighter-rouge">priority_level</code> labels, so who is saturating the server is visible at a glance — provided the distinguisher is chosen well (<code class="language-plaintext highlighter-rouge">ByUser</code>, not nothing). The documentation is blunt about it: <strong>controllers with many instances should authenticate with distinct usernames</strong>, or they are all counted as one flow and fair queuing does nothing for them.</p>

  <p><strong>Three, the default configuration is editable, and probably should be edited.</strong> What ships is a <em>reasonable</em> set, not a set that fits you. Three specific traps: the defaults give kubelet local health checks (<code class="language-plaintext highlighter-rouge">/healthz</code>, <code class="language-plaintext highlighter-rouge">/livez</code>, <code class="language-plaintext highlighter-rouge">/readyz</code>, on the secure port but unauthenticated) no special treatment, so they land in <code class="language-plaintext highlighter-rouge">global-default</code> and get crowded out; <code class="language-plaintext highlighter-rouge">catch-all</code> does not queue, so it is 429 by default; and under APF <code class="language-plaintext highlighter-rouge">--max-requests-inflight</code> no longer means a ceiling — it means a numerator.</p>

  <p><strong>Four, it is designed to fail in layers, not to not fail.</strong> When one trade house held five hundred and eighty of six hundred seats, the ford did not collapse — it merely stopped Old Chen. <code class="language-plaintext highlighter-rouge">exempt</code> exists for exactly one reason: if the notice itself is written wrong, somebody must still be able to get in and rewrite it. Admitting you can be misconfigured, and leaving a way through for the misconfiguration, is more reliable than pretending you cannot be.</p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li>One quay on the far bank, one boat at a time, ledger entry before the next boat → etcd writes are committed serially by the leader; that is the apiserver’s real bottleneck</li>
    <li>A seat = one place from cast-off to ledger entry → a <em>seat</em> = one unit of concurrency held by one in-flight request</li>
    <li>The old rule: four hundred seats for people, two hundred for goods, first come first served, no shed limit → <code class="language-plaintext highlighter-rouge">--max-requests-inflight</code> 400 and <code class="language-plaintext highlighter-rouge">--max-mutating-requests-inflight</code> 200, globally shared, no queueing</li>
    <li>Zhou hears “once every half hour” as “keep looking”, twelve thousand trips a day → a hot-looping controller (tight reconcile / list-watch loop) saturating the apiserver</li>
    <li>Every unload copies the slip for every watcher, so more watchers means slower unloads → write requests pay seat-time for <code class="language-plaintext highlighter-rouge">watch</code> fan-out, and fan-out volume feeds the seat estimate</li>
    <li>Watchers hold a seat only while the old ledger is copied the first time → <code class="language-plaintext highlighter-rouge">watch</code> releases its seat once the initial object burst is delivered</li>
    <li>Towing, watching a man work: no seat, no queue → <code class="language-plaintext highlighter-rouge">exec</code> and log <code class="language-plaintext highlighter-rouge">tail</code> bypass APF entirely</li>
    <li>One slip asking how many crates there are occupies many seats → <code class="language-plaintext highlighter-rouge">list</code> requests occupy seats proportional to estimated object count</li>
    <li>The more it jams the slower it gets; 200 loads a quarter hour becomes 60 → congestion collapse: past the knee, concurrency up and throughput down (<code class="language-plaintext highlighter-rouge">L = λW</code>)</li>
    <li>Old Chen cannot report for four hours; the bank is judged lost and grain stops → kubelet heartbeat starved → node <code class="language-plaintext highlighter-rouge">NotReady</code> → eviction → more requests</li>
    <li>A cap of six hundred: the ford stands, Old Chen still cannot cross → a single global max-inflight protects the server and no particular request</li>
    <li>Peace report / guild documents / magistrate business / daily goods / onlookers / unclassifiable → <code class="language-plaintext highlighter-rouge">node-high</code>/<code class="language-plaintext highlighter-rouge">system</code>, <code class="language-plaintext highlighter-rouge">leader-election</code>, <code class="language-plaintext highlighter-rouge">workload-high</code>, <code class="language-plaintext highlighter-rouge">workload-low</code>, <code class="language-plaintext highlighter-rouge">global-default</code>, <code class="language-plaintext highlighter-rouge">catch-all</code></li>
    <li>The last grade holds a tiny share and gets no lanes, refused on the spot → <code class="language-plaintext highlighter-rouge">catch-all</code> has a very small share and does not queue, returning 429</li>
    <li>The imperial warrant: no seat, no queue, because someone must be able to fix a broken register → the <code class="language-plaintext highlighter-rouge">exempt</code> level, <code class="language-plaintext highlighter-rouge">system:masters</code> included, so a bad flow-control config cannot lock the apiserver out</li>
    <li>The register grades by who sent it, what business, which granary → FlowSchema <code class="language-plaintext highlighter-rouge">subjects</code>, <code class="language-plaintext highlighter-rouge">resourceRules</code>/<code class="language-plaintext highlighter-rouge">nonResourceRules</code>, <code class="language-plaintext highlighter-rouge">namespaces</code>/<code class="language-plaintext highlighter-rouge">nonResourceURLs</code></li>
    <li>Numbered register, read from the lowest number, first match wins → <code class="language-plaintext highlighter-rouge">matchingPrecedence</code>: lowest number first, first match applies</li>
    <li>Six hundred seats divided by share, shares being proportions not counts → <code class="language-plaintext highlighter-rouge">nominalConcurrencyShares</code> apportion total concurrency and scale with it</li>
    <li>The peace-report grade holds the largest share → <code class="language-plaintext highlighter-rouge">system</code> (kubelets) gets one of the largest defaults, because it is the cluster’s pulse</li>
    <li>Idle grades lend but never more than three tenths, so they can take seats back at once → lending and borrowing bounds as percentages of the nominal limit, recomputed periodically, reclaim before reallocate</li>
    <li>Nobody already aboard is thrown off; throwing a man off costs more than a wait → no preemption: cancelling an in-flight write leaves half-committed state, worse than queueing</li>
    <li>One hundred and twenty-eight lanes; a house lands in six by the number of its name → <code class="language-plaintext highlighter-rouge">queues</code> and <code class="language-plaintext highlighter-rouge">handSize</code> shuffle sharding, bounding one flow’s blast radius to <code class="language-plaintext highlighter-rouge">handSize</code> queues</li>
    <li>Six lanes full still leaves the other hundred and twenty-two untouched → one flow queues at most <code class="language-plaintext highlighter-rouge">handSize × queueLengthLimit</code> requests</li>
    <li>Each departure taken from the lane furthest behind on virtual time → fair queuing dispatches by virtual finish time; neither loudness nor capital gets ahead</li>
    <li>Two hands of one house walk separate lanes and neither starves → the distinguisher (<code class="language-plaintext highlighter-rouge">ByUser</code> / <code class="language-plaintext highlighter-rouge">ByNamespace</code>) sets the granularity of fairness</li>
    <li>A full lane is refused with “come back in so many minutes” → 429 with <code class="language-plaintext highlighter-rouge">Retry-After</code>; <code class="language-plaintext highlighter-rouge">rejected_requests_total</code> splits <code class="language-plaintext highlighter-rouge">queue-full</code>/<code class="language-plaintext highlighter-rouge">concurrency-limit</code>/<code class="language-plaintext highlighter-rouge">time-out</code>/<code class="language-plaintext highlighter-rouge">cancelled</code></li>
    <li>A queue that never refuses becomes a queue that times out, and the timed-out still hold seats → refusal beats pile-up: queue timeouts occupy seats and trigger retry storms</li>
    <li>Zhou stuck in his own six lanes, refused clearly, halves his trips by morning → 429 plus <code class="language-plaintext highlighter-rouge">Retry-After</code> lets clients self-correct; metrics make blame attributable</li>
    <li>The line beside the notice: six hundred seats are the quay’s limit, not the ford’s → total concurrency is set by the bottleneck, not by ambition; raising max-inflight only relocates the jam</li>
  </ul>
</section>]]></content><author><name></name></author><category term="kubernetes" /><category term="performance" /><summary type="html"><![CDATA[青石渡在沉沙河西岸。对岸只有一座埠头，叫独石埠。]]></summary></entry><entry><title type="html">谁都能立一块路牌 · Anyone May Plant a Signpost</title><link href="https://fables.day/2026/11/08/bgp-origin-validation-rpki/" rel="alternate" type="text/html" title="谁都能立一块路牌 · Anyone May Plant a Signpost" /><published>2026-11-08T00:00:00+00:00</published><updated>2026-11-08T00:00:00+00:00</updated><id>https://fables.day/2026/11/08/bgp-origin-validation-rpki</id><content type="html" xml:base="https://fables.day/2026/11/08/bgp-origin-validation-rpki/"><![CDATA[<section class="zh">
  <p>从悬河渡往西，翻三道岭，是一条走了一千年的驿道。</p>

  <p>道上一共四十七家驿铺，铺子之间没有上下之分，谁也不管谁。规矩只有一条，是几百年里自己长出来的：<strong>每个铺子在门口立一块木牌，写上”本铺可发往某地”；脚夫到了岔口，看哪块牌子写的地方跟信上的地址对得上，就把信交给哪一铺。</strong></p>

  <p>这条规矩能成，靠的是一个补丁式的细节：<strong>牌子写得越细，越算数。</strong> 一块写”盐铁镇”的牌，和一块写”盐铁镇第七坊”的牌，同时出现在一个岔口，脚夫一定走后者。谁写得细，信就归谁。</p>

  <p>一千年来，从来没有人问过一句话：<strong>这块牌子，是谁准你立的？</strong></p>

  <p>盐铁镇的盐引、汇通号的汇票、官府的火漆札子，全都走这条路。旺季一天四万封。</p>

  <p>出事那年秋天，断崖口多了一家铺子。</p>

  <p>铺主姓侯，排行第三，早年在盐铁镇做过账，认得盐铁镇的路数。他在断崖口租了半间土房，门口立了一块牌，用旧墨描了五个字：</p>

  <p><strong>盐铁镇第七坊。</strong></p>

  <p>岔口上原本那块牌写的是”盐铁镇”，粗。侯三的牌，细。于是那一天发往第七坊的全部信件——盐引、汇票、札子——一封不落，全进了侯三的土房。</p>

  <p>侯三没抢，也没撕。他只是把信拆开，抄一遍，再原样封好，隔三天缓缓发出去。抄一遍，汇票上的数额、盐引上的批号、札子上的人名，全在他手里了。</p>

  <p>官府来查，只查到一件事：<strong>那块牌子，不假。</strong></p>

  <p>因为这条路从来就没有”假牌”这回事。谁都可以立牌，立了就算。侯三甚至不需要撒谎，他只需要挑一个没人写过那么细的地方，写得更细一点。</p>

  <p>那年秋天，盐铁镇丢了十一万两银子的汇票。</p>

  <p><strong>第一个办法是派人守岔口。</strong></p>

  <p>盐铁镇雇了二十个拳师，分守在通往镇上的五个岔口，见陌生牌子就砸。砸了一个月，砸了十一块，信照旧丢。</p>

  <p>因为路是传递的。信从断崖口进来，侯三并不需要直接把信送到第七坊——他把信交给下家，下家再交给下下家。每个岔口守着的拳师只看得见自己眼前那一段路，而信在四十七家铺子之间转手六七道。田家的信从南岭来，先到中路驿，再到西关驿，最后从断崖口进镇。断崖口那块牌子，拳师砸了七次，侯三立了八次。</p>

  <p>砸不动。这条路的问题不在岔口，在<strong>每一家铺子都默认相信上家给它的牌</strong>。</p>

  <p><strong>第二个办法是总驿署的关防。</strong></p>

  <p>总驿署在省城，管着全道的驿册。它想出的是另一条路：不靠砸，靠<strong>备案</strong>。</p>

  <p>每家铺子每年到总驿署立一份<strong>关防</strong>。关防上写三样东西，缺一不可：</p>

  <ul>
    <li>一是这个铺子<strong>代收哪一段地方</strong>，写到镇为止（比如”盐铁镇全域”）；</li>
    <li>二是这块地方<strong>只有哪一家铺子可以代收</strong>，写上那家铺子的名字，一个名字，不许改；</li>
    <li>三是<strong>最细能到哪一层</strong>：盐铁镇的关防上写着”至镇为限，坊巷不在其内”。</li>
  </ul>

  <p>关防由总驿署用火漆封好，一份留在省城，一份抄成一册，叫<strong>关防册</strong>。册子由塘报发往全道每一个岔口的<strong>关验吏</strong>，每半个时辰补一次新条目。</p>

  <p>关验吏手里只有这一本册子，和一句话：<strong>你只验一块牌上的最后一个名字。</strong></p>

  <p>从此岔口上多了一道工序。牌立起来，关验吏翻册子，只有三种结果：</p>

  <ul>
    <li><strong>有案可稽，且牌上的名字跟关防上写的对得上</strong> —— 准。牌子挂起来，信优先往这条路上发。</li>
    <li><strong>有案可稽，但名字对不上</strong> —— 斥。牌子当场拆掉，信退回去。这一条杀死了绝大多数伪造。</li>
    <li><strong>册子上查无此案</strong> —— 依老规矩，放行。</li>
  </ul>

  <p>侯三的牌子就是在这个秋天被拆的。关验吏翻开册子，盐铁镇的关防明明白白写着”至镇为限，坊巷不在其内”。侯三写的是”第七坊”，已经越过了关防准许的最细一层。<strong>斥。</strong></p>

  <p>牌子拆了，人也押走了。盐铁镇那一年冬天睡得踏实。</p>

  <p><strong>但第二年春天，信又开始走了不该走的路。</strong></p>

  <p>这一次没有假牌。这一次每一块牌都是真的。</p>

  <p>事情出在中路驿。中路驿是西关驿的老客户，每年按季结一次驿费——它的信，本来就该从西关驿过。可中路驿同时也在替北岭驿发信：北岭驿的信路过中路驿，本该往北走，回北岭自己那片山里；中路驿却把那些信一并塞进了往西关驿的口袋。</p>

  <p>牌上写的是”盐铁镇”，名字是真的，关防对得上，<strong>准</strong>。</p>

  <p>关验吏放行了。他按规矩只验了最后一个名字，而这个规矩是他被教会的：一块牌要证明的，只是”这些信确实是盐铁镇发出的”。</p>

  <p>它从来不证明<strong>这些信是怎么走到这儿的</strong>。</p>

  <p>于是整整一个春天，北岭那片山谷的信，全都从西关驿绕了一个大圈。北路三家铺子一封也收不到，驿费白交；西关驿被硬塞了四倍的量，先是慢，后来是塌——雪化那半个月，西关驿门口积了九千封信，铺子当天关门，从悬河渡到盐铁镇，整条道的信停了四天。</p>

  <p>这一种，比侯三厉害得多。侯三要伪造一个身份；中路驿什么都不用伪造，它只需要<strong>把别人的信走一条不该走的路</strong>。前者是撒谎，后者是占道。</p>

  <p>而关防管不住占道，因为<strong>关防管的是名字，不是路</strong>。</p>

  <p><strong>第三个办法，是给路上每一段也备案。</strong></p>

  <p>总驿署最后补的那一册，叫<strong>《上游名册》</strong>。</p>

  <p>每一家铺子除了立关防，还要再立一份：<strong>我这一铺，只从哪几家上游铺子领信。</strong> 别家的信我不接，我也不往别家转。这份名册同样火漆封存，同样进塘报。</p>

  <p>关验吏从此要看两样东西，看牌上的整条路线：</p>

  <ul>
    <li>牌上那个<strong>最后的</strong>名字，对得上关防吗？（信是谁发的）</li>
    <li>牌上<strong>每一段路的上下两家</strong>，对得上名册吗？（路是谁准的）</li>
  </ul>

  <p>第二条一加，中路驿的把戏当场露了：牌上”中路驿→西关驿”这一段，西关驿的名册里，中路驿不在其上游之列。<strong>斥。</strong></p>

  <p>可是到了这一步，官府也终于看清了一件事：<strong>这套东西并不能拦住所有的坏牌，它只能拦住”写着别人名字的坏牌”。</strong></p>

  <p>册子上查无此案的，依老规矩，放行。</p>

  <p>全道四十七家铺子里，按时立了关防的只有三十一家。另外十六家嫌麻烦——立关防要跑一趟省城，要交火漆钱，要每年续。它们不立，它们的信照旧走，关验吏照旧放行，只是排在后面。</p>

  <p>也就是说，最容易被伪造的那一片地方，恰恰是没人去备案的那一片。</p>

  <p>而且塘报每半个时辰才补一次。侯三真正该挑的日子，不是秋天，是塘报刚送出、册子还没到的那半个时辰。</p>

  <p>至于那块写着”盐铁镇第七坊”的木牌——它现在被钉在省城驿署的墙上，作为一条教训挂着。牌上那五个字是假的，可在那天之前，<strong>没有任何一枚火漆证明过它是假的</strong>，因为这条路从设计的第一天起，就只保证了”信能被送到”，没保证过”信是谁发的”。</p>

  <p>到这儿，你大概已经认出来了：这就是 <strong>BGP 路由起源验证</strong>（<em>BGP route origin validation</em>，<em>RPKI</em> 加 <em>ROV</em>），以及它旁边那两件绕不开的事——<strong>路由泄漏</strong>（<em>route leak</em>）与 <strong>ASPA</strong>（<em>Autonomous System Provider Authorization</em>）。</p>

  <h3 id="section">这是什么</h3>

  <p>先说旧的世界。</p>

  <p>互联网由几万个自治系统（<em>Autonomous System</em>，<em>AS</em>）拼成。每个 AS 拥有一段或几段 IP 地址（<em>prefix</em>，前缀），它向邻居宣告：”这段地址在我这儿，发给我。” 邻居收下，改一改，再往外宣告。这就是 <em>BGP</em>。</p>

  <p>BGP 的原初设计里<strong>没有任何起源认证</strong>。收到一条宣告，你能确认的只有一件事：它是从邻居那儿来的。至于”这段地址真的属于宣告它的那个 AS 吗”，协议一个字都没说。路由表因此是一张<strong>默认信任</strong>的表。</p>

  <p>再加上一条转发规则，问题就被放大了：<strong>最长前缀匹配</strong>（<em>longest prefix match</em>）。一个 AS 合法地宣告了 <code class="language-plaintext highlighter-rouge">203.0.113.0/16</code>，攻击者宣告一条 <code class="language-plaintext highlighter-rouge">203.0.113.7/24</code>——更具体的那条永远赢。他不需要覆盖整个 /16，只需要挑一个 /24 写得更细。这就是侯三的五个字。</p>

  <p><strong>RPKI</strong>（<em>Resource Public Key Infrastructure</em>）的做法是给”牌子”配一本<strong>备案册</strong>。</p>

  <p>它的信任链是自顶向下的：IANA 是根（<em>trust anchor</em>）→ 五大区域注册局（RIR）→ 国家级注册局 → 会员单位（ISP、云厂商、企业）→ 具体的前缀持有者。每一层用自己的私钥为下一层签一张证书，声明”我授权你持有这一段地址”。整棵树是一套 <em>X.509</em> 证书体系，公开可校验。</p>

  <p>叶子节点上签发的，就是 <strong>ROA</strong>（<em>Route Origin Authorization</em>，RFC 6482）。一个 ROA 只有三个字段：</p>

  <ul>
    <li><code class="language-plaintext highlighter-rouge">prefix</code> —— 这段地址（”盐铁镇”）</li>
    <li><code class="language-plaintext highlighter-rouge">maxLength</code> —— 这份授权最细覆盖到多少位（”至镇为限”）</li>
    <li><code class="language-plaintext highlighter-rouge">origin AS</code> —— 唯一被授权宣告它的 AS（”只有这一家可以代收”）</li>
  </ul>

  <p>独立的 <em>RPKI validator</em> 把所有 ROA 汇总、逐条验签，产出一个集合，叫 <strong>VRP</strong>（<em>Validated ROA Payload</em>）。VRP 通过 <strong>RTR 协议</strong>（<em>RPKI-to-Router</em>，RFC 8210，增量推送）送到路由器上——这就是塘报。</p>

  <p>路由器对每一条收到的 BGP 宣告做一次判定，只有三种结果，术语叫 <strong>ROV</strong>（<em>Route Origin Validation</em>，RFC 6811）：</p>

  <ul>
    <li><strong><em>Valid</em></strong> —— 存在覆盖该前缀的 ROA，<code class="language-plaintext highlighter-rouge">maxLength</code> 也够，且 AS_PATH 末位的 origin 与 ROA 一致：接受，可以调高 <em>local-pref</em>。</li>
    <li><strong><em>Invalid</em></strong> —— 存在覆盖该前缀的 ROA，但 origin 不符，或者前缀长度超过 <code class="language-plaintext highlighter-rouge">maxLength</code>：<strong>丢弃 / 降级</strong>。全部收益都在这一格里。</li>
    <li><strong><em>NotFound</em></strong> —— 根本没有任何 ROA 覆盖它：<strong>按传统规则接受</strong>，只是不给优先级。</li>
  </ul>

  <p>注意第三行。ROV 的默认动作是”没备案就放行”，不是”没备案就拒绝”。这不是疏忽，是部署现实：一步走到”白名单”会让半个互联网在一天内断掉。代价是明确的——<strong>攻击者最理性的选择，永远是去打那片没有备案的地方。</strong></p>

  <p>而 <code class="language-plaintext highlighter-rouge">maxLength</code> 是这套体系里最容易被写松的一格。写成 <code class="language-plaintext highlighter-rouge">/16-/24</code> 就等于说”这段地址下面任何一条 /24 都算我授权的”，于是别人宣告一条 /24，判定是 <em>Valid</em>，不是 <em>Invalid</em>。收得越松，越安全的是自己的可用性，越危险的是别人的。</p>

  <p>还有一件关防永远做不到的事：<strong>ROV 只看 AS_PATH 的最后一个 AS。</strong></p>

  <p>它回答的是”这封信是谁发的”，不是”这封信是怎么走到这儿的”。于是有了另一类事故——<strong>路由泄漏</strong>：一个 AS 把从上游 A 学到的路由，转手告诉了上游 B。前缀是真的，origin 是真的，签也验得过，只是这条路本来不该存在。它的杀伤力和前缀劫持同级，甚至更大：2019 年那家把自家优化器配置推错、把全网的 Google 流量吸进来两小时的 ISP，用的就是这个机制，全程没有一条伪造的宣告。</p>

  <p><strong>ASPA</strong> 就是补这一格的：AS 到 RPKI 里再签一份”我授权的上游是谁”的清单，验证器沿 AS_PATH 逐跳检查上下家关系是否被授权。它检的是路，不是名字。</p>

  <p>顺带把两个词分清楚，它们在新闻里常被混着写：</p>

  <ul>
    <li><strong>前缀劫持</strong>（<em>hijack</em>）：宣告不属于自己的前缀，或用自己的 AS 冒充别人的 origin —— 侯三。</li>
    <li><strong>路由泄漏</strong>（<em>leak</em>）：宣告一条自己有资格宣告、但没有资格传出去的路由 —— 中路驿。</li>
  </ul>

  <h3 id="section-1">为什么重要</h3>

  <p>原因只有一句：<strong>整个互联网的转发，建立在一个没有认证的协议上，而认证是二十年后才补上去的一层外挂。</strong></p>

  <p>三个直接的后果：</p>

  <p><strong>一、更具体的宣告永远赢，这使”小规模”的攻击变得极其便宜。</strong> 2008 年巴基斯坦电信为封锁国内 YouTube，宣告了一条 /24，结果这条更具体的路由顺着 BGP 扩散到全世界，把 YouTube 全球流量吸了过去、黑洞了两小时——不是攻击，是一次配置失误。2018 年，攻击者用同样的手法，通过劫持一家云厂商的 DNS 服务所依赖的路由，把 myetherwallet.com 的解析导向了自己的服务器。这两件事的机制，和侯三那五个字完全一样。</p>

  <p><strong>二、ROV 的收益是真实的，但它的覆盖面才是它的上限。</strong> <em>Invalid</em> 一旦被丢弃，绝大多数起源伪造立刻失效，这是这些年最大的一次互联网加固。但剩下的 <em>NotFound</em> 是一大片默认放行的空间，而部署速度受制于一件很朴素的事：<strong>得有人愿意跑一趟省城，交那笔火漆钱。</strong> 这也是为什么几乎所有 RIR 都在把 ROA 签发做成点几下鼠标的事，为什么大型云厂商在替客户代签。</p>

  <p><strong>三、它把一个旧直觉推翻了：路由是有权威的。</strong> 长期以来”路由”被当成一个纯技术事实——网络说这条路径通，它就通。RPKI 把它变成一个<strong>授权事实</strong>：这条路径通不通，取决于某个注册体系里有没有一枚签过字的记录。今天你在云上给一段弹性 IP 配 BGP 宣告，背后就挂着这条链：前缀在谁的证书下、origin 是谁、<code class="language-plaintext highlighter-rouge">maxLength</code> 写多宽。</p>

  <p>这条链的每一环都是薄的，但缺任何一环，剩下的都不成立。这和 <em>mTLS</em> 的证书链、<em>DNSSEC</em> 的委派链、OIDC 的 <em>JWKS</em> 是同一个形状：<strong>信任不是”我知道你是你”，信任是”有人为我签过字，而我能一路查到根”。</strong></p>

  <p><em>隐喻对应表</em></p>

  <ul>
    <li>每个驿铺立块木牌写明”可发往某地”，脚夫照牌走 → BGP 宣告：每个 AS 向邻居宣告自己持有并服务的前缀</li>
    <li>岔口上没有任何人核验牌子的来处 → BGP 原初设计不含起源认证，路由表默认信任</li>
    <li>“牌子写得越细越算数” → 最长前缀匹配（longest prefix match）</li>
    <li>侯三立一块”盐铁镇第七坊”，盖住别人的”盐铁镇” → 前缀劫持（hijack）：宣告更具体的 /24 吸走 /16 的流量</li>
    <li>路人只看得见自己眼前那段路 → 逐跳信任：每个 AS 只信任邻居递过来的路由</li>
    <li>信在四十七家铺子间转手六七道 → 出问题时定位极难，路径在途、随时变</li>
    <li>田家的信经中路驿、西关驿到断崖口 → AS_PATH：一条路由经过的 AS 序列</li>
    <li>拳师守五个岔口，砸一块立一块 → 用 ACL/前缀过滤逐点封堵，事实上不可维护</li>
    <li>总驿署的关防：本铺代收哪一段、唯一代收者、最细到哪一层 → ROA 的三个字段：prefix、origin AS、maxLength</li>
    <li>关防一年一签、火漆封存、存根留底 → X.509 证书与 ROA 的签名体系</li>
    <li>关防从 IANA→RIR→国家注册局→会员→持有者逐层授权 → RPKI 的分层信任链（trust anchor 到 end entity）</li>
    <li>关防册汇总全道所有关防，塘报发到每个岔口 → VRP 集合由 validator 汇总，经 RTR 协议增量推送到路由器</li>
    <li>半个时辰补一次新条目 → RTR 增量刷新；册子到手之前是盲区</li>
    <li>关验吏手里只验牌上最后一个名字 → ROV 只校验 AS_PATH 末位的 origin AS</li>
    <li>三种结果：准 / 斥 / 依老规矩放行 → Valid / Invalid / NotFound</li>
    <li>“查无此案，依老规矩放行” → NotFound 默认接受，是全部残余风险的来源</li>
    <li>“至镇为限，坊巷不在其内” → maxLength 限制 ROA 覆盖的最大前缀长度</li>
    <li>maxLength 写成”凡其下属坊巷皆可” → maxLength 过宽，把本该 Invalid 的宣告变成 Valid</li>
    <li>按时立关防的只有三十一家，其余十六家放行但排后 → ROA 覆盖率不足，未覆盖空间是攻击面的主体</li>
    <li>立关防要跑一趟省城、交火漆钱 → RPKI 的部署成本与 ROA 签发的组织摩擦</li>
    <li>侯三牌子被拆，但全镇只睡了一个冬天 → 单点修补解决了劫持，却不解决机制本身</li>
    <li>中路驿把北岭驿的信塞进往西关驿的口袋 → 路由泄漏（route leak）：合法前缀、非法路径</li>
    <li>牌上的名字是真的、路是错的 → ROV 不检路径，只检名单</li>
    <li>西关驿一夜积九千封，全道停四天 → 泄漏造成的拥塞、丢包与大规模不可达</li>
    <li>关防管的是名字，不是路 → 起源验证与路径验证的能力边界</li>
    <li>《上游名册》：我这一铺只从哪几家领信 → ASPA：AS 签名的授权上游集合</li>
    <li>逐段核对上下家是否对得上名册 → ASPA 沿 AS_PATH 逐跳校验上下家关系</li>
    <li>关防被钉在省城墙上做教训 → 事故与 RFC 一起写进这套体系的记忆里</li>
  </ul>
</section>
<section class="en">
  <p>West from Suspended River Ford, over three ridges, runs a post road a thousand years old.</p>

  <p>Along it stand forty-seven relay posts. There is no hierarchy among them and no authority over them. Only one rule, grown up out of centuries by itself: <strong>each post plants a wooden board at its door reading “this post carries mail toward such-and-such a place”; a porter reaching a fork reads the boards, finds the one whose place matches the address on the letter, and hands the letter over there.</strong></p>

  <p>That rule works because of one patch-like detail: <strong>the more specific the board, the more it counts.</strong> A board reading “Salt-Iron Town” and a board reading “Salt-Iron Town, Seventh Ward” hanging at the same fork — the porter always takes the second. Whoever writes it finer, gets the mail.</p>

  <p>In a thousand years, nobody ever asked one question: <strong>who gave you permission to plant this board?</strong></p>

  <p>Salt certificates from Salt-Iron Town, drafts from the Exchange House, government pouches sealed in red wax — all of it travelled this road. In high season, forty thousand letters a day.</p>

  <p>The autumn it broke, a new post appeared at Broken Cliff Mouth.</p>

  <p>Its keeper was named Hou the Third. He had kept books in Salt-Iron Town years before and knew its ways. He rented half a mud hut at Broken Cliff Mouth and planted a board at the door, inked in old soot, five characters:</p>

  <p><strong>Salt-Iron Town, Seventh Ward.</strong></p>

  <p>The board already at that fork read “Salt-Iron Town” — coarse. Hou’s board was fine. So every letter bound for the Seventh Ward that day — salt certificates, drafts, pouches — went into Hou’s mud hut, not one lost.</p>

  <p>He robbed nothing and tore nothing. He opened each letter, copied it, sealed it back exactly, and sent it on after three days. Copying it put the amount on every draft, the batch number on every certificate, the names on every pouch, in his hands.</p>

  <p>The magistrate came and found exactly one thing: <strong>the board was not false.</strong></p>

  <p>Because on this road there had never been such a thing as a false board. Anyone may plant one, and a planted board counts. Hou did not even need to lie. He needed only to pick a place nobody had bothered to describe that finely, and describe it one notch finer.</p>

  <p>That autumn Salt-Iron Town lost drafts worth a hundred and ten thousand taels.</p>

  <p><strong>The first remedy was to post guards at the forks.</strong></p>

  <p>The town hired twenty boxers and set them at the five forks leading in, to smash any unfamiliar board. A month of smashing: eleven boards destroyed, and the letters kept vanishing.</p>

  <p>Because the road is transitive. From Broken Cliff Mouth, Hou never had to deliver a letter to the Seventh Ward himself — he handed it to the next post, which handed it to the next. A boxer at a fork sees only his own stretch of road, and letters changed hands six or seven times among the forty-seven posts. The Tian family’s letters came from the South Ridge, reached Middle Road Post, then West Gate Post, and finally entered town at Broken Cliff Mouth. The boxers smashed that board seven times. Hou planted it eight.</p>

  <p>Smashing could not win. The road’s problem was not at the forks. It was that <strong>every post believed, by default, the board its neighbour handed it.</strong></p>

  <p><strong>The second remedy was the Grand Relay Office’s seal-registers.</strong></p>

  <p>The Grand Relay Office sat in the provincial capital and held the road’s whole roll. What it devised was not smashing but <strong>filing</strong>.</p>

  <p>Every post came once a year and deposited a <strong>sealed register entry</strong>. It carried three things, and omitting any one voided it:</p>

  <ul>
    <li>first, <strong>which stretch this post collects for</strong>, described down to the town and no further (say, “the whole of Salt-Iron Town”);</li>
    <li>second, <strong>which single post may collect for that stretch</strong> — one name, unchangeable;</li>
    <li>third, <strong>the finest level permitted</strong>: Salt-Iron Town’s entry read “to the town as limit; wards and lanes not included.”</li>
  </ul>

  <p>The Office sealed each with red wax, kept one copy, and had the rest copied into a <strong>register book</strong>. Couriers carried the book to the <strong>validating clerk</strong> at every fork on the road, with new entries appended every half hour.</p>

  <p>The clerk had only that book, and one instruction: <strong>you verify the last name on the board, and nothing else.</strong></p>

  <p>So the forks gained one procedure. A board goes up, the clerk opens the book, and there are three outcomes:</p>

  <ul>
    <li><strong>On record, and the name on the board matches</strong> — allowed. The board hangs, and mail is dispatched down that road first.</li>
    <li><strong>On record, but the name does not match</strong> — refused. The board comes down and the letters go back. This killed the great majority of forgeries outright.</li>
    <li><strong>No record at all</strong> — by the old custom, passed through.</li>
  </ul>

  <p>Hou’s board came down that autumn. The clerk opened the book, and Salt-Iron Town’s entry said plainly: “to the town as limit; wards and lanes not included.” Hou had written “Seventh Ward,” one notch past the finest level the entry permitted. <strong>Refused.</strong></p>

  <p>The board came down, the man was taken away, and Salt-Iron Town slept well that winter.</p>

  <p><strong>But the next spring, letters began travelling roads they should never have travelled.</strong></p>

  <p>This time there was no false board. This time every board was true.</p>

  <p>It happened at Middle Road Post. Middle Road was West Gate’s old customer, settling its relay fee quarterly — its mail was supposed to come through West Gate. But Middle Road was also carrying for North Ridge: North Ridge’s letters passed through Middle Road and should have turned north, back into its own mountains; Middle Road instead stuffed them into the same sack headed for West Gate.</p>

  <p>The board said “Salt-Iron Town.” The name was real. The register matched. <strong>Allowed.</strong></p>

  <p>The clerk passed it. By the rule he had been taught, he had verified only the last name — and that rule rested on an assumption he never examined: that a board need only prove <strong>these letters really were sent by Salt-Iron Town</strong>.</p>

  <p>It never proved <strong>how these letters got here</strong>.</p>

  <p>So for a whole spring, the mail of an entire northern valley went the long way round through West Gate. Three posts on the north road collected nothing and paid their relay fees for air; West Gate took four times its capacity — slow at first, then collapsing. In the fortnight the snow melted, nine thousand letters piled at West Gate’s door, the post shut that same day, and from Suspended River Ford to Salt-Iron Town the whole road stopped for four days.</p>

  <p>This was far worse than Hou. Hou had to forge an identity. Middle Road forged nothing — it only <strong>put other people’s letters on a road they had no right to take.</strong> Hou lied. Middle Road trespassed.</p>

  <p>And no register could stop trespass, because <strong>the register governed names, not roads.</strong></p>

  <p><strong>The third remedy was to file a record for every stretch of the road as well.</strong></p>

  <p>The last volume the Grand Relay Office added was called the <strong>Upstream Roll</strong>.</p>

  <p>Beside its register entry, every post deposited a second document: <strong>these are the only upstream posts I take mail from.</strong> Another post’s mail I do not accept; to another post I do not forward. Sealed in wax the same way, carried in the same courier dispatch.</p>

  <p>The clerk now checked two things, against the whole route written on the board:</p>

  <ul>
    <li>does the <strong>final</strong> name on the board match a register entry? (who sent this)</li>
    <li>does <strong>every up-and-down pair along the route</strong> match the Upstream Rolls? (who permitted this road)</li>
  </ul>

  <p>With the second check in place, Middle Road’s trick fell apart at once: on the stretch “Middle Road → West Gate,” Middle Road did not appear in West Gate’s upstream list. <strong>Refused.</strong></p>

  <p>But by this stage the magistrate could finally see plainly: <strong>this apparatus cannot stop all bad boards. It can only stop bad boards that name someone else.</strong></p>

  <p>No record at all, and by the old custom it passes.</p>

  <p>Of the forty-seven posts, only thirty-one filed their entries on time. The other sixteen found it a nuisance — filing meant a trip to the capital, a wax fee, a yearly renewal. They did not file. Their mail moved the same as ever, the clerk passed it the same as ever, only further down the queue.</p>

  <p>Which is to say: <strong>the ground easiest to forge was precisely the ground nobody had filed for.</strong></p>

  <p>And the courier dispatch refreshed only every half hour. The day Hou should really have chosen was not in autumn. It was the half hour after the dispatch left and before the book arrived.</p>

  <p>As for the board reading “Salt-Iron Town, Seventh Ward” — it now hangs on the wall of the provincial relay office as a lesson. The five characters on it were false. But until that day, <strong>no drop of wax had ever attested that they were false</strong>, because from its first day this road guaranteed only that letters could be delivered. It never guaranteed who sent them.</p>

  <p>By now you have probably recognized it: this is <strong>BGP route origin validation</strong> — <em>RPKI</em> and <em>ROV</em> — together with the two things it cannot escape, <strong>route leaks</strong> and <strong>ASPA</strong> (<em>Autonomous System Provider Authorization</em>).</p>

  <h3 id="what-it-is">What it is</h3>

  <p>Start with the old world.</p>

  <p>The internet is assembled from tens of thousands of autonomous systems (<em>AS</em>). Each AS holds one or more blocks of IP addresses (<em>prefixes</em>) and announces to its neighbours: “this block is mine, send it to me.” A neighbour accepts, edits, announces onward. That is <em>BGP</em>.</p>

  <p>BGP’s original design contains <strong>no origin authentication whatsoever</strong>. From an announcement, the only thing you can confirm is that it came from your neighbour. Whether the block truly belongs to the AS announcing it is a question the protocol never addresses. The routing table is therefore a table of <strong>default trust</strong>.</p>

  <p>And one forwarding rule magnifies the problem: <strong>longest prefix match</strong>. An AS legitimately announces <code class="language-plaintext highlighter-rouge">203.0.113.0/16</code>; an attacker announces <code class="language-plaintext highlighter-rouge">203.0.113.7/24</code> — the more specific always wins. He does not need to take the whole /16. He needs one /24 described one notch finer. That is Hou’s five characters.</p>

  <p><strong>RPKI</strong> (<em>Resource Public Key Infrastructure</em>) answers by giving boards a <strong>register</strong>.</p>

  <p>The trust chain runs top-down: IANA is the root (<em>trust anchor</em>) → the five regional internet registries → national registries → member organizations (ISPs, cloud providers, enterprises) → the individual prefix holder. Each level signs a certificate for the level below with its own private key, declaring “I authorize you to hold this block.” The whole tree is an <em>X.509</em> hierarchy, publicly verifiable.</p>

  <p>What gets signed at the leaf is a <strong>ROA</strong> (<em>Route Origin Authorization</em>, RFC 6482). A ROA has exactly three fields:</p>

  <ul>
    <li><code class="language-plaintext highlighter-rouge">prefix</code> — the block (“Salt-Iron Town”)</li>
    <li><code class="language-plaintext highlighter-rouge">maxLength</code> — the finest length this authorization covers (“to the town as limit”)</li>
    <li><code class="language-plaintext highlighter-rouge">origin AS</code> — the single AS authorized to announce it (“only this post may collect”)</li>
  </ul>

  <p>A standalone <em>RPKI validator</em> gathers every ROA, verifies each signature, and emits a set called the <strong>VRP</strong> (<em>Validated ROA Payload</em>) set. VRPs reach routers over the <strong>RTR protocol</strong> (<em>RPKI-to-Router</em>, RFC 8210, pushed incrementally) — that is the courier dispatch.</p>

  <p>For each BGP announcement it receives, a router renders one of three verdicts, a procedure standardized as <strong>ROV</strong> (<em>Route Origin Validation</em>, RFC 6811):</p>

  <ul>
    <li><strong><em>Valid</em></strong> — a covering ROA exists, <code class="language-plaintext highlighter-rouge">maxLength</code> permits the length, and the last AS in AS_PATH matches the ROA origin: accept, possibly raising <em>local-pref</em>.</li>
    <li><strong><em>Invalid</em></strong> — a covering ROA exists, but the origin differs or the prefix is longer than <code class="language-plaintext highlighter-rouge">maxLength</code>: <strong>drop / de-prefer</strong>. All the benefit lives in this cell.</li>
    <li><strong><em>NotFound</em></strong> — no ROA covers it at all: <strong>accept by the old rules</strong>, simply without preference.</li>
  </ul>

  <p>Note the third row. ROV’s default action is “no record, then pass,” not “no record, then refuse.” That is not an oversight; it is deployment reality. Going straight to a whitelist would take half the internet down in a day. The cost is explicit: <strong>the attacker’s most rational choice is always the ground with no record.</strong></p>

  <p>And <code class="language-plaintext highlighter-rouge">maxLength</code> is the loosest cog in the machine. Writing <code class="language-plaintext highlighter-rouge">/16-/24</code> amounts to saying “any /24 inside this block counts as mine to announce,” so somebody else announcing one is judged <em>Valid</em>, not <em>Invalid</em>. The looser you write it, the safer your own availability and the more dangerous everyone else’s.</p>

  <p>There is one more thing no register can ever do: <strong>ROV inspects only the last AS of AS_PATH.</strong></p>

  <p>It answers “who sent this,” not “how did this get here.” Which gives the other class of accident — a <strong>route leak</strong>: an AS takes routes learned from upstream A and hands them to upstream B. The prefix is real, the origin is real, the signature verifies; only the road should not exist. Its blast radius rivals prefix hijacking, and often exceeds it: the 2019 incident in which an ISP pushed a bad optimizer configuration and pulled the world’s Google traffic into its network for two hours ran entirely on this mechanism, with not one forged announcement.</p>

  <p><strong>ASPA</strong> closes that gap: an AS signs, in RPKI, a list of <strong>the upstreams it authorizes</strong>, and validators check the up-and-down relationship hop by hop along AS_PATH. It inspects the road, not the names.</p>

  <p>Two terms worth separating, because the press runs them together:</p>

  <ul>
    <li><strong>hijack</strong> — announcing a prefix that is not yours, or impersonating another’s origin AS: Hou.</li>
    <li><strong>leak</strong> — announcing a route you are entitled to announce but not entitled to pass along: Middle Road Post.</li>
  </ul>

  <h3 id="why-it-matters">Why it matters</h3>

  <p>For one reason: <strong>the entire internet’s forwarding rests on a protocol with no authentication, and authentication arrived twenty years later as a bolt-on layer.</strong></p>

  <p>Three consequences follow directly.</p>

  <p><strong>One, more-specific always wins, which makes “small” attacks extremely cheap.</strong> In 2008 Pakistan Telecom, trying to block YouTube domestically, announced a /24; that more-specific route spread through BGP worldwide, pulled YouTube’s global traffic toward it, and blackholed it for two hours — not an attack, a misconfiguration. In 2018 an attacker used the same technique, hijacking routes belonging to a cloud provider’s DNS service, to steer resolution of myetherwallet.com to his own servers. The mechanism in both is exactly Hou’s five characters.</p>

  <p><strong>Two, ROV’s benefit is real, but its coverage is its ceiling.</strong> Once <em>Invalid</em> routes are dropped, the great majority of origin forgeries stop working, and it is one of the largest hardenings the internet has had. But the remaining <em>NotFound</em> is a broad expanse that passes by default, and deployment speed is bound by something mundane: <strong>someone has to be willing to make the trip to the capital and pay the wax fee.</strong> Which is why essentially every RIR has driven ROA issuance down to a few clicks, and why large cloud providers sign on their customers’ behalf.</p>

  <p><strong>Three, it overturns an old intuition: a route has an authority.</strong> For a long time a route was treated as a pure technical fact — the network says this path works, so it works. RPKI makes it an <strong>authorization fact</strong>: whether the path works depends on whether a signed record exists in some registry. When you configure a BGP announcement for an elastic IP on a cloud today, this chain hangs behind it: whose certificate the prefix sits under, which origin is named, how wide <code class="language-plaintext highlighter-rouge">maxLength</code> was written.</p>

  <p>Every link in that chain is thin, and without any one of them the rest does not hold. It is the same shape as an <em>mTLS</em> certificate chain, a <em>DNSSEC</em> delegation chain, or OIDC’s <em>JWKS</em>: <strong>trust is not “I know you are you.” Trust is “someone signed for you, and I can trace it to the root.”</strong></p>

  <p><em>Metaphor mapping</em></p>

  <ul>
    <li>Each post plants a board naming the places it carries toward, and porters follow the boards → BGP announcements: every AS advertises the prefixes it holds and serves</li>
    <li>Nobody at the forks verifies where a board came from → BGP’s original design has no origin authentication; the routing table runs on default trust</li>
    <li>“The finer the board, the more it counts” → longest prefix match</li>
    <li>Hou plants “Salt-Iron Town, Seventh Ward” over someone’s “Salt-Iron Town” → prefix hijack: a more-specific /24 swallowing a /16’s traffic</li>
    <li>A passer-by sees only his own stretch of road → hop-by-hop trust: each AS trusts only what its neighbour hands it</li>
    <li>Letters changing hands six or seven times among forty-seven posts → hard attribution, paths in flight, constantly changing</li>
    <li>The Tian family’s letters via Middle Road and West Gate to Broken Cliff Mouth → AS_PATH: the AS sequence a route traverses</li>
    <li>Boxers guarding five forks, one smashed and one replanted → point ACLs and prefix filters, unmaintainable in practice</li>
    <li>The Grand Relay Office’s entry: which stretch, which single collector, finest level allowed → the ROA’s three fields: prefix, origin AS, maxLength</li>
    <li>Signed yearly under red wax, copy retained → X.509 certificates and ROA signatures</li>
    <li>Authority flowing IANA → RIR → national registry → member → holder → RPKI’s hierarchical trust chain, trust anchor to end entity</li>
    <li>The register book gathering every entry, carried to each fork → the VRP set produced by a validator and pushed to routers over RTR</li>
    <li>New entries appended every half hour → RTR incremental refresh; the window before the book arrives is blind</li>
    <li>The clerk verifies only the last name on the board → ROV validates only the origin AS at the end of AS_PATH</li>
    <li>Three outcomes: allowed / refused / passed by old custom → Valid / Invalid / NotFound</li>
    <li>“No record, then by the old custom it passes” → NotFound accepted by default; the source of all residual risk</li>
    <li>“To the town as limit; wards and lanes not included” → maxLength bounding the longest prefix a ROA covers</li>
    <li>Writing maxLength as “any ward beneath it counts too” → an over-wide maxLength turning what should be Invalid into Valid</li>
    <li>Only thirty-one posts filed; sixteen passed but queued last → incomplete ROA coverage; uncovered space is the bulk of the attack surface</li>
    <li>Filing costs a trip to the capital and a wax fee → RPKI’s deployment cost and the organizational friction of issuing ROAs</li>
    <li>Hou’s board comes down and the town sleeps one winter → fixing the hijack does not fix the mechanism</li>
    <li>Middle Road stuffs North Ridge’s letters into West Gate’s sack → route leak: legitimate prefix, illegitimate path</li>
    <li>The name on the board is true and the road is wrong → ROV inspects the name list, not the path</li>
    <li>Nine thousand letters at West Gate overnight, the whole road down four days → congestion, loss, and large-scale unreachability caused by a leak</li>
    <li>The register governs names, not roads → the capability boundary between origin validation and path validation</li>
    <li>The Upstream Roll: the only posts I take mail from → ASPA: an AS’s signed set of authorized upstreams</li>
    <li>Checking each up-and-down pair against the Rolls → ASPA validating provider-customer relationships hop by hop along AS_PATH</li>
    <li>The board nailed to the capital’s wall as a lesson → incidents and RFCs written into the system’s memory</li>
  </ul>
</section>]]></content><author><name></name></author><category term="networking" /><category term="security" /><summary type="html"><![CDATA[从悬河渡往西，翻三道岭，是一条走了一千年的驿道。]]></summary></entry></feed>