Repository navigation
bun: arm64 build hangs intermittently and wedges build-bot (3 hangs since 09-19, latest 10-28 and 10-29) #1160
Description
bun's arm64 build hangs intermittently and wedges build-bot
When bun's arm64 build hangs, it hangs silently. It produces no output, never finishes, and build-bot sits on it until someone cancels the run. amd64 builds the same input in about 90 s. Any PR whose closure includes bun is stuck behind it, and on the serial build-bot that holds up everything else too.
| When | PR | amd64 | arm64 |
|---|---|---|---|
| 2026-09-19/20/21 | (three sandboxes) | ok | hung 6 h at zig obj -> bun-zig.{0..71}.o, no cache writes. Fixed then by capping zig's LLVM codegen shards at 8 on arm64 (build.sh) |
| 2026-10-28 20:55–21:29 | #1147 (go ladder; bun in the closure) | 88 s | silent for 35 min until cancelled; res-server-arm64: bun failed to build: sandbox error: Execution cancelled |
| 2026-10-29 17:25–17:42 | #1134 (bun system CA store) | 95 s | 0 packages built, build.events_stalled at 17:30:51, cancelled 17:42:35 |
In between, arm64 bun builds do succeed: 175–525 s on 10-27/28. So the hang is intermittent, not deterministic.
What we know
- It's not the network, as far as the logs show. Cloud NAT logged no drops in the 10-29 window (no
nat_gatewayrows). bun'sneeds.internetis forbun install. - The build log is invisible. Without
--verbose, a sandbox build streams nothing, so a hang looks exactly like a slow compile. - The September hang was zig codegen at 72 shards. The 8-shard cap (
scripts/build/zig.ts, verified by grep) made it rarer, not gone. Upstream release builds use 1 shard (full IPO); 8 is their ASAN CI value.
Proposed
- Fail fast instead of wedging. Wrap
bun run build:releaseintimeout(e.g. 45 min; a cold arm64 build is under 10 min) with a clear error. That way a hang turns into a fast, labelled failure rather than a stuck build-bot that needs a manual cancel. - Reprod
6316
uce once with the build log. Run
min session exec <s> -- min package build --rebuild --verbose bunon arm64 until it hangs, and record the last step (zig codegen, codegen scripts, cmake deps orbun install). - If it's codegen again, pin arm64 to upstream's release value of 1 shard. That's slower but known-good upstream.
🤖 Generated with Claude Code
Activity
bryan-minimal commentedon Oct 29, 2026
Mitigation step 1 is on #1134 (3a5c977):
bun run build:releasenow runs undertimeout --kill-after=60 3600, so a hang fails in 60 min withERROR: bun build:release hung ... (gominimal/pkgs#1160)instead of holding build-bot. A watchdog lists every live process (/proc/*/cmdline) at 55 min, so the next hang also gives us step 2's answer: the step it stuck in.Cold builds are 3-9 min on both arches, so the limit only fires on a hang.
🤖 Generated with Claude Code