Repository navigation
minimal: upload failure sometimes hangs the client indefinitely instead of erroring #886
Description
Summary
When a workspace upload fails, the client sometimes hangs indefinitely instead of returning an error. The daemon has already logged the failure and torn the connection down; the client never notices and never exits.
Observed four times during the gominimal/minimal#869 investigation, at ~2 min, ~9 min, 13 min, and 75 s — each killed by an external timeout, never by the client itself. The same command on the same fixture usually returns error: Failed to upload project files: copying tar stream to channel: channel closed within a few seconds, so this is an alternate outcome of the same failure, not a different one.
Reproduction
Same fixture as #869: a project whose single file compresses to ~12.78 MB, uploaded to a guest whose vsock connection is reset mid-transfer.
min activate --no-loadouts -n <name> <fixture>
Most runs error out; a minority hang. No timeout bounds the wait, so an interactive user sees Uploading project files... and nothing else, indefinitely.
Guest side
The daemon logs the connection dying, at the same point it does in the erroring case:
WARN conn{conn=N transport="vsock"}: minimald::server: session ended with error
error=Protocol error: No buffer space available (os error 105)
So the information needed to fail fast exists on the daemon side and reaches the transport; the client simply does not act on it.
Likely area
crates/minimal/src/file_upload.rs / crates/minimal/src/client.rs. The upload streams a tar into an SSH channel via tokio::io::copy with no deadline anywhere in the path. Candidate mechanisms, unconfirmed:
stream_tar_zstdspawns the tar builder and awaitsbuild.awaitafter the copy; if the copy's error path and the builder task interleave unluckily, one side can wait on a duplex whose peer is gone.- russh's
ChannelTxblocks on window availability. A channel whose peer died without a window adjustment leaves the writer parked with no waker.
Worth noting the client has no timeout on the upload at all — the transport-level HANDSHAKE_TIMEOUT (client.rs) bounds only connection setup, and there is no equivalent for the transfer.
Impact
Worse than the failure it accompanies. An error is diagnosable; a silent hang looks like a slow upload, and in CI it burns the job's whole time budget. Independent of the #869 root cause: once that is fixed the hang becomes rarer but the missing deadline remains.
Suggested direction
A deadline (or an idle-progress timeout) on the upload, so a dead peer surfaces as an error rather than an indefinite wait — plus a check of whether the error path can deadlock against the tar builder task.
Activity
gominimal-aw-bot commentedon Jul 22, 2026
Routing triage — #886
Kind: bug · Route: patch · Confidence: 0.88 · Target: gominimal/minimal
Evidence
crates/minimal/src/client.rsconfirmed via code search, with thestream_tar_zstdcall site present exactly where the issue describes it.crates/minimal/src/file_upload.rsreferenced as the function's home.inboxbyinbox-route-retro. Multiple prior decisions routed minimal CLI bugs to gominimal/minimal, consistent with this route. F2 outcomes indicate the patch agent sometimes struggles with async Rust, not that the target is wrong. (informed by routing-outcome entries)Verdict
file_upload.rs,client.rs), code search confirms both exist with the relevant call site, and the fix direction is specific (add a deadline or idle-progress timeout to the upload). 2 files, withinmax_patch_files=5.inbox-patch. If the patch confidence gate downgrades tosdd, a maintainer comments/specon the transferred issue to start the SDD pipeline.[inbox-route:decision]
{"schema":"inbox-route/v1","issue":335,"issue_node_id":"I_kwDOSUhdos8AAAABJr7f3g","dry_run":false,"kind":"bug","route":"patch","confidence":0.88,"targets":[{"repo":"gominimal/minimal","reason":"crates/minimal/src/file_upload.rs and client.rs confirmed in this repo; stream_tar_zstd call site confirmed by code search; referenced issue #869 is in the same repo"}],"labels":["kind:bug","priority:should-have"],"patch":{"target_ref":"main","brief":"On vsock-backed workspace upload, a minority of failure attempts park the client indefinitely when the peer connection is reset. The stream_tar_zstd function in file_upload.rs uses tokio::io::copy with no deadline; evidence suggests either a race between the copy error path and the builder task's build.await, or russh's ChannelTx parking on window availability when the peer is gone. Fix direction: add a deadline or idle-progress timeout to the upload, and confirm the builder task does not deadlock on error.","files":["crates/minimal/src/file_upload.rs","crates/minimal/src/client.rs"]},"work_issue":{"title":"bug: upload sometimes hangs indefinitely when transfer peer dies","body":"## Summary\n\nOn a workspace upload over the vsock-backed SSH channel, a minority of failure attempts result in the client hanging indefinitely rather than returning an error. The daemon has already torn down the connection; the client never detects this and never exits. Observed during the ENOBUFS investigation (#869) at durations of 75 s to 13 min before external kill.\n\n## Expected behavior\n\nWhen the guest-side vsock connection is reset mid-transfer, the client exits with an error — as it does in the majority-case fast path (`error: Failed to upload project files: copying tar stream to channel: channel closed`).\n\n## Actual behavior\n\nIn a minority of failure attempts, the client parks indefinitely on `Uploading project files...`. No timeout fires. In CI this burns the job's full time budget; interactively the user sees no progress and no error, indefinitely.\n\n## Reproduction\n\nSame fixture as #869: a single file compressing to ~12.78 MB uploaded via `min activate --no-loadouts -n <name> <fixture>` to a guest whose vsock connection is reset mid-transfer. Most runs error out; a minority hang. Observed at 75 s, ~2 min, ~9 min, and 13 min before external kill.\n\n## Scope notes\n\nThe upload path lives in `crates/minimal/src/file_upload.rs` (`stream_tar_zstd`, using `tokio::io::copy` with no deadline) and `crates/minimal/src/client.rs` (call site). The transport-level `HANDSHAKE_TIMEOUT` in `client.rs` bounds only connection setup; no equivalent timeout covers the transfer. Two candidate mechanisms, unconfirmed: (1) the `build.await` in `stream_tar_zstd` could race with the copy error path when the builder task and a dropped channel interleave; (2) russh's `ChannelTx` may park on window availability when the peer has died without a window adjustment. Suggested fix: add a deadline or idle-progress timeout on the upload, and verify the error path does not deadlock against the tar builder task. Independent of the #869 root cause fix — the missing deadline remains even after that is resolved. Expected files: `crates/minimal/src/file_upload.rs`, `crates/minimal/src/client.rs`.\n\nRelated: gominimal/minimal#869\n\n---\n<!-- SDD command vocabulary. Do not edit this footer. -->\n\nThis issue is the tracking issue for this work. A maintainer with write\naccess starts the SDD pipeline by commenting `/spec` on this issue or by\nflipping the `sdd:spec` label.\n\nSteer the pipeline with these comment commands. Each is gated to commenters\nwith write access to the repository.\n\n| Command | Where | Effect |\n|---|---|---|\n| `/spec` | this tracking issue | re-run `sdd-spec` to draft or revise the spec |\n| `/fastpath` | this tracking issue | confirm a single-session change; `sdd-spec` produces a stub spec PR and an execution plan comment in one run (ADR 0012) |\n| `/triage` | this tracking issue, after the spec PR is merged | start `sdd-triage` phase A (architecture) |\n| `/approve` | this tracking issue | full path: confirm the proposed plan so `sdd-triage` creates the Unit and task sub-issues. Fast path: dispatch the execution plan against `sdd-execute-{tier}` |\n| `/dispatch` | this tracking issue (full path only) | arm the cascade across the task tree |\n| `/revise <note>` | a spec, architecture, or implementation PR, or this tracking issue | re-run the owning agent with the note as an added instruction |\n| `/execute` | a task sub-issue | run `sdd-execute` for that task ahead of the schedule |\n\nMerging the spec PR advances to the architecture phase; merging the\narchitecture PR advances to a plan-comment phase, where `sdd-triage` posts\nthe proposed plan as one comment on this tracking issue. When an agent needs\na human decision it applies the `needs-human` label and posts one comment;\nclear the label once you have answered and the agent resumes."},"related":[{"repo":"gominimal/minimal","number":869,"relation":"related"},{"repo":"gominimal/minimal","number":788,"relation":"related"}],"rationale":"Issue gominimal/minimal#886 describes a Rust async bug in the minimal CLI upload path: stream_tar_zstd in crates/minimal/src/file_upload.rs uses tokio::io::copy with no deadline, and when the vsock peer dies the client can park indefinitely — either on build.await racing the copy error path, or on russh ChannelTx parking on window availability. Code search confirmed crates/minimal/src/client.rs exists with the stream_tar_zstd call site. Referenced issue #869 is also in gominimal/minimal. No duplicate or prior routing found; minimal#788 (closed/completed) is a related hang symptom on the attach stream path, not the upload path. Routing precedents confirm gominimal/minimal is the correct target for minimal CLI bugs. The fix is bounded: add a tokio::time::timeout or idle-progress watchdog on the upload in at most 2 files, well within max_patch_files=5. Confidence 0.88: strong circumstantial fit with direct code evidence."}gominimal-aw-bot commentedon Jul 22, 2026
[inbox-route:ledger]
Routed from gominimal/inbox#335 by inbox-route (https://github.com/gominimal/inbox/actions/runs/29887532639)
Routing-Key: inbox-route/I_kwDOSUhdos8AAAABJr7f3g
Route: patch
Patch-Dispatched: https://github.com/gominimal/inbox/actions/runs/29887532639
fix(minimal): bound workspace upload by an idle-progress timeout
fix(minimal): add idle-progress timeout to upload to prevent indefini…
fix(minimal): add idle-progress timeout to upload to prevent indefini…
fix(minimal): add idle-progress timeout to upload to prevent indefini…
fix(minimal): add idle-progress timeout to upload to prevent indefini…