Repository navigation
minvmd: scheduled guest maintenance on a timer — cache sweep + fstrim #990
Description
Goal
Run periodic guest-state maintenance on a timer, driven by minvmd: a cache sweep (delete stale build artifacts and dead sandbox/task/temp dirs) followed by fstrim on /var/lib/minimal, so both the guest filesystem and the host's backing raw image stay bounded over a long-lived VM's life.
The two halves are ordered and both required: the sweep frees blocks inside ext4, the trim returns those blocks to the host image. Neither alone is sufficient.
Motivation
Nothing reclaims guest state today. cache/built grows monotonically, and each session keeps its own uploaded workspace tree for the session's lifetime. mip cache clean exists but is a manual, project-scoped command that nobody runs inside the VM.
Deleting files does not shrink the host image. The state volume is mounted with MS_NOATIME only — no discard option (crates/minimald/src/guest.rs:439). Measured during #658 / #583 on libkrun 1.19.0: rm alone reclaims nothing on the host, while guest fstrim /var/lib/minimal took a host image from 2061 MiB down to 11 MiB — libkrun's virtio-blk advertises discard and forwards virtio UNMAP to an APFS hole-punch. Without a trim, data-vol.raw is a high-water mark of everything ever written to the volume, and it only ever goes up.
Evidence
From a user support bundle collected 2026-07-24 on 0.5.0-rc2.dev.36.g5720bf64 — a MacBook Air running two sessions against the same repo, roughly two weeks of light use since the volume was created:
| Subtree | Size | Files |
|---|---|---|
cache/built |
1.44 GB | 18,429 |
sessions/425ba (tree + home) |
0.80 GB | 2,996 |
sessions/e976c (tree + home) |
0.79 GB | 2,557 |
Each session tree holds its own copy of the same 726 MB git pack, uploaded once per session. Nothing in the system will ever remove any of it.
Worth being precise about what this is not: the volume is not currently bloated. ext4 reports 15.7 GiB used, but ~12.5 GiB of that is ext4's default 5% root reserve on a 251 GiB filesystem, leaving ~3.4 GiB of real data — consistent with the table above plus journal and metadata. This issue is about unbounded growth over time, not a present-day leak.
Side observation from the same bundle, probably its own cleanup: the host state dir carries two 256 GiB-nominal raw volumes — the live providers/local-0/data-vol.raw and a stale minvmd/data-vol.raw from the pre-provider layout, last touched two weeks earlier. (Apparent sizes; the listing does not carry allocated block counts, so how much either actually occupies is unknown.)
What already exists
- The sweep logic.
crates/mip/src/cmd_cache.rsalready implements it: read-tracker atimes viaCache::atimes(), an--older-thancutoff defaulting to 14d, entries with no recorded use at all, plus reaping sandbox/task/temp dirs whose owning pid is gone (a/proc/<pid>check, so guest-side only — which is where we want it anyway). - The host→guest control pattern.
minvmd'srpc_client.rsalready drives the guest through the Shutdown RPC to make it drain sessions and quiesce the volume. A maintenance trigger is the same shape. fstrimin the guest. The rootfs shipsutil-linuxande2fsprogs(rootfs #365), and the production-path reclaim was verified through it.
What's missing
- Reuse of
cache cleanis not a drop-in. It is project-scoped: it protectsctx.scaffolding_packages()for the current project. A daemon-side sweep has no single project and must instead protect the union of packages referenced by every live session, or it will delete cache entries out from under a running session. This is the main correctness question in the issue. - No scheduler in
minvmd. Only ad-hoctokio::time::sleepcalls innet.rs. This needs a real interval task with a configurable period. - No maintenance RPC. Whatever triggers the guest-side work needs a verb; today only Shutdown exists.
Design notes
Schedule on the host, not in the guest. The guest clock is not trustworthy for scheduling: it sets from the pl031 RTC once at boot and nothing resyncs it on host wake, so it drifts by the host's accumulated sleep. The same support bundle above shows the guest 6 h 36 m behind the host — its two halves are timestamped 14:01 and 20:37 for a collection that took 2.5 seconds. A guest-side cron would fire on a clock that silently loses hours a day; a host-side timer would not. (The clock drift is a real defect in its own right and should be filed separately.)
That also matches the Shutdown precedent — policy stays on the host, the guest executes — and it keeps working when the guest is wedged enough to miss its own timer.
Ordering and observability. Sweep, sync, settle, then trim; measuring the trim too soon after the deletes reads as "no reclaim." Log bytes deleted and bytes trimmed (FITRIM reports the trimmed count) — without that, a silent no-op is indistinguishable from a healthy run.
Concurrency. Online fstrim on a live ext4 is safe, but FITRIM holds block-group locks and can be slow, so it should defer rather than contend when a build is in flight. Distinct from the shutdown quiesce path: this is steady-state maintenance, not teardown.
Open questions
- Retention policy: inherit
cache clean's 14d default, or make it aminvmd.tomlknob? Should the trim run on a different (longer) period than the sweep? - Does the sweep also touch session trees for stopped/destroyed sessions, or is that out of scope for a first cut?
- Should the timer be suppressed while the host is on battery? On the sampled host, the VM lives through heavy clamshell sleep cycling.
Acceptance
- A timer in
minvmdfires on a configurable interval and drives the guest through sweep-then-trim. - Both steps log what they reclaimed.
- Host-side proof:
st_blocks * 512forproviders/<name>/data-vol.rawmeasurably drops after a run that sweeps a large stale cache entry. - No live session loses a cache entry it depends on.
Activity
gominimal-aw-bot commentedon Jul 28, 2026
Routing triage — #990
Kind: feature — Route: sdd — Confidence: 0.92 — Target: gominimal/minimal
Periodic maintenance scheduler for
minvmd: host-side interval timer drives a guest-side cache sweep +fstrim. All evidence points unambiguously togominimal/minimal; this is genuine new capability with unresolved design questions, makingsddthe right route.Launch: a maintainer comments
/specon the work issue to start the SDD pipeline.Evidence
Repository peeks (direct file hits):
crates/minvmd/src/rpc_client.rs— confirmed ✓ (the issue's Shutdown RPC precedent; new maintenance RPC goes here)crates/minvmd/src/net.rs— confirmed ✓ (the issue notes only ad-hoctokio::time::sleepcalls here; interval scheduler goes here or in a new module)crates/minvmd/src/config.rs— confirmed ✓ (maintenance period config knob)crates/mip/src/cmd_cache.rs— confirmed ✓ (existing sweep logic, project-scoped; daemon adaptation required)crates/minimald/src/guest.rs— confirmed ✓ (referenced at line 439 for volume mount flags)Similar issues: No duplicate or prior-routing hits found in gominimal/inbox or gominimal/minimal for this topic. The issue references minimal#658 and minimal#583 as motivating evidence (disk measurement context), not duplicates; and minimal#365 as a rootfs dependency. None are routing-relevant.
Distillery enrichment: Multiple indexed specs and architecture docs for
minvmdin projectminimal(spec-minvmd-host-daemon, minvmd macOS/Linux architecture docs). Therpc_client.rsShutdown RPC pattern is the declared host→guest control precedent this feature follows.Precedent pass: Several routing-outcome entries retrieved; all prior
minvmd-adjacent work routed togominimal/minimal. No precedent found that contradicts this route or suggests a different target.Manifest:
gominimal/minimalisroute: allow,sdd: true,distillery_project: minimal. No other candidate matches —minvmdis a crate in this workspace exclusively.Why sdd, not patch
The work explicitly requires: (1) a new interval scheduler (no scheduler exists today — only ad-hoc sleeps in
net.rs); (2) a new maintenance RPC verb alongside the existing Shutdown RPC; (3) adapted session-aware sweep logic (existingcmd_cache.rsis project-scoped and cannot be used as-is in a daemon context without correctness risk); (4) guest-side handler; (5) config. That spans five or more files and carries three unresolved design decisions (retention policy, session-tree scope, battery suppression). Confidence is high (0.92) on target; the complexity puts it firmly insdd.[inbox-route:decision]
{"schema":"inbox-route/v1","issue":364,"issue_node_id":"I_kwDOSUhdos8AAAABKaEQvA","dry_run":false,"kind":"feature","route":"sdd","confidence":0.92,"targets":[{"repo":"gominimal/minimal","reason":"all referenced files (crates/minvmd/src/rpc_client.rs, crates 8000 /minvmd/src/net.rs, crates/mip/src/cmd_cache.rs, crates/minimald/src/guest.rs) confirmed in this repo; minvmd is an established crate in this workspace with indexed specs in Distillery"}],"labels":["kind:feature","priority:should-have"],"patch":null,"work_issue":{"title":"feature: minvmd scheduled guest maintenance — cache sweep + fstrim","body":"## Summary\n\nAdd a periodic maintenance cycle to `minvmd`: a configurable host-side timer triggers a guest-side cache sweep (stale build artifacts and dead sandbox/task/temp dirs) followed by `fstrim /var/lib/minimal`, so the guest filesystem and the host raw image stay bounded over a long-lived VM's life.\n\n## Problem\n\nBuild artifacts in `cache/built` accumulate monotonically — `mip cache clean` is project-scoped and manual; nobody runs it inside the VM. Session trees persist for their lifetime, each holding a full copy of the uploaded workspace git pack. Deleting files alone does not shrink the host image: the volume is mounted without `discard`, and evidence shows `fstrim /var/lib/minimal` reduced a host image from 2061 MiB to 11 MiB on libkrun 1.19.0.\n\nThe scheduler must run on the host: the guest clock drifts by accumulated host sleep time (observed at 6 h 36 m behind in a support bundle), making guest-side cron unreliable.\n\n## Expected behavior\n\n- A configurable interval timer in `minvmd` fires on the host and drives the guest through a new maintenance RPC.\n- The maintenance cycle: session-aware cache sweep → sync → settle → `fstrim /var/lib/minimal`.\n- Both steps log what was reclaimed (bytes deleted; bytes trimmed via FITRIM's reported count).\n- Host-side proof: `st_blocks * 512` for the provider's `data-vol.raw` measurably drops after sweeping a large stale entry.\n- No live session loses a cache entry it depends on.\n\n## Actual behavior\n\nNo periodic maintenance exists. `cache/built` grows monotonically; session trees are never removed. The host raw image is a permanent high-water mark of everything ever written.\n\n## Scope notes\n\nKey decisions for the spec:\n\n1. **Session-aware sweep**: `mip cache clean` protects only `ctx.scaffolding_packages()` for the current project. The daemon-side sweep must protect the union of packages referenced by every live session to avoid evicting cache entries from under a running session. This is the main correctness question.\n2. **Scheduler location**: host side in `crates/minvmd/src/net.rs` or a new module; today the file contains only ad-hoc `tokio::time::sleep` calls and needs a real interval task with a configurable period.\n3. **New maintenance RPC**: a new verb alongside the existing Shutdown RPC in `crates/minvmd/src/rpc_client.rs` to trigger guest-side work.\n4. **Ordering and observability**: sweep → sync → settle → fstrim. Log both steps — a silent no-op is indistinguishable from a healthy run without it.\n5. **Concurrency**: online `fstrim` on live ext4 is safe, but FITRIM holds block-group locks — should defer rather than contend when a build is in flight.\n6. **Open questions**: retention policy (inherit 14d default or a `minvmd.toml` knob?); whether to sweep stopped-session trees in a first cut; battery suppression on laptop hosts.\n\nEvidence-confirmed files: `crates/minvmd/src/rpc_client.rs`, `crates/minvmd/src/net.rs`, `crates/minvmd/src/config.rs`, `crates/mip/src/cmd_cache.rs`, and the guest-side handler (likely `crates/minimald/src/rpc.rs`).\n\n---\n<!-- SDD command vocabulary. Do not edit this footer. -->\n\nThis issue is the tracking issue for this work. A maintainer with write\naccess starts the SDD pipeline by commenting `/spec` on this issue or by\nflipping the `sdd:spec` label.\n\nSteer the pipeline with these comment commands. Each is gated to commenters\nwith write access to the repository.\n\n| Command | Where | Effect |\n|---|---|---|\n| `/spec` | this tracking issue | re-run `sdd-spec` to draft or revise the spec |\n| `/fastpath` | this tracking issue | confirm a single-session change; `sdd-spec` produces a stub spec PR and an execution plan comment in one run (ADR 0012) |\n| `/triage` | this tracking issue, after the spec PR is merged | start `sdd-triage` phase A (architecture) |\n| `/approve` | this tracking issue | full path: confirm the proposed plan so `sdd-triage` creates the Unit and task sub-issues. Fast path: dispatch the execution plan against `sdd-execute-{tier}` |\n| `/dispatch` | this tracking issue (full path only) | arm the cascade across the task tree |\n| `/revise <note>` | a spec, architecture, or implementation PR, or this tracking issue | re-run the owning agent with the note as an added instruction |\n| `/execute` | a task sub-issue | run `sdd-execute` for that task ahead of the schedule |\n\nMerging the spec PR advances to the architecture phase; merging the\narchitecture PR advances to a plan-comment phase, where `sdd-triage` posts\nthe proposed plan as one comment on this tracking issue. When an agent needs\na human decision it applies the `needs-human` label and posts one comment;\nclear the label once you have answered and the agent resumes."},"related":[],"rationale":"All files named in the issue (crates/minvmd/src/rpc_client.rs, crates/minvmd/src/net.rs, crates/minvmd/src/config.rs, crates/mip/src/cmd_cache.rs, crates/minimald/src/guest.rs) confirmed verbatim in gominimal/minimal via GitHub API. The minvmd crate has multiple Distillery-indexed specs and architecture docs in the minimal project. No prior routing or duplicate issues found in inbox or minimal for this topic. The work requires a new interval scheduler in minvmd, a new maintenance RPC verb, adapted session-aware sweep logic (existing cmd_cache.rs is project-scoped and cannot be used as-is for daemon context), fstrim orchestration, config, and a guest-side handler — well beyond max_patch_files and carrying unresolved design questions about retention policy, session-tree scope, and battery suppression. Route sdd at confidence 0.92."}gominimal-aw-bot commentedon Jul 28, 2026
[inbox-route:ledger]
Routed from #990 by inbox-route (https://github.com/gominimal/inbox/actions/runs/30315685635)
Routing-Key: inbox-route/I_kwDOSUhdos8AAAABKaEQvA
Route: sdd
Work-Issue: #990 (in-place — the transferred issue is the deliverable)
refactor(minvmd): move the maintenance schedule into the guest (#990)
twitchyliquid64 commentedon Aug 10, 2026
Status update: cache clean is implemented but not
fstrim.