feat(minvmd): export the VM host daemon's spans and carry telemetry into the guest (4/7, spec 25) by bryan-minimal · Pull Request #1996 · gominimal/minimal

bryan-minimal · 2026-10-05T20:52:38Z

feat(minvmd): export the VM host daemon's spans and carry telemetry into the guest (4/7, spec 25)

Stack: spec #1988 → test hardening → mlog export → CLI+daemon → this → diag → requirement tests → guest telemetry over vsock (7/7). Base: otel/3-cli-daemon.

12 files, +2445/−157:

crates/minvmd/src/telemetry.rs — minvmd joins the request trace from the CLI/daemon and exports its own spans (boot, stop, remount, egress gate) under the same opt-in; the Rust-log bridge warns once, never panics (TEL-040..044).
crates/minvmd/src/vm.rs — the guest boot line: telemetry settings reach the guest daemon only through an allowlist of MINIMAL_* keys with a byte budget; a long or non-ASCII filter cannot kill the boot; no endpoint ever crosses: the guest's exporters are forced to none, and its records reach the host only over vsock (feat(otel): guest telemetry rides the VM channel (7/7, spec 25) #2039). With telemetry off the boot line is byte-identical to a build without this stack.
egress_gate.rs, cmd/run.rs, main.rs: spans around the paths that already existed; the new Status { json, row } arm returns through dispatch's Result<i32> like the rest.

LangSec before/after in the lab: the secret-on-the-boot-line, re-emitted trace flags and long/non-ASCII filter cases each REPRODUCED on the unpatched tree and PASS here. Lab evidence (job 00244, four distros): 715/715b/715c/715e/715f (boot/stop and the guest daemon in one trace), 715g/715h (allowlist, bytes, budget), 749b (clean stop and remount); minvmd traces 100 % joined.

Not covered: macOS (minvmd on arm64) is untested in the lab, which runs Linux hosts only; the stack's CI macOS job is the first run there.

Plan: #1992 (the spec's plan comment). Plan tasks in this PR: T12 minvmd root, boot and stop spans · T13 guest boot line allowlist (and the guest side of T6's clean stop).

🤖 Generated with Claude Code

coderabbitai · 2026-10-05T20:52:45Z

Warning

Review limit reached

Run on-demand review

_{This review includes 77 billable files and costs up to $19.25.}

Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 59 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Essentials
Run ID: b8ee8116-33e6-4fdd-af5e-7e7b85d2c08c

📥 Commits

Reviewing files that changed from the base of the PR and between a214f31 and 440cecf.

⛔ Files ignored due to path filters: 1

Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing: 77

Cargo.toml
crates/check/src/lib.rs
crates/check/src/outputs.rs
crates/checkouts/src/lib.rs
crates/checkouts/src/repo.rs
crates/mctx/src/lib.rs
crates/minimal-client/src/attach.rs
crates/minimal-client/src/lib.rs
crates/minimal/Cargo.toml
crates/minimal/src/autospawn.rs
crates/minimal/src/cmd/mod.rs
crates/minimal/src/cmd/net.rs
crates/minimal/src/cmd/session.rs
crates/minimal/src/main.rs
crates/minimald-rpc/src/taskenv.rs
crates/minimald-rpc/src/trace.rs
crates/minimald/Cargo.toml
crates/minimald/clippy.toml
crates/minimald/src/audit.rs
crates/minimald/src/connection.rs
crates/minimald/src/diag.rs
crates/minimald/src/env.rs
crates/minimald/src/env_min_helper.sh
crates/minimald/src/exec.rs
crates/minimald/src/lib.rs
crates/minimald/src/logging.rs
crates/minimald/src/main.rs
crates/minimald/src/maintenance.rs
crates/minimald/src/net/answerer.rs
crates/minimald/src/net/classifier.rs
crates/minimald/src/net/dns.rs
crates/minimald/src/net/gvproxy_network.rs
crates/minimald/src/net/listeners.rs
crates/minimald/src/net/mod.rs
crates/minimald/src/net/policy.rs
crates/minimald/src/net/proxy.rs
crates/minimald/src/net/switch.rs
crates/minimald/src/net/wg.rs
crates/minimald/src/reaper.rs
crates/minimald/src/rpc.rs
crates/minimald/src/server.rs
crates/minimald/src/session.rs
crates/minimald/src/session/kind.rs
crates/minimald/src/session/tests.rs
crates/minimald/src/session_delta.rs
crates/minimald/src/session_host.rs
crates/minimald/src/session_host/tests.rs
crates/minimald/src/session_sop.rs
crates/minimald/src/sessions.rs
crates/minimald/src/sessions/composables.rs
crates/minimald/src/sftp.rs
crates/minimald/src/store.rs
crates/minimald/src/test_harness.rs
crates/minimald/src/traced.rs
crates/minimald/tests/classifier_root_integration.rs
crates/minimald/tests/netns_root_integration.rs
crates/minvmd/Cargo.toml
crates/minvmd/src/cmd/boot.rs
crates/minvmd/src/cmd/run.rs
crates/minvmd/src/cmd/stop.rs
crates/minvmd/src/control.rs
crates/minvmd/src/lib.rs
crates/minvmd/src/main.rs
crates/minvmd/src/net.rs
crates/minvmd/src/net/bep_stub.rs
crates/minvmd/src/net/egress_gate.rs
crates/minvmd/src/net/forward_revoke.rs
crates/minvmd/src/rpc_client.rs
crates/minvmd/src/telemetry.rs
crates/minvmd/src/vm.rs
crates/mlog/Cargo.toml
crates/mlog/src/lib.rs
crates/mlog/src/otel.rs
crates/mlog/src/otel/spool.rs
crates/mlog/src/otel/switches.rs
crates/orchestrator/src/lib.rs
crates/sessions/src/store.rs

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

minimal-foundry · 2026-10-07T00:08:01Z

Reviewed: 1 finding, none blocking at 440cecf10c57.

Address the comments on the diff.

Clobber: none: git log on the changed files shows only this PR atop the otel/3-cli-daemon base; no recent fix is reverted.

minimal-foundry

`minimal-review` — advisory review

Verdict COMMENT — 1 finding, 1 inline.

bryan-minimal · 2026-10-07T00:37:55Z

@coderabbitai review

coderabbitai · 2026-10-07T00:38:03Z

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

minimal-foundry

`minimal-review` — approved

Verdict APPROVE — No findings.

minimal-foundry

`minimal-review` — advisory review

Verdict COMMENT — 1 finding, 1 inline.

norrietaylor

Review of 4/7 at 574fbb2, against its stack parent #1995 and spec 25 at eed80b0.

Summary:

The minvmd span structure is good. The start phase ends at READY, supervision runs under its own linked root, and long-lived tasks use vm_task, so they do not hold the start phase open. TEL-032 and TEL-035 have tests here.
The egress gate, net.rs and run.rs changes add only spans. They do not change gate decisions.
Must fix: this PR puts the OTLP endpoint on the guest boot line, and its tests assert that. TEL-033 forbids it, and #2039 removes it. Remove the endpoint arm here, or merge 4/7 to 7/7 as one unit.
Should fix: the full boot line is logged at info level and exported. Update the module doc.
Minor: minvmd boot does not record the error status on vm.boot. The spawn scan test has gaps.

mitodrummer · 2026-10-07T18:39:01Z

Review pass (read-only, head 574fbb2a). With telemetry off the boot line is unchanged, TRACEPARENT is re-serialised, tokens are alphabet-checked and all-or-none within the budget, and nothing guest-originated is trusted here.

Must fix — +1 to @norrietaylor at telemetry.rs:237: guest_env puts the host's OTLP endpoint on the guest boot line (MINIMAL_OTEL_EXPORTER_OTLP_<SIG>_ENDPOINT=<url>), which TEL-033 in this stack's own spec forbids. /proc/cmdline isn't namespaced, so untrusted code in any box can read the internal URL, and secret_shaped (telemetry.rs:394) misses e.g. a 20-char all-letter key in the path. #2039 removes this arm, but if #1996 ships in a release without #2039 the forbidden behaviour ships. Suggest dropping the endpoint arm (endpoint_refusal, parse_http_url, endpoint_origin, the url dep and the tests asserting an endpoint crosses) here. The PR body's "bare OTEL_* never crosses" is also misleading — the value crosses, renamed.

Also open, all valid:

vm.rs:327 logs the full boot line (endpoint included) at info — not exported (the VMM process never inits OTel) but still written to the local log, against TEL-040.
cmd/boot.rs:121 declares otel.status_code on vm.boot but never records it, so a failed boot exports as OK.
egress_gate.rs:1147 relays.spawn(serve_connection(...)) has no span, and the spawn guard only matches tokio::spawn(.

minimal-foundry

`minimal-review` — advisory review

Verdict COMMENT — 1 finding, 1 inline.

mitodrummer · 2026-10-07T21:15:50Z

Verified at 92e4b98: the endpoint arm, url dep and the endpoint tests are gone, exporters cross as none, the boot line logs keys only, the spawn scan is widened — all four items fixed. Two things before I re-approve:

The bot's run.rs:896 finding holds: only minvmd boot got ErrorUnlessOk. In run_foreground the vm.boot span is marked ERROR on the READY timeout only; the spawn ?, lifecycle lock/read/write ? and the bail!("lifecycle changed … during spawn") paths still export OK. Hold the guard there too and .ok() before the span drops (~:986).
PR body: it still says "bare OTEL_* never crosses" and cites 715g/715h; say that no en 10000 dpoint crosses and the guest exporters are forced to none. The vm.rs test fixtures that model an endpoint token crossing (guest_env_tokens_follow_the_log_filter, the overflow and next_arg tests) could use a MINIMAL_OTEL_FILTER-style token so they stop contradicting TEL-033.

minimal-foundry

`minimal-review` — advisory review

Verdict COMMENT — 1 finding, 1 inline.

bryan-minimal · 2026-10-08T00:55:34Z

@mitodrummer both of your re-approve items are in round 4 (e6757af):

run_foreground takes telemetry::ErrorUnlessOk on vm.boot right after entering it and calls .ok() only on reaching READY. So the spawn ?, the lifecycle lock, read and write ?s, the "lifecycle changed … during spawn" bail! and a failed wait all export ERROR. The hand-written ERROR record is gone. Tests: an_early_return_from_run_marks_vm_boot_failed, and every_vm_boot_is_held_by_the_guard, a mutation-checked source check over boot.rs and run.rs.
The vm.rs fixtures (guest_env_tokens_follow_the_log_filter, the overflow test and the next_arg test) now use MINIMAL_OTEL_FILTER and TRACEPARENT tokens, never an endpoint. The PR body now says that no endpoint crosses, that the guest's exporters are forced to none, and that its records reach the host only over vsock (feat(otel): guest telemetry rides the VM channel (7/7, spec 25) #2039).

minimal-foundry

`minimal-review` — advisory review

Verdict COMMENT — 1 finding, 1 inline.

minimal-foundry

`minimal-review` — approved

Verdict APPROVE — No findings.

minimal-foundry

`minimal-review` — approved

Verdict APPROVE — No findings.

minimal-foundry

`minimal-review` — advisory review

Verdict COMMENT — 1 finding, 1 inline.

Three small test-only changes that the telemetry series relies on, kept separate so they can land on their own: - the walked-publication probe waits for the publication to settle - bep_stub tolerates EPIPE when the client hangs up first - the fresh-streak test waits for the streak counter to advance No product code changes.

The telemetry core for every minimal process (spec 25 TEL-001 to TEL-020, principles 1, 2, 7 and 8). Off unless MINIMAL_TELEMETRY=1; when off nothing starts: no thread, no exporter, no file. When on: - spans and logs leave over OTLP/HTTP (opentelemetry 0.33) to the endpoint the MINIMAL_OTEL_EXPORTER_OTLP_* variables name, base or per signal, and the plain OTEL_* ones only when no MINIMAL_ endpoint is set; per-signal endpoints and `*_EXPORTER=none` are honoured; - DO_NOT_TRACK (any non-empty value) and OTEL_SDK_DISABLED=true always win; - the exporter's own HTTP stack and the SDK never produce telemetry, errors never reach a console, and the exit flush is bounded (200 ms in the CLI); - every finished span and log record is appended synchronously to an OTLP/JSON spool under <state>/telemetry/spool (0700/0600, 4 MiB segments, pruned to 50 MiB / 7 days, torn lines contained), so a crash or SIGKILL keeps what was already written; - the endpoint and the switches have one recognizer each (url::Url, a byte-exact flag grammar), ambient OTEL_EXPORTER_OTLP_HEADERS are refused when the endpoint came from a MINIMAL_ variable, and secrets never appear in the init summary or the logs. The switch decision and the spool prune plan are pure functions with Kani harnesses beside their unit tests.

Wire mlog's telemetry into the CLI and the daemon (spec 25 TEL-021 to TEL-029). - The CLI mints or adopts a TRACEPARENT, exports its `cmd` span, and ends and flushes it before any hand-off; the daemon adopts the context from the channel environment and records `rpc` and `exec` spans under it. - `min session exec` / `run` relays ssh rather than exec()ing it (#1881), so every way out of a command is an exit or signal hand-off and the `cmd` span covers the command; `run` finishes the hand-off once the span has ended and been flushed. - Work the daemon does for a request stays in that request's trace: actor mailboxes carry the sender's span, the blocking pool, spawned tasks (`traced::spawn`, with a guard test against bare spawns), package fetches, orchestrator and check tasks, checkouts (remotes scrubbed), git upload and receive, and the periodic cache clean in its own root. - Boxes receive only the exec span's TRACEPARENT (never headers, baggage or OTEL_* variables), and only when that span is recorded; a nested `min` inside a box, a task run and the in-box helper socket forward it, so daemon-side work for them joins the outer trace. - `min net forward` carries the context on the direct-tcpip open; attach adopts it; SIGHUP and exit paths flush within the shutdown bound.

…nto the guest minvmd exports a bounded root per command with `net.switch.start`, `vm.boot`, `vm.stop` and `guest.shutdown` as children, in the CLI's trace (spec 25 TEL-032 to TEL-036). The host decides the telemetry switches and writes them onto the guest's boot line with one byte-exact writer under a fixed budget: the opt-in, the spool and filter settings, the exporter switches and TRACEPARENT, never an endpoint, headers, resource attributes or baggage (TEL-033), because everything in the guest can read the boot line as /proc/cmdline. With no endpoint the guest exports nothing here: both signals are `none` and its records stay in its own spool until the VM channel carries them to the host. The whole set is forwarded or dropped, so a guest never boots with a partial configuration, and the logged boot line names the telemetry tokens by key only. The guest daemon builds its OTLP client without platform CA roots (an http:// collector needs none) and flushes before it powers off. RUST_LOG filters the console per layer, so it never gates export. A failed `minvmd boot` marks `vm.boot` with an error status, as `run` does. The boot line's alphabet, budget and all-or-none rules are proven with Kani.

minimal-foundry

`minimal-review` — advisory review

Verdict COMMENT — 1 finding, 1 inline.

bryan-minimal force-pushed the otel/4-minvmd-guest branch from 0b473cb to d405d53 Compare October 6, 2026 23:32

bryan-minimal force-pushed the otel/3-cli-daemon branch from 544de75 to 19e55bd Compare October 6, 2026 23:32

bryan-minimal changed the title ~~feat(minvmd): export the VM host daemon's spans and carry telemetry into the guest (4/6, spec 25)~~ feat(minvmd): export the VM host daemon's spans and carry telemetry into the guest (4/7, spec 25) Oct 6, 2026

bryan-minimal marked this pull request as ready for review October 7, 2026 00:06

bryan-minimal requested a review from a team as a code owner October 7, 2026 00:06

minimal-foundry Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread crates/minvmd/src/cmd/boot.rs Resolved

bryan-minimal force-pushed the otel/3-cli-daemon branch from 19e55bd to a61e33d Compare October 7, 2026 00:21

bryan-minimal force-pushed the otel/4-minvmd-guest branch from d405d53 to e576df1 Compare October 7, 2026 00:21

minimal-foundry Bot approved these changes Oct 7, 2026

View reviewed changes

minimal-foundry Bot added the review:approved minimal-review approved the current head label Oct 7, 2026

bryan-minimal force-pushed the otel/3-cli-daemon branch from a61e33d to 8e27af5 Compare October 7, 2026 01:38

bryan-minimal force-pushed the otel/4-minvmd-guest branch from e576df1 to 574fbb2 Compare October 7, 2026 01:38

minimal-foundry Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread crates/minvmd/src/net/egress_gate.rs Resolved

minimal-foundry Bot removed the review:approved minimal-review approved the current head label Oct 7, 2026

norrietaylor reviewed Oct 7, 2026

View reviewed changes

Comment thread crates/minvmd/src/telemetry.rs Resolved

Comment thread crates/minvmd/src/telemetry.rs Resolved

Comment thread crates/minvmd/src/vm.rs Resolved

Comment thread crates/minvmd/src/cmd/boot.rs Resolved

Comment thread crates/minvmd/src/telemetry.rs Resolved

bryan-minimal force-pushed the otel/3-cli-daemon branch from 8e27af5 to 58e33fd Compare October 7, 2026 20:39

bryan-minimal force-pushed the otel/4-minvmd-guest branch from 574fbb2 to 92e4b98 Compare October 7, 2026 20:39

minimal-foundry Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread crates/minvmd/src/cmd/run.rs Resolved

bryan-minimal force-pushed the otel/4-minvmd-guest branch from 92e4b98 to e6757af Compare October 8, 2026 00:08

bryan-minimal force-pushed the otel/3-cli-daemon branch from 58e33fd to 8589c71 Compare October 8, 2026 00:08

minimal-foundry Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread crates/minvmd/src/rpc_client.rs Resolved

bryan-minimal force-pushed the otel/4-minvmd-guest branch from e6757af to 34b99e7 Compare October 8, 2026 01:17

bryan-minimal force-pushed the otel/3-cli-daemon branch from 8589c71 to 2db9bfa Compare October 8, 2026 01:17

bryan-minimal force-pushed the otel/4-minvmd-guest branch from 5618f2d to c09ef70 Compare October 13, 2026 18:25

bryan-minimal force-pushed the otel/3-cli-daemon branch from 244d63d to b76a52b Compare October 13, 2026 18:25

minimal-foundry Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread crates/minvmd/src/cmd/stop.rs Resolved

minimal-foundry Bot removed the review:approved minimal-review approved the current head label Oct 13, 2026

bryan-minimal force-pushed the otel/3-cli-daemon branch from b76a52b to f380e8f Compare October 13, 2026 20:01

bryan-minimal force-pushed the otel/4-minvmd-guest branch from c09ef70 to eca58d5 Compare October 13, 2026 20:01

minimal-foundry Bot approved these changes Oct 13, 2026

View reviewed changes

minimal-foundry Bot added the review:approved minimal-review approved the current head label Oct 13, 2026

bryan-minimal force-pushed the otel/4-minvmd-guest branch from eca58d5 to 8c61be2 Compare October 13, 2026 21:40

bryan-minimal force-pushed the otel/3-cli-daemon branch 2 times, most recently from 02261d3 to 6ed880a Compare October 13, 2026 21:47

bryan-minimal force-pushed the otel/4-minvmd-guest branch from 8c61be2 to 0b57a99 Compare October 13, 2026 21:47

minimal-foundry Bot removed the review:approved minimal-review approved the current head label Oct 13, 2026

minimal-foundry Bot approved these changes Oct 13, 2026

View reviewed changes

bryan-minimal force-pushed the otel/3-cli-daemon branch from 6ed880a to dce2819 Compare October 13, 2026 21:59

bryan-minimal force-pushed the otel/4-minvmd-guest branch from 0b57a99 to fe44427 Compare October 13, 2026 21:59

minimal-foundry Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread crates/minvmd/src/telemetry.rs Resolved

bryan-minimal added 4 commits October 13, 2026 16:52

bryan-minimal force-pushed the otel/4-minvmd-guest branch from fe44427 to 440cecf Compare October 13, 2026 23:52

bryan-minimal force-pushed the otel/3-cli-daemon branch from dce2819 to 4ff7dbe Compare October 13, 2026 23:52

bryan-minimal mentioned this pull request Oct 13, 2026

OTel stack follow-ups: ten review findings deferred from the r11 merge #2202

Closed

bryan-minimal changed the base branch from otel/3-cli-daemon to main October 14, 2026 00:02

minimal-foundry Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread crates/minvmd/src/cmd/boot.rs Resolved

bryan-minimal closed this Oct 14, 2026

bryan-minimal reopened this Oct 14, 2026

bryan-minimal merged commit 8af4e1d into main Oct 14, 2026
37 checks passed

bryan-minimal deleted the otel/4-minvmd-guest branch October 14, 2026 01:54

Conversation

Uh oh!

Uh oh!

Uh oh!

Review limit reached

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

minimal-review — advisory review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

minimal-review — approved

Uh oh!

Choose a reason for hiding this comment

minimal-review — advisory review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

minimal-review — advisory review

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

minimal-review — advisory review

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

minimal-review — advisory review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

minimal-review — approved

Uh oh!

Choose a reason for hiding this comment

minimal-review — approved

Uh oh!

Choose a reason for hiding this comment

minimal-review — advisory review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

minimal-review — advisory review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

`minimal-review` — advisory review

`minimal-review` — approved

`minimal-review` — advisory review

`minimal-review` — advisory review

`minimal-review` — advisory review

`minimal-review` — advisory review

`minimal-review` — approved

`minimal-review` — approved

`minimal-review` — advisory review

`minimal-review` — advisory review