Repository navigation
ci: full OTel stack on main (throwaway, do not merge) - #2193
Closed
bryan-minimal wants to merge 7 commits into
Closed
bryan-minimal wants to merge 7 commits into
bryan-minimal wants to merge 7 commits into
Conversation
Contributor
8000
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Contributor
Spec lint
Advisory. None of this blocks the merge; the review decides. Advisory, and never a merge gate. Linter: |
bryan-minimal
force-pushed
the
otel/7-vsock
branch
6 times, most recently
from
October 13, 2026 21:59
b51ea64 to
b627c85
Compare
Three small test-only changes that the telemetry series relies on, kept separate so they can land on their own: - the walked-publication probe waits for the publication to settle - bep_stub tolerates EPIPE when the client hangs up first - the fresh-streak test waits for the streak counter to advance No product code changes.
The telemetry core for every minimal process (spec 25 TEL-001 to TEL-020, principles 1, 2, 7 and 8). Off unless MINIMAL_TELEMETRY=1; when off nothing starts: no thread, no exporter, no file. When on: - spans and logs leave over OTLP/HTTP (opentelemetry 0.33) to the endpoint the MINIMAL_OTEL_EXPORTER_OTLP_* variables name, base or per signal, and the plain OTEL_* ones only when no MINIMAL_ endpoint is set; per-signal endpoints and `*_EXPORTER=none` are honoured; - DO_NOT_TRACK (any non-empty value) and OTEL_SDK_DISABLED=true always win; - the exporter's own HTTP stack and the SDK never produce telemetry, errors never reach a console, and the exit flush is bounded (200 ms in the CLI); - every finished span and log record is appended synchronously to an OTLP/JSON spool under <state>/telemetry/spool (0700/0600, 4 MiB segments, pruned to 50 MiB / 7 days, torn lines contained), so a crash or SIGKILL keeps what was already written; - the endpoint and the switches have one recognizer each (url::Url, a byte-exact flag grammar), ambient OTEL_EXPORTER_OTLP_HEADERS are refused when the endpoint came from a MINIMAL_ variable, and secrets never appear in the init summary or the logs. The switch decision and the spool prune plan are pure functions with Kani harnesses beside their unit tests.
Wire mlog's telemetry into the CLI and the daemon (spec 25 TEL-021 to TEL-029). - The CLI mints or adopts a TRACEPARENT, exports its `cmd` span, and ends and flushes it before any hand-off; the daemon adopts the context from the channel environment and records `rpc` and `exec` spans under it. - `min session exec` / `run` relays ssh rather than exec()ing it (#1881), so every way out of a command is an exit or signal hand-off and the `cmd` span covers the command; `run` finishes the hand-off once the span has ended and been flushed. - Work the daemon does for a request stays in that request's trace: actor mailboxes carry the sender's span, the blocking pool, spawned tasks (`traced::spawn`, with a guard test against bare spawns), package fetches, orchestrator and check tasks, checkouts (remotes scrubbed), git upload and receive, and the periodic cache clean in its own root. - Boxes receive only the exec span's TRACEPARENT (never headers, baggage or OTEL_* variables), and only when that span is recorded; a nested `min` inside a box, a task run and the in-box helper socket forward it, so daemon-side work for them joins the outer trace. - `min net forward` carries the context on the direct-tcpip open; attach adopts it; SIGHUP and exit paths flush within the shutdown bound.
…nto the guest minvmd exports a bounded root per command with `net.switch.start`, `vm.boot`, `vm.stop` and `guest.shutdown` as children, in the CLI's trace (spec 25 TEL-032 to TEL-036). The host decides the telemetry switches and writes them onto the guest's boot line with one byte-exact writer under a fixed budget: the opt-in, the spool and filter settings, the exporter switches and TRACEPARENT, never an endpoint, headers, resource attributes or baggage (TEL-033), because everything in the guest can read the boot line as /proc/cmdline. With no endpoint the guest exports nothing here: both signals are `none` and its records stay in its own spool until the VM channel carries them to the host. The whole set is forwarded or dropped, so a guest never boots with a partial configuration, and the logged boot line names the telemetry tokens by key only. The guest daemon builds its OTLP client without platform CA roots (an http:// collector needs none) and flushes before it powers off. RUST_LOG filters the console per layer, so it never gates export. A failed `minvmd boot` marks `vm.boot` with an error status, as `run` does. The boot line's alphabet, budget and all-or-none rules are proven with Kani.
…ry settings Spec 25 TEL-044: the daemon's bundle carries the newest spool files under telemetry/spool/, tail-capped and line-scrubbed like a log; the host bundle does the same for the CLI's and minvmd's spool, which on a VM host exists nowhere else. MINIMAL_OTEL_* endpoint and header values are reported by name only, in both bundles.
…i lanes Integration tests pin spec 25's requirements end to end: opt-in and switches, the spool under SIGKILL and ENOSPC, rotation and prune, a silent or absent collector, trace adoption across the CLI/daemon boundary, the boot line's contents, the minvmd error status, and a URL-confusion corpus. scripts/kani.sh runs the mlog and minvmd harnesses under a memory cap.
The microVM guest daemon no longer exports over the guest's network, which the node's egress baseline governs (minimal#1786) and a `network = none` box host does not have at all. Its spool processors hand each finished span and log record, as the spool's OTLP-JSON line, to a bounded queue; a sender thread in the daemon writes the lines, length-delimited, to the VM host over a vsock port (7353, beside the READY marker's 7350) that the VMM child wires for that VM alone. The `run` supervisor serves the port on a door beside the control socket, appends each accepted line to its own spool, and forwards batches as OTLP/JSON to the host's endpoint for the signal under the host's switches and header rule. The boot line still carries no endpoint: the guest's exporter switches now follow the host's, and `MINIMAL_OTEL_FORWARD=vsock:7353` joins the opt-in, the spool and filter settings and `TRACEPARENT`. A box cannot reach the door: no socket-family seal in sandbox admits AF_VSOCK, now pinned by a test. The door treats what arrives as data. It parses each line strictly (serde_json: no comments, trailing commas or repeated keys) as one request with exactly one resource, at most 1 MiB, stamps every resource with the host's provenance (replacing any minimal.* attribute the guest set), and spools and forwards only what it serialized again from the parsed value. A per-VM byte budget (64 KiB/s, 4 MiB burst) that survives reconnects bounds what a guest can add to the host spool. The guest drops a record over the frame bound instead of sending it, and the door keeps a frame's partial bytes across a read timeout. With the receiver away the guest spools, drops at its queue bound, warns once and retries every 5 s (TEL-046). A stop delivers the guest's last spans before the VM goes down. The guest's `rpc` span for Shutdown ends only when the acknowledgement is written, so `minvmd stop` now lets an acknowledging guest power the VM off for up to 3 s before it signals the VMM (TEL-049); signalling at the ack raced the guest's exit-time flush and lost on a loaded host (lab 715k). The guest's exit drain waits for lines written or dropped, not lines taken off the queue, so its power-off does not cut the last write. When the VMM exits, the supervisor drains the door's forward for up to 2 s (TEL-050), so a collector gets the guest's last batch and not only the spool. Spec 25: TEL-033 and TEL-034 rewritten to the design, TEL-046 added, the invariants and the trust model updated; the design note is docs/specs/25-spec-telemetry/guest-vsock.md. A VM-lane integration test (minvmd/tests/otel_vsock_integration.rs, MINVMD_E2E=1) boots a real guest and checks the host spool, the host's endpoint and the boot line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bryan-minimal
force-pushed
the
otel/7-vsock
branch
from
October 13, 2026 23:52
b627c85 to
1534859
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI run for the stacked PRs #1993–#2039, which the base-branch filter skips. Close after green.