Repository navigation
feat(minvmd): forward RUST_LOG to the guest via the kernel cmdline - #872
Conversation
The microVM's /init is minimald itself, so the kernel starts the guest daemon with an empty environment: nothing from the minvmd host process crosses the VM boundary and the guest log level cannot be raised from the host at all. Confirmed empirically — a VMM child holding RUST_LOG=info,russh=debug,minimald=debug in its environment produced zero DEBUG lines in the guest console log, which blocked a live debugging session. libkrun's krun_set_env / krun_set_exec(envp) do not reach this guest: they inject into libkrun's own guest-init path, which this initramfs boot model does not use. The kernel command line does reach it — Linux hands init every unrecognised KEY=VALUE boot token as an environment variable (init/main.c, unknown_bootoption -> envp_init) — and it is already plumbed through VmConfig::apply. Extract kernel_cmdline(), which takes the host value as an argument rather than reading the environment, so the boot line is unit-testable. A value that cannot survive the command line is skipped with a warning instead of corrupting the boot: whitespace would be split into separate boot tokens, an empty value carries nothing, and an over-long one would overrun the kernel's 2048-byte COMMAND_LINE_SIZE. With RUST_LOG unset the line stays byte-identical to before. Note that a forwarded RUST_LOG replaces minimald's default filter outright, including its topiary=off / libcgroups=off directives, and that the boot line is world-readable in the guest as /proc/cmdline, so it must never carry a secret. Both are documented on the helper.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 55 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
Comment |
|
@macroscope review |
|
Manual reviews triggered for commit All prior checks · these links stay valid even if you push more commits. |
|
Just FYI for future @mentions, I'm Review in progress. Previously completed check runs: |
ApprovabilityVerdict: Would Approve Self-contained feature addition that forwards RUST_LOG to guest VMs for debugging purposes. Implementation includes defensive validation with graceful fallback, comprehensive tests, and the author owns this file. Macroscope would have approved this PR. Enable approvability here. |
5 MiB was chosen when the daemons logged at INFO. #872 now forwards the host's RUST_LOG into the guest, and RUST_LOG=debug is a documented workaround on #869, so the bundle most likely to be captured is the one whose daemon is at DEBUG. Measured on a real RUST_LOG=debug capture from this machine, re-enveloped into mlog's JSON-lines shipping format: - 214 B/record, so 5 MiB held ~24.5k records and 32 MiB holds ~157k - log text compresses ~70x in the bundle's zstd stream, so five rotated files cost ~2.2 MiB of archive at the new cap, up from ~0.3 MiB The wire cost is therefore negligible; the cost that binds is uncompressed. add_file_tail buffers a whole tail in memory, so the cap is also peak allocation per file, and a bundle collects up to ~16 log files, bounding extraction near 512 MiB. Going higher buys linear history for linear extraction cost with no categorical win: the same capture produced 36 MB of records in eight seconds, nearly all of it russh chatter rather than minimal's own records, so no per-file cap holds a DEBUG incident. 32 MiB also keeps 2x headroom under MAX_LOG_TAIL_BYTES so --log-tail-bytes stays useful in both directions; a const assertion now enforces that rather than leaving it to be noticed in review. Refs: #869 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ake it a flag (#898) * feat(minimal): add --log-tail-bytes to min bug The diagnostic bundle's per-log tail has been fixed at LOG_TAIL_CAP (5 MiB) since `min bug` landed. That was a comfortable window while the daemons logged at INFO, but #872 now forwards the host's RUST_LOG to the guest daemon over the kernel cmdline, and RUST_LOG=debug is one of the documented workarounds on #869 — so the user most likely to be capturing a bundle is the one whose guest daemon is at DEBUG. There is no intra-day size rotation either, so the current day's file grows without bound while the tail stays fixed, and the captured window can begin well after the incident. The flag defaults to LOG_TAIL_CAP, so an unflagged `min bug` passes the same value to the same call sites and its output is unchanged. Values above the writer's ceiling are rejected at parse time rather than clamped: a clamp reports success while capturing less than was asked for, and the resulting short log then reads as a fact about the system instead of a fact about the flag. MAX_LOG_TAIL_BYTES lives in `diagnostics::bundle` because the reason for the ceiling lives there — add_file_tail seeks -(cap as i64) from the end, so a cap past i64::MAX is a nonsense seek. The guest-side half waits for the daemon bundle RPC: there is no DiagBundleRequest on main, so `min bug` is host-only here. Refs: #869 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(diagnostics,minimal): raise the default log tail cap to 32 MiB 5 MiB was chosen when the daemons logged at INFO. #872 now forwards the host's RUST_LOG into the guest, and RUST_LOG=debug is a documented workaround on #869, so the bundle most likely to be captured is the one whose daemon is at DEBUG. Measured on a real RUST_LOG=debug capture from this machine, re-enveloped into mlog's JSON-lines shipping format: - 214 B/record, so 5 MiB held ~24.5k records and 32 MiB holds ~157k - log text compresses ~70x in the bundle's zstd stream, so five rotated files cost ~2.2 MiB of archive at the new cap, up from ~0.3 MiB The wire cost is therefore negligible; the cost that binds is uncompressed. add_file_tail buffers a whole tail in memory, so the cap is also peak allocation per file, and a bundle collects up to ~16 log files, bounding extraction near 512 MiB. Going higher buys linear history for linear extraction cost with no categorical win: the same capture produced 36 MB of records in eight seconds, nearly all of it russh chatter rather than minimal's own records, so no per-file cap holds a DEBUG incident. 32 MiB also keeps 2x headroom under MAX_LOG_TAIL_BYTES so --log-tail-bytes stays useful in both directions; a const assertion now enforces that rather than leaving it to be noticed in review. Refs: #869 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Problem
The microVM's
/initis minimald (pid-1), so the kernel starts the guest daemon with an empty environment — nothing from the minvmd host process crosses the VM boundary. The guest daemon's log level is therefore unraisable from the host, which blocked a live debugging session: the host VMM child hadRUST_LOG=info,russh=debug,minimald=debugin its environment and zero DEBUG lines reached the guest console log.libkrun's
krun_set_env/krun_set_exec(envp)do not help — they inject into "the context of the executable" for libkrun's own guest-init path (anexec_pathrelative tokrun_set_root), which this initramfs boot model does not use. minvmd does not even bindkrun_set_env.Note
Forward
RUST_LOGfrom host to guest via kernel cmdline inminvmdRUST_LOGfrom the host environment when building the kernel command line and appendsRUST_LOG=<value>to the baseconsole=hvc0cmdline if the value passes validation.kernel_cmdlinefunction rejects empty values, values containing whitespace, and values that would cause the total cmdline to exceed the kernel'sCOMMAND_LINE_SIZE(2048 bytes); rejected values log a warning and fall back to the base cmdline.Cow<'static, str>to avoid allocation when no forwarding is needed.Macroscope summarized 44d72ae.