8000 fix(minimald): join background session work to the trace of the request that caused it by bryan-minimal · Pull Request #3150 · gominimal/minimal · GitHub
[go: up one dir, main page]

Skip to content

fix(minimald): join background session work to the trace of the request that caused it - #3150

Merged
bryan-minimal merged 4 commits into
mainfrom
fix/otel-stray-roots
Nov 10, 2026
Merged

bryan-minimal merged 4 commits into
mainfrom
fix/otel-stray-roots

Conversation

@bryan-minimal
Copy link
Copy Markdown
Member

Found in the nightly OTel health on anvil (job 01958, 2026-11-06, build cand-base-1e236070).
Every telemetry-on scenario reported the same stray roots in mr otel health, once per daemon
start:

000-baseline (cand-base-1e236070-1e236070, pass): 70 spans, minimald joined 80.0% (incl. linked 90.9%), 4 stray traces
   stray roots: minimald:sessions.manager.message x1, minimald:sessions.hand_overs x1, minimald:sessions.publish_moves x1

None of the four strays was caused by a request. Each was work that belongs to a known trace but
started a trace of its own.

Stray What starts it Why it was a root
sessions.hand_overs the sessions manager's init, at daemon start: a loop that lives as long as the manager spawned as an unlinked root of its own, beside sessions.manager, but it is not one of the daemon-lifetime roots the spec lists (TEL-030, architecture.md)
sessions.publish_moves the same, for publish moves (#2748) the same
sessions.manager.message (ResumeActive) Server::run, once at daemon start, before the accept loop sent from no span, so the message minted a fresh trace
the 4th trace: conn -> sessions.reap_unfinalized the teardown of a connection that created a session ran under the connection's conn root, outside the client's trace, though it is the cleanup of the client's CreateSession

A single hand-over or publish move is caused by a request: a destroy, a dynamic-port release, a
publish, or a session-start range walk. Until now that work ran in the loop's root trace, not in
the request's.

What changes

  • The manager's loops run under its root. BoxManager::init makes the sessions.manager
    root first (traced::detach, as spawn_detached did) and spawns the mainloop and both loops
    inside it. sessions.hand_overs and sessions.publish_moves are now children of
    sessions.manager, which is a daemon-lifetime root that both the spec and mr otel health
    already expect. A microVM daemon's start walk (RangeWalk::daemon_start) is spawned in the
    root too, so its log lines and anything it opens are in the manager's trace, not in no span.
  • The loops end with the mainloop. A drop guard on the mainloop's task cancels a token both
    loops select on, so when the mainloop ends (its last sender gone, a panic, the runtime's drop)
    the loops end, their spans close, and the sessions.manager root closes and exports with the
    actor. Without this the loops, which hold the registry's senders through the weak handle,
    outlived the mainloop and held the root open until the runtime dropped them, so a drop after
    the last flush would have turned the boot work into orphans under a root that never arrives.
  • Each hand-over and publish move joins the trace of the work that made it (TEL-022). The
    hostname registry sends each item with the sender's trace context (traced::caller_context()
    at the send, under the registry's lock; it never waits). The loop handles each in a
    sessions.hand_over (port) or sessions.publish_move span that adopts the sender's context,
    as the manager handles its mailbox messages. The manager messages that handling sends are
    children of that span.
  • Work the daemon does by itself joins the manager's trace. The root's ids are noted
    (adopt_trace with no caller) and kept on the handle (lifetime), and on the weak handle, so
    an upgraded handle has them too. The boot ResumeActive
    carries the caller's context, else these ids (traced::Sender::send_from). A hand-over or
    move with no sender context (the daemon-start range walk) gets them too. That work is in the
    sessions.manager trace, not in a root of its own.
  • The teardown reap joins the creating request's trace. The connection records each created
    session with the context of its CreateSession (record_created_session captures it). The reap
    runs one sessions.reap_unfinalized span per session (with session_id, no longer one span
    with count). In the file log the span is a child of conn, as the rpc span is, and in the
    export it is a child of the CreateSession rpc span.
  • Separate commit, can be cherry-picked alone: under PROTOCOL_TRACE=1 the daemon logged
    every channel env request as name=value, which put whatever a client sends in its environment
    (tokens included) into the daemon log. It now logs the name only. It is unrelated to the stray
    traces and is one line in connection.rs with no dependency on the other two commits, so it
    can be cherry-picked onto main by itself if it should land sooner than the rest.

Design choices

  • Children of an expected root, not new expected roots. The task allowed a genuinely
    autonomous loop to stay a root if it is named as an expected one. The spec already has that
    convention: TEL-030 and the architecture table list the long-lived roots, and
    mr otel health mirrors that list in EXPECTED_DAEMON_ROOTS (maintenance,
    sessions.manager, store). Adding two names would have needed a matching change in
    min-regress, and the loops are not independent actors anyway: they start and end with the
    manager. So they became the manager's children, and no reader needs changing. TEL-030 gains a
    note: a loop that lives exactly as long as an actor runs under that actor's root, and a root
    with no link that is not a daemon-lifetime actor reads as a stray trace.
  • The daemon's own work joins the manager's trace, not a fresh root. This is the
    conservative choice: no span is reclassified as expected by name. A hand-over that does carry a
    request always takes the request's context first (caller.or(lifetime)), so a future leak
    still shows up as a span outside the request's trace.
  • A killed daemon. In 711, where the daemon is SIGKILLed, the sessions.manager root never
    exports. The boot resume's message is now its child, so mr otel health counts it as an orphan
    under a live actor's root, not as a stray trace. The PUSH check allows for this and nothing
    else: per scenario, the head's orphans may not exceed the base's, except where the base never
    exported its sessions.manager root either, where the bound is the base's orphans plus strays.

Tests

  • crates/minimald/src/box_manager/tests.rs:
    the_managers_own_work_runs_in_its_root_trace (both loops are children of the
    sessions.manager root; the boot ResumeActive and a hand-over no request made are in the
    root's trace, as its children), the_managers_root_closes_when_its_mainloop_ends (with the
    last handle dropped, the root and both loops close; before, the loops held the root open)
    and a_hand_over_and_a_publish_move_join_the_requests_trace
    (each is a child of the request's rpc span in the client's trace, and its GetSession or
    GetRunning lookup is a child of it).
  • crates/minimald/src/server/tests.rs:
    the_reap_of_an_unfinalized_session_joins_its_creators_trace (through Server::run, whose
    per-connection task runs the teardown: a CreateSession with a TRACEPARENT, then a dropped
    client; the reap is a child of the CreateSession rpc span and stays under conn in the file
    log, and its record read is its child).
  • The net/dns.rs hand-over tests read the (hand-over, context) pairs the channel now
    carries.

Docs: TEL-022 names hand-overs, publish moves and the reap, and lists the new tests. TEL-030
lists its test and carries the note above. The architecture table lists the loops and the
per-item spans.

Lab evidence

Prepush on anvil (CI's exact commands): green (job 05199 at 93ce3f8).

mr otel health otel-stray-roots-c (job 05201), base rel-1e236070 against head cand-otel-stray-roots-c-3030b509 (build job 05200), summed over each scenario's runs:

Scenario Stray traces Orphans (both kinds) sessions.manager exported minimald joined (incl. linked)
000-baseline 8 -> 0 0 -> 0 2/2 -> 2/2 90.9% -> 100.0%
711-otel-spool-survives-kill 2 -> 0 24 -> 26 0/2 -> 0/2 97.5% -> 97.5%
717-otel-nested-min-trace 8 -> 0 0 -> 0 2/2 -> 2/2 93.2% -> 100.0%
710d-graph-clone-stall-blocks-ls 16 -> 0 4 -> 4 2/2 -> 2/2 97.1% -> 100.0%

34 stray traces across the base runs, 0 on the head. Orphans are no more than the base's;
where the base never exported its sessions.manager root (711's SIGKILLed daemon) the bound is
the base's orphans plus its strays, since the boot work there now sits under a root that never
arrives.

Not covered in the lab: no scenario in the lab tree drives a request-side hand-over or publish move (NET-129 shared ports, #2748 moves), so commit 1's request-context path is covered by a_hand_over_and_a_publish_move_join_the_requests_trace only. The matrix covers the daemon-side paths (boot resume, loops) and 710d's request load.

000-baseline (cand-otel-stray-roots-c-3030b509, pass): 70 spans, minimald joined 88.5% (incl. linked 100.0%), 0 stray traces
000-baseline (rel-1e236070, pass): 70 spans, minimald joined 80.0% (incl. linked 90.9%), 4 stray traces
000-baseline (rel-1e236070, pass): 70 spans, minimald joined 80.0% (incl. linked 90.9%), 4 stray traces
000-baseline (cand-otel-stray-roots-c-3030b509, pass): 70 spans, minimald joined 88.5% (incl. linked 100.0%), 0 stray traces
710d-graph-clone-stall-blocks-ls (rel-1e236070, pass): 954 spans, minimald joined 95.4% (incl. linked 97.1%), 8 stray traces
710d-graph-clone-stall-blocks-ls (cand-otel-stray-roots-c-3030b509, pass): 953 spans, minimald joined 98.2% (incl. linked 100.0%), 0 stray traces
710d-graph-clone-stall-blocks-ls (rel-1e236070, pass): 953 spans, minimald joined 95.3% (incl. linked 97.1%), 8 stray traces
710d-graph-clone-stall-blocks-ls (cand-otel-stray-roots-c-3030b509, pass): 954 spans, minimald joined 98.2% (incl. linked 100.0%), 0 stray traces
711-otel-spool-survives-kill (cand-otel-stray-roots-c-3030b509, pass): 48 spans, minimald joined 97.5%, 0 stray traces
711-otel-spool-survives-kill (rel-1e236070, pass): 48 spans, minimald joined 97.5%, 1 stray traces
711-otel-spool-survives-kill (rel-1e236070, pass): 48 spans, minimald joined 97.5%, 1 stray traces
711-otel-spool-survives-kill (cand-otel-stray-roots-c-3030b509, pass): 48 spans, minimald joined 97.5%, 0 stray traces
717-otel-nested-min-trace (rel-1e236070, pass): 138 spans, minimald joined 86.3% (incl. linked 93.2%), 4 stray traces
717-otel-nested-min-trace (cand-otel-stray-roots-c-3030b509, pass): 138 spans, minimald joined 92.9% (incl. linked 100.0%), 0 stray traces
717-otel-nested-min-trace (rel-1e236070, pass): 138 spans, minimald joined 86.3% (incl. linked 93.2%), 4 stray traces
717-otel-nested-min-trace (cand-otel-stray-roots-c-3030b509, pass): 138 spans, minimald joined 92.9% (incl. linked 100.0%), 0 stray traces

Matrix, job 05201:

Lab evidence (min-regress batch otel-stray-roots-c on anvil, fedora, ubuntu-24.04)

Head: 93ce3f8fffdcd4ffc10bbef839cf0bbdbf71781b at 93ce3f8 (lab name cand-otel-stray-roots-c-3030b509). Base: rel-1e236070 (rel-1e236070).

scenario distro base head
000-baseline: baseline (CPU + install sanity; not a bug) fedora pass pass
000-baseline: baseline (CPU + install sanity; not a bug) ubuntu-24.04 pass pass
710d-graph-clone-stall-blocks-ls: found by 710b (2026-09-28): an activation whose package-graph clone stalls blocks min ls (ListSessions) until the clone fails fedora pass pass
710d-graph-clone-stall-blocks-ls: found by 710b (2026-09-28): an activation whose package-graph clone stalls blocks min ls (ListSessions) until the clone fails ubuntu-24.04 pass pass
711-otel-spool-survives-kill: gominimal/inbox#515 Story 1: telemetry spools locally as it runs; crashes and hangs survive fedora pass pass
711-otel-spool-survives-kill: gominimal/inbox#515 Story 1: telemetry spools locally as it runs; crashes and hangs survive ubuntu-24.04 pass pass
717-otel-nested-min-trace: gominimal/inbox#515: a min run inside a session joins the caller's trace (Option A) fedora pass pass
717-otel-nested-min-trace: gominimal/inbox#515: a min run inside a session joins the caller's trace (Option A) ubuntu-24.04 pass pass

Fixed since base: 0. Still reproducing on both: 0. Regressed: 0. Passing on both: 8. A verdict links to the run's trace in Cloud Trace (project access needed); the scenario link is its source with the # issue: it reproduces.

🤖 Generated with Claude Code

8000
bryan-minimal and others added 4 commits November 6, 2026 15:16
…over in its request's

Every telemetry-on lab run showed three stray roots per daemon (lab job 01958):
`sessions.hand_overs` and `sessions.publish_moves`, the two loops the manager starts at
boot, each spawned as an unlinked root of its own, and the `sessions.manager.message` of the
boot ResumeActive, which `Server::run` sends from no span, so it minted a trace.

- The manager's `sessions.manager` root is made first, its ids noted; both loops run under it as
  children, so they leave no trace of their own. A microVM daemon's start walk runs in it too.
- The loops end when the mainloop ends (a drop guard on its task), so the root closes and
  exports with the actor, not at the runtime's drop.
- The registry sends each hand-over and publish move with the trace context of the work that
  made it (a destroy, a release, a publish). The loops handle each in a `sessions.hand_over`
  or `sessions.publish_move` span adopting that context (TEL-022), or the manager root's
  when the daemon made it by itself.
- The boot resume carries the caller's context, else the manager root's (`send_from`). The
  weak handle keeps the root's ids, so an upgraded handle has them too.

Tests: the_managers_own_work_runs_in_its_root_trace,
the_managers_root_closes_when_its_mainloop_ends and
a_hand_over_and_a_publish_move_join_the_requests_trace (box_manager/tests.rs). The span table in
architecture.md and TEL-022/TEL-030 name the new spans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PROTOCOL_TRACE=1 logged every channel env request with its value, so a secret a client sends
in its environment landed in the daemon log. Log the variable's name only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e request that made them

The teardown reap of a session its connection left unfinalized ran in one
`sessions.reap_unfinalized` span under the connection's `conn` root, outside the client's
trace: the fourth stray trace in every lab run that creates a session (000-baseline, 717 in
lab job 01958).

The connection now records each created session with the trace context of its CreateSession
request, and the reap runs one span per session, a child of `conn` in the file log (as the
`rpc` span is) whose export parent is that request's span (TEL-022).

Test: the_reap_of_an_unfinalized_session_joins_its_creators_trace (server/tests.rs), through
`Server::run`: the teardown reap is the accept loop's, which `TestServer::connect` never runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nterim check

The runtime-publications landing test reads the box's listeners from the test
process's network namespace, which concurrent tests share. Another test's any-
address listener inside the range is published too, and the watcher moves its
forwards one port at a time, so prepush 02020 caught 127.0.0.1:40939 still at
the interim after this test's own two ports had moved. The wait now also
awaits every other port's move; this test's own ports still face the
no-interim-row assertion at once. Product code is unchanged: round b's
server.rs, traced.rs, connection.rs and dns.rs are identical to round a's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bryan-minimal
bryan-minimal requested a review from a team as a code owner November 7, 2026 17:47
@coderabbitai
coderabbitai Bot commented Nov 7, 2026
Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 11 billable files and costs up to $2.75.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 54 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: d8945cf3-b95e-485a-9feb-45a056e89d10
Commits

Reviewing files that changed from the base of the PR and between e9be453 and 93ce3f8.

Files selected for processing: 11

  • crates/minimald/src/box_actor/tests.rs
  • crates/minimald/src/box_manager.rs
  • crates/minimald/src/box_manager/handle.rs
  • crates/minimald/src/box_manager/tests.rs
  • crates/minimald/src/connection.rs
  • crates/minimald/src/net/dns.rs
  • crates/minimald/src/server.rs
  • crates/minimald/src/server/tests.rs
  • crates/minimald/src/traced.rs
  • docs/specs/25-spec-telemetry/25-spec-telemetry.md
  • docs/specs/25-spec-telemetry/architecture.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@minimal-foundry
minimal-foundry Bot commented Nov 7, 2026 •
Copy link
Copy Markdown
Contributor

Approved: no findings at 93ce3f8fffdc.

Nothing to do.

Clobber: none found.

@minimal-foundry minimal-foundry Bot left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@minimal-foundry minimal-foundry Bot added the review:approved minimal-review approved the current head label Nov 7, 2026
@agent-137 agent-137 left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review on Karl's behalf, at 93ce3f8ff. This is a comment, not an approval. Nothing was built for it: every point comes from reading the code, and each finding says how it was checked.

  • CI at 93ce3f8f: all green.
  • Reviews: minimal-review APPROVE at head (3/3, no findings). CodeRabbit is rate-limited ("Review limit reached"), so there is no CodeRabbit review on head.

Should-fix: none.

Nit: none.

No findings. What I checked:

  • The sessions.manager root now holds the mainloop and both loops (box_manager.rs).
    • The loops select! on mainloop_done, and a drop guard inside the mainloop's future cancels it.
    • So the root closes when the mainloop ends, whether its senders are gone, it panics, or the runtime drops it.
    • the_managers_root_closes_when_its_mainloop_ends pins this.
  • Each hand-over and publish move adopts caller.or(lifetime). The context is captured at the send in HostnameRegistry (net/dns.rs), under the registry lock. The send is an unbounded send, so it never waits.
  • Reap (server.rs): there is now one sessions.reap_unfinalized span per session.
    • In the file log it stays under conn. In the export it adopts the CreateSession context.
    • When the creator opted out, the context is flags-00 and nothing records, which fits TEL-045.
  • connection.rs: PROTOCOL_TRACE now logs the env-request name only. This is a real secret-leak fix, and it is in its own commit (3e01c64e6).
  • Tests: they assert parent and trace ids against the client trace and against the root's ids. They are not just presence checks.
  • box_actor/tests.rs: the flake fix there is in its own commit (93ce3f8ff) and is unrelated to tracing.

Verdict: ready to approve.

@agent-137 agent-137 left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 93ce3f8ff. The earlier comment review found nothing that needs fixing, and its nits are optional.

@bryan-minimal
bryan-minimal added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 6a3376d Nov 10, 2026
37 checks passed
@bryan-minimal
bryan-minimal deleted the fix/otel-stray-roots branch November 10, 2026 01:03
bryan-minimal added a commit that referenced this pull request Nov 10, 2026
Main's #3150 worked around another test's listener standing at the
interim; under this branch the watcher keeps to the listeners this test
process holds, so the merged test keeps the branch's wording and drops
the filter the workaround added.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved minimal-review approved the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0