Repository navigation
fix(minimald): join background session work to the trace of the request that caused it - #3150
Conversation
…over in its request's Every telemetry-on lab run showed three stray roots per daemon (lab job 01958): `sessions.hand_overs` and `sessions.publish_moves`, the two loops the manager starts at boot, each spawned as an unlinked root of its own, and the `sessions.manager.message` of the boot ResumeActive, which `Server::run` sends from no span, so it minted a trace. - The manager's `sessions.manager` root is made first, its ids noted; both loops run under it as children, so they leave no trace of their own. A microVM daemon's start walk runs in it too. - The loops end when the mainloop ends (a drop guard on its task), so the root closes and exports with the actor, not at the runtime's drop. - The registry sends each hand-over and publish move with the trace context of the work that made it (a destroy, a release, a publish). The loops handle each in a `sessions.hand_over` or `sessions.publish_move` span adopting that context (TEL-022), or the manager root's when the daemon made it by itself. - The boot resume carries the caller's context, else the manager root's (`send_from`). The weak handle keeps the root's ids, so an upgraded handle has them too. Tests: the_managers_own_work_runs_in_its_root_trace, the_managers_root_closes_when_its_mainloop_ends and a_hand_over_and_a_publish_move_join_the_requests_trace (box_manager/tests.rs). The span table in architecture.md and TEL-022/TEL-030 name the new spans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PROTOCOL_TRACE=1 logged every channel env request with its value, so a secret a client sends in its environment landed in the daemon log. Log the variable's name only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e request that made them The teardown reap of a session its connection left unfinalized ran in one `sessions.reap_unfinalized` span under the connection's `conn` root, outside the client's trace: the fourth stray trace in every lab run that creates a session (000-baseline, 717 in lab job 01958). The connection now records each created session with the trace context of its CreateSession request, and the reap runs one span per session, a child of `conn` in the file log (as the `rpc` span is) whose export parent is that request's span (TEL-022). Test: the_reap_of_an_unfinalized_session_joins_its_creators_trace (server/tests.rs), through `Server::run`: the teardown reap is the accept loop's, which `TestServer::connect` never runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nterim check The runtime-publications landing test reads the box's listeners from the test process's network namespace, which concurrent tests share. Another test's any- address listener inside the range is published too, and the watcher moves its forwards one port at a time, so prepush 02020 caught 127.0.0.1:40939 still at the interim after this test's own two ports had moved. The wait now also awaits every other port's move; this test's own ports still face the no-interim-row assertion at once. Product code is unchanged: round b's server.rs, traced.rs, connection.rs and dns.rs are identical to round a's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reached
This review includes 11 billable files and costs up to $2.75.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 54 minutes for your next included review. View limit details
|
|
Approved: no findings at Nothing to do. Clobber: none found. |
There was a problem hiding this comment.
Review on Karl's behalf, at 93ce3f8ff. This is a comment, not an approval. Nothing was built for it: every point comes from reading the code, and each finding says how it was checked.
- CI at
93ce3f8f: all green. - Reviews: minimal-review APPROVE at head (3/3, no findings). CodeRabbit is rate-limited ("Review limit reached"), so there is no CodeRabbit review on head.
Should-fix: none.
Nit: none.
No findings. What I checked:
- The
sessions.managerroot now holds the mainloop and both loops (box_manager.rs).- The loops
select!onmainloop_done, and a drop guard inside the mainloop's future cancels it. - So the root closes when the mainloop ends, whether its senders are gone, it panics, or the runtime drops it.
the_managers_root_closes_when_its_mainloop_endspins this.
- The loops
- Each hand-over and publish move adopts
caller.or(lifetime). The context is captured at the send inHostnameRegistry(net/dns.rs), under the registry lock. The send is an unboundedsend, so it never waits. - Reap (
server.rs): there is now onesessions.reap_unfinalizedspan per session.- In the file log it stays under
conn. In the export it adopts the CreateSession context. - When the creator opted out, the context is flags-00 and nothing records, which fits TEL-045.
- In the file log it stays under
connection.rs:PROTOCOL_TRACEnow logs the env-request name only. This is a real secret-leak fix, and it is in its own commit (3e01c64e6).- Tests: they assert parent and trace ids against the client trace and against the root's ids. They are not just presence checks.
box_actor/tests.rs: the flake fix there is in its own commit (93ce3f8ff) and is unrelated to tracing.
Verdict: ready to approve.
There was a problem hiding this comment.
Approving at 93ce3f8ff. The earlier comment review found nothing that needs fixing, and its nits are optional.
Main's #3150 worked around another test's listener standing at the interim; under this branch the watcher keeps to the listeners this test process holds, so the merged test keeps the branch's wording and drops the filter the workaround added. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Found in the nightly OTel health on anvil (job 01958, 2026-11-06, build
cand-base-1e236070).Every telemetry-on scenario reported the same stray roots in
mr otel health, once per daemonstart:
None of the four strays was caused by a request. Each was work that belongs to a known trace but
started a trace of its own.
sessions.hand_overssessions.manager, but it is not one of the daemon-lifetime roots the spec lists (TEL-030, architecture.md)sessions.publish_movessessions.manager.message(ResumeActive)Server::run, once at daemon start, before the accept loopconn->sessions.reap_unfinalizedconnroot, outside the client's trace, though it is the cleanup of the client's CreateSessionA single hand-over or publish move is caused by a request: a destroy, a dynamic-port release, a
publish, or a session-start range walk. Until now that work ran in the loop's root trace, not in
the request's.
What changes
BoxManager::initmakes thesessions.managerroot first (
traced::detach, asspawn_detacheddid) and spawns the mainloop and both loopsinside it.
sessions.hand_oversandsessions.publish_movesare now children ofsessions.manager, which is a daemon-lifetime root that both the spec andmr otel healthalready expect. A microVM daemon's start walk (
RangeWalk::daemon_start) is spawned in theroot too, so its log lines and anything it opens are in the manager's trace, not in no span.
loops select on, so when the mainloop ends (its last sender gone, a panic, the runtime's drop)
the loops end, their spans close, and the
sessions.managerroot closes and exports with theactor. Without this the loops, which hold the registry's senders through the weak handle,
outlived the mainloop and held the root open until the runtime dropped them, so a drop after
the last flush would have turned the boot work into orphans under a root that never arrives.
hostname registry sends each item with the sender's trace context (
traced::caller_context()at the send, under the registry's lock; it never waits). The loop handles each in a
sessions.hand_over(port) orsessions.publish_movespan that adopts the sender's context,as the manager handles its mailbox messages. The manager messages that handling sends are
children of that span.
(
adopt_tracewith no caller) and kept on the handle (lifetime), and on the weak handle, soan upgraded handle has them too. The boot
ResumeActivecarries the caller's context, else these ids (
traced::Sender::send_from). A hand-over ormove with no sender context (the daemon-start range walk) gets them too. That work is in the
sessions.managertrace, not in a root of its own.session with the context of its CreateSession (
record_created_sessioncaptures it). The reapruns one
sessions.reap_unfinalizedspan per session (withsession_id, no longer one spanwith
count). In the file log the span is a child ofconn, as therpcspan is, and in theexport it is a child of the CreateSession
rpcspan.PROTOCOL_TRACE=1the daemon loggedevery channel env request as
name=value, which put whatever a client sends in its environment(tokens included) into the daemon log. It now logs the name only. It is unrelated to the stray
traces and is one line in
connection.rswith no dependency on the other two commits, so itcan be cherry-picked onto main by itself if it should land sooner than the rest.
Design choices
autonomous loop to stay a root if it is named as an expected one. The spec already has that
convention: TEL-030 and the architecture table list the long-lived roots, and
mr otel healthmirrors that list inEXPECTED_DAEMON_ROOTS(maintenance,sessions.manager, store). Adding two names would have needed a matching change in
min-regress, and the loops are not independent actors anyway: they start and end with the
manager. So they became the manager's children, and no reader needs changing. TEL-030 gains a
note: a loop that lives exactly as long as an actor runs under that actor's root, and a root
with no link that is not a daemon-lifetime actor reads as a stray trace.
conservative choice: no span is reclassified as expected by name. A hand-over that does carry a
request always takes the request's context first (
caller.or(lifetime)), so a future leakstill shows up as a span outside the request's trace.
sessions.managerroot neverexports. The boot resume's message is now its child, so
mr otel healthcounts it as an orphanunder a live actor's root, not as a stray trace. The PUSH check allows for this and nothing
else: per scenario, the head's orphans may not exceed the base's, except where the base never
exported its
sessions.managerroot either, where the bound is the base's orphans plus strays.Tests
crates/minimald/src/box_manager/tests.rs:the_managers_own_work_runs_in_its_root_trace(both loops are children of thesessions.managerroot; the boot ResumeActive and a hand-over no request made are in theroot's trace, as its children),
the_managers_root_closes_when_its_mainloop_ends(with thelast handle dropped, the root and both loops close; before, the loops held the root open)
and
a_hand_over_and_a_publish_move_join_the_requests_trace(each is a child of the request's
rpcspan in the client's trace, and its GetSession orGetRunning lookup is a child of it).
crates/minimald/src/server/tests.rs:the_reap_of_an_unfinalized_session_joins_its_creators_trace(throughServer::run, whoseper-connection task runs the teardown: a CreateSession with a
TRACEPARENT, then a droppedclient; the reap is a child of the CreateSession
rpcspan and stays underconnin the filelog, and its record read is its child).
net/dns.rshand-over tests read the(hand-over, context)pairs the channel nowcarries.
Docs: TEL-022 names hand-overs, publish moves and the reap, and lists the new tests. TEL-030
lists its test and carries the note above. The architecture table lists the loops and the
per-item spans.
Lab evidence
Prepush on anvil (CI's exact commands): green (job 05199 at 93ce3f8).
mr otel health otel-stray-roots-c(job 05201), base rel-1e236070 against headcand-otel-stray-roots-c-3030b509(build job 05200), summed over each scenario's runs:sessions.managerexported34 stray traces across the base runs, 0 on the head. Orphans are no more than the base's;
where the base never exported its
sessions.managerroot (711's SIGKILLed daemon) the bound isthe base's orphans plus its strays, since the boot work there now sits under a root that never
arrives.
Not covered in the lab: no scenario in the lab tree drives a request-side hand-over or publish move (NET-129 shared ports, #2748 moves), so commit 1's request-context path is covered by
a_hand_over_and_a_publish_move_join_the_requests_traceonly. The matrix covers the daemon-side paths (boot resume, loops) and 710d's request load.Matrix, job 05201:
Lab evidence (min-regress batch
otel-stray-roots-con anvil, fedora, ubuntu-24.04)Head:
93ce3f8fffdcd4ffc10bbef839cf0bbdbf71781bat 93ce3f8 (lab namecand-otel-stray-roots-c-3030b509). Base:rel-1e236070(rel-1e236070).min ls(ListSessions) until the clone failsmin ls(ListSessions) until the clone failsminrun inside a session joins the caller's trace (Option A)minrun inside a session joins the caller's trace (Option A)Fixed since base: 0. Still reproducing on both: 0. Regressed: 0. Passing on both: 8. A verdict links to the run's trace in Cloud Trace (project access needed); the scenario link is its source with the
# issue:it reproduces.🤖 Generated with Claude Code