Repository navigation
test(minimal): wait for the daemon spans in the commands-trace test - #3401
Conversation
exec_run_attach_and_task_run_share_the_commands_trace read the span log once, right after the verbs returned. A verb's answer can reach the CLI before the daemon records the trace fields on its request span, and in agent-verify's loaded full-suite runs the test failed every time while CI passed. Poll the log, bounded at 10s, until every verb's request span is there, then assert on that read as before. The lookup moves into daemon_request and request_span so the wait and the assertion use one predicate. Fixes #3398
9af363d to
2946b3f
Compare
|
Approved: every review thread resolved at @norrietaylor resolved the last thread, which releases the approval this review had withheld. Nothing to do. |
There was a problem hiding this comment.
Read at 2946b3f. The diagnosis matches the code: the daemon's request spans declare their trace fields empty and record them later, so one read of the span log right after the verbs return can miss a span that is about to match. Polling the same predicate the assertion uses, with a bound, is the right repair. A real regression still fails with the same panic and span dump, ten seconds later. Test only; the two helpers keep the wait and the assertion from drifting apart.
Two comments, neither blocking:
- The telemetry-off half has the same single read. After the second
run_the_verbsthe test counts theopt-outmarks once and asserts exactly 4. That mark is recorded on the daemon span the same way, after the answer may already have reached the CLI, so on a loaded box this count can come up short for the same reason. The review bot's thread says this and I agree with it. The same bounded poll, waiting for the count to reachmarked_before + 4, would close it here, or in a follow-up if you want this one in quickly since it unblocks the agent PRs. - The wait's predicate does not include
span_id, which the loop reads next for the exec span and unwraps. If the trace fields are recorded in separaterecordcalls, a read can land betweenparent_span_idandspan_id. I have not seen it fail; addings.field("span_id").is_some()torequest_spanwould make the wait cover what the assertions read.
The negative assertion (in_trace(&log, INHERITED).is_empty()) is also a single read, but an early read only makes it weaker, not flaky, so I would leave it.
The telemetry-off half counted the daemon spans' opt-out marks with one read right after the verbs ran. The daemon records the mark on its request span after it answers, so that read raced as the first one did (#3398). Poll for all four marks with the same 10 s bound, then assert.
|
@bryan-minimal one change since your approval at |
There was a problem hiding this comment.
minimal-review — approved
Verdict APPROVE — every review thread is resolved; @norrietaylor closed the last one. The review of 88cd079920ed withheld this approval on the threads open then.
|
@bryan-minimal on your second point: I'm adding |
Root cause
exec_run_attach_and_task_run_share_the_commands_trace(crates/minimal/tests/otel_cli.rs) runs the verbs, then reads the in-process span log once (in_trace(&log, TRACE)) and looks for each verb's daemon request span under itscmdspan.SpanLogrecords a span when it opens and fills its fields as they are recorded. A verb's answer can reach the CLI before the daemon's request span carries the trace fields the lookup matches, so a single read can miss it.In CI the test passes. In agent-verify's full-suite runs on a loaded box it failed in every run today: #3361, #3363, #3372, #3386, #3391, #3392, #3394, #3395. Each one turned an agent PR into a draft.
Fix
daemon_requestandrequest_span, so the wait and the assertion use one predicate.Test only. No product code changes.
Test plan
tests(Linux) runsotel_cligreen. The file is#![cfg(target_os = "linux")], so it does not run on macOS. A local cross-check for Linux was not possible here (aws-lc-sysneeds a target C toolchain).cargo fmt --all -- --checkFixes #3398
Summary by CodeRabbit