8000 fix(base): force IPv4 on the classifier call, and log why it failed by simon-inkie · Pull Request #19 · simon-inkie/inkie-deck · GitHub
[go: up one dir, main page]

Skip to content

fix(base): force IPv4 on the classifier call, and log why it failed - #19

Merged
simon-inkie merged 3 commits into
mainfrom
chore/port-stuck-sweep-ipv4-errorlog
Sep 13, 2026
Merged

simon-inkie merged 3 commits into
mainfrom
chore/port-stuck-sweep-ipv4-errorlog

Conversation

@simon-inkie
Copy link
Copy Markdown
Owner

Ports a fix from the internal deployment this package was extracted from into packages/base/bin/stuck-pane-sweep.sh's model-tier classifier call (model_raw()).

What changed

  1. --ipv4 on the classifier's curl call. Some hosts have no working IPv6 route to the endpoint but DNS still returns an AAAA record for it, so a default dual-stack curl can try v6 first and stall to the full timeout before falling back to v4. Generic robustness fix, not specific to this tool.
  2. Log the HTTP status and first 300 bytes of the response body on every classifier failure path, to a new append-only log (CLASSIFIER_ERROR_LOG, env-overridable via STUCK_SWEEP_CLASSIFIER_ERROR_LOG, defaulting under STATE_DIR next to the spend ledger). curl -sS doesn't fail on a non-2xx status, so without this a quota outage, a server fault, and a genuine transport failure are indistinguishable from each other and from routine noise in the sweep's own records. Only the response is ever logged, never the request or its headers (the API key travels in a header and must never reach this file).

Verification

  • 130 → 146 assertions, all green. pane-idle (49), inbox-push (89), activity-sweep (84) unaffected.
  • Revert-to-red on three narrow reverts (the --ipv4 flag alone, the three log call sites alone, the 300-byte cap alone), each reddening only its own assertions. Independently re-verified myself: dropped --ipv4 alone, confirmed exactly 1 assertion reddens, restored.
  • leak-gate.mjs, licence-gate.mjs, lib-version-gate.mjs all clean.

Built by a sub-agent, independently reviewed by me: read the full diff, re-ran the suite and all three gates myself, and did my own revert-to-red spot check rather than trusting the report alone.

Note for whoever merges this alongside #(line-anchor PR): both touch packages/base/test/stuck-pane-sweep.test.sh by appending a new section at the same tail location, so whichever merges second will show a trivial append-only conflict there. Ping me and I'll rebase it.

🤖 Generated with Claude Code

Two independent changes, both inside model_raw(), the file's only curl call
site.

1. --ipv4 on the classifier request. Some hosts have no working IPv6 route to
   the endpoint but DNS still returns an AAAA record for it, so a default
   dual-stack curl can try v6 first and stall to the full --max-time before
   falling back, rather than trying v4 straight away. This is a generic curl
   dual-stack robustness fix, not specific to this tool: any self-hosted
   deployment on a similarly-configured host can hit the identical stall.
   Forcing v4 removes the failure mode outright rather than only diagnosing
   it faster.

2. Log the HTTP status and the first 300 bytes of the response body on every
   failure path, to a new append-only log (CLASSIFIER_ERROR_LOG, env-overridable
   as STUCK_SWEEP_CLASSIFIER_ERROR_LOG, defaulting under STATE_DIR next to the
   spend ledger -- same directory, same override convention). Deliberately NOT
   the state file, which other things parse and must not grow a variable-shape
   error blob. `curl -sS` does not fail on a non-2xx status, so the status is
   now asked for explicitly via `-w` and split from the body; three failure
   paths are distinguished and logged separately: transport failure, HTTP
   error status, and an unparseable 2xx. Without this a "model-unavailable"
   verdict carries no payload at all, so a quota outage, a server fault and a
   genuine network failure are indistinguishable from routine noise and from
   each other in the sweep's own records.

Only the response is ever logged, never the request or its headers -- the API
key travels in the x-goog-api-key header and must never reach this file. The
300-byte cap is a deliberate redaction on a response body of unknown shape,
not a formatting preference. Failing to write the log is non-fatal by design:
an observability addition must not become a new way for the classifier itself
to break.

Tests: 130 -> 146 assertions (16 new). None of the existing STUCK_MODEL_CMD-
seamed cases exercise any of this -- that seam stands in for the entire call
and has no argv, HTTP status or curl exit code of its own to assert on -- so
the new section adds a stub curl on PATH and drives the real (non-seamed)
branch instead, recording its own argv and emulating `-w '\n%{http_code}'`
(body, newline, status) so every failure shape can be dialled in with no
network call ever made.

Revert-to-red, three narrow reverts against the finished change:
  - dropping --ipv4 alone reddens exactly the IPv4 assertion (1 fail)
  - neutering the three classifier_error_log call sites alone reddens exactly
    the nine log-content assertions (9 fails), leaving the IPv4 assertions and
    the model-unavailable/success controls green -- i.e. those two properties
    are genuinely independent of the logging calls
  - widening the 300-byte cap to 100000 alone reddens exactly the truncation
    assertion (1 fail, got 624)

inbox-push, pane-idle and activity-sweep suites are unaffected by this change
(89/49/84 passed, unchanged from baseline).

Ported from the internal deployment this package was extracted from, and
genericised on the way: that change's incident references, internal paths and
private document citations are not reproduced here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Simon and others added 2 commits September 13, 2026 18:17
Follow-up to ecc3787 on this branch (not amended -- kept separate so the
review history stays honest about what changed in response to feedback).
A model-diverse peer reviewer (Codex-runtime) found a real, confirmed gap in
that commit: it wired CLASSIFIER_ERROR_LOG into model_raw(), which only sees
failures at the HTTP/envelope level. model_judge(), the caller, does further
validation AFTER model_raw() succeeds -- extracts .verdict via jq, checks it
is stuck/clear -- and on any failure there it just returned 1 with zero
logging. A well-formed HTTP response whose own generated text is not JSON
(the reviewer's confirmed repro,
`{"candidates":[{"content":{"parts":[{"text":"not JSON"}]}}]}`) read as
"unavailable" with no diagnostic trail: exactly the blind spot the whole
feature was built to close, one layer up.

Three new failure paths, each logged via the existing classifier_error_log()
helper (no new mechanism):
  - jq -er '.verdict' failing (text isn't valid JSON, or lacks .verdict) ->
    kind=unparseable, REUSED from model_raw()'s own case rather than invented
    fresh. Conceptually the same failure -- a well-formed response whose
    CONTENT doesn't parse as expected -- one layer up. Matches the reviewer's
    own test, which used this kind for exactly this scenario.
  - a valid .verdict value outside {stuck, clear} -> kind=invalid-verdict, a
    NEW and DISTINCT kind. This is not a parse failure, it's an enum violation
    on content that parsed cleanly, and conflating it with "the model didn't
    even return JSON" would hide a real distinction from the operator this log
    is for.
  - empty extracted text (the existing `[ -n "$raw" ] || return 1` guard) ->
    kind=empty-text, also new: nothing to attempt to parse, different again
    from "got something and it was garbage".
Since http-code/curl-exit genuinely don't apply at this layer, both columns
are logged as "n/a" rather than a misleading 200/0.

THE SEAM QUESTION, made explicit rather than left to infer: model_raw()'s own
logging is deliberately unreachable through STUCK_MODEL_CMD (documented
there -- an injected seam has no HTTP status or curl exit to report). That
reasoning does NOT carry over to this new logging: model_judge() never asks
whether $raw came from curl or the seam, and a STUCK_MODEL_CMD-swapped
provider returning a malformed verdict is just as real a production failure
as the built-in provider doing so (the seam's own doc comment names "a swap
to any other provider" as a legitimate configuration, not only a test
double). Said explicitly in a comment at both the classifier_error_log() and
model_judge() ends, and proven by a test that drives the same malformed-text
shape through STUCK_MODEL_CMD instead of curl. Checked whether any existing
test asserted (implicitly or explicitly) that a malformed verdict through the
seam produced no log line: none does -- the seam's existing coverage (the
"model-down" stub, a transport-style exit 1) never inspects
CLASSIFIER_ERROR_LOG at all, so nothing needed to change, only to be added.

Also fixed, flagged by the same reviewer, non-blocking: the 300-byte
truncation cap on logged bodies was described in a code comment (and in
ecc3787's own commit message, which cannot be edited after the fact) as "a
deliberate redaction". That overstates what it proves -- truncating arbitrary
text bounds log size, it does not redact content, and the existing test only
shows the API key doesn't reach the log via a fixture that never reflects it
in the body, not that truncation redacts anything in general. Reworded the
classifier_error_log() doc comment to say what the cap actually does and
does not guarantee.

Tests: 146 -> 156 assertions (10 new): the reviewer's repro verbatim, the
invalid-verdict-enum case, the empty-text case, and the seam-parity case.
Revert-to-red on exactly the three new classifier_error_log() call sites: 6
of the 10 new assertions go red (the ones reading log content), the other 4
(the tier=model-unavailable classification, unaffected by logging) stay
green, and nothing pre-existing moves -- 150 passed, 6 failed on the reverted
tree; restored back to 156/0.

Full suite (vitest wrapper): 45/45 across all four ported suites. Adjacent
suites run standalone, unaffected: pane-idle 49/49, inbox-push 89/89,
activity-sweep 84/84, inbox 69/69, boot-agents 33/33. Gates: leak-gate,
licence-gate and lib-version-gate all clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A model-diverse peer review confirmed an exploitable regression introduced
by 9246343 on this branch. STUCK_MODEL_CMD's seam subprocess runs with the
full inherited shell environment, which includes
GOOGLE_GENERATIVE_AI_API_KEY whenever this tier is reachable at all (the
curl path needs it too, so it is present on every run that gets this far).
With STUCK_MODEL_CMD='printenv GOOGLE_GENERATIVE_AI_API_KEY', that prints
the real key to stdout; model_judge() correctly classifies it as an
unparseable verdict; and 9246343's own new logging then wrote the literal
key into CLASSIFIER_ERROR_LOG. Before that commit the same leak failed
with `return 1` and no persistent record -- 9246343 made a latent leak into
a written one.

Two independent fixes, deliberately not just one:

1. ROOT FIX, in model_raw(): the seam subprocess is now invoked via
   `env -u GOOGLE_GENERATIVE_AI_API_KEY --`, so it genuinely cannot read,
   echo, or crash-dump the key by any means. A provider swap -- the whole
   point of STUCK_MODEL_CMD -- has no legitimate need for this variable; a
   different provider authenticates through its own mechanism.
2. BACKSTOP, in classifier_error_log(): every body, on every call site, now
   has any literal occurrence of the configured key redacted to
   "[REDACTED]" before being written, ahead of the existing 300-byte cap
   (redacting first, not after, so truncation can't leave a partial
   fragment past the cut). Belt-and-braces -- it closes the "the key never
   reaches the log" claim even against a leak the env fix cannot see, and
   is proven to do exactly that by a test where the key reaches the log
   body through a path other than the environment.

Each fix is tested alone as well as together, so neither test result is
masking a failure in the other, plus the reviewer's own exact repro
end-to-end. All three are revert-to-red checked: reverting either fix in
isolation fails exactly its own tests and no others.

Also closes the named functional gap in the same review: a nonzero
STUCK_MODEL_CMD exit was never logged, unlike every other failure path in
this tier (kind=seam-exit, in model_raw(), reusing the curl_exit column for
the seam's own exit status since there is no HTTP code on this path). The
existing model-down test exercised this path already; it now asserts on
the log too.

Fixed in passing: model_judge() checked for empty model text BEFORE
stripping a ```json fence, so a response that is only an empty fence
(e.g. "```json\n```") fence-stripped to empty AFTER the check and was
misclassified as "unparseable" garbage instead of "empty-text". The strip
now runs first, with a single emptiness check after.

Left alone, as explicitly out of scope for this round: widening
classifier_error_log()'s control-character handling beyond the existing
newline/CR/tab normalisation. Doing it safely (without corrupting
legitimate non-ASCII pane text under a naive tr/printable-class approach)
is more than a "keep it simple" fix, so it is left for a dedicated pass
rather than rushed alongside a security fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@simon-inkie
simon-inkie merged commit c762f76 into main Sep 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0