8000 feat(base): transactional Codex provider switch and quota telemetry by simon-inkie · Pull Request #44 · simon-inkie/inkie-deck · GitHub
[go: up one dir, main page]

Skip to content

feat(base): transactional Codex provider switch and quota telemetry - #44

Merged
simon-inkie merged 1 commit into
mainfrom
feat/multi-provider-runtime-and-telemetry
Oct 4, 2026
Merged

simon-inkie merged 1 commit into
mainfrom
feat/multi-provider-runtime-and-telemetry

Conversation

@simon-inkie
Copy link
Copy Markdown
Owner

What

Finishes the multi-provider runtime and quota telemetry branch: a Codex agent can be moved between the Plus plan and an Azure/OpenAI-compatible endpoint without losing its agent envelope, and the deck shows the resulting quota state.

Two halves:

  • Runtime (packages/base/bin/boot-agents.sh, codex-provider-switch, provider-switch-lock): the canonical Codex launch command now carries the full provider envelope across both the initial launch and the crash fallback, including CODEX_HOME, model and the agent's .env.secrets. The switch itself is a transactional five-stage flow, prepare -> launch -> verify-canary -> commit, serialised by an on-disk lock, with a short-lived confirmation token bound to the exact pane and provider. Both directions are supported.
  • Telemetry (packages/deck/app/lib/codex-quota*): observation, storage and display of Codex quota, plus the status component.

Why it is transactional

Switching the provider means killing a live Codex process and relaunching it against a different home. A half-done switch leaves an agent with no working runtime, so the flow is deliberately staged: nothing is committed until a canary turn has been verified in the new rollout, and any interruption leaves the authoritative provider state pointing at the old provider.

Reviewer notes, and please read these

  • This is the riskiest item in the current deck queue. It kills and relaunches live agent panes. A bug here can take an agent down mid-task, and the failure mode is a dead pane rather than an error.
  • The default launch path is unchanged when no switch is in progress; the transactional path only runs when explicitly invoked.
  • Credentials are never written by this code. The Azure key reaches the process the same way it does today, through the agent's gitignored .env.secrets sourced in the pane shell. Worth a reviewer confirming that reading rather than taking it on trust.
  • --model is appended when the agent has a .model file; the resume path is explicitly refused (provider switch must never resume a physical thread).
  • Recovery: if the azure canary never verifies, commit is never reached, so the provider state still names the old provider. The stale lock can be inspected with provider-switch-lock.
  • This branch is green as-is on main, which is part of why this review matters more than usual: a clean suite here does not exercise a real tmux switch.

Review legs

  • Remote CodeRabbit unavailable (subscription, 2 Oct). Not treated as a clean review.
  • Peer reviewer: pending, cross-lane.
  • Independent model-diverse check: pending, and required. This touches process lifecycle and credentials, so it wants the deepest read of the four PRs.
  • Suites: full workspace green on this branch, deck 2124 passed / 11 skipped (102 files), base 46 passed (5 files), typecheck clean, check:leaks clean (402 files, 30 rules).

Not tested

  • No live provider switch was performed. The flow is covered by unit and shell tests (codex-provider-switch.test.py, boot-agents.test.sh), not by killing a real pane.
  • The canary verification reads a real rollout file; only its parsing is tested here.
  • No quota telemetry against a live quota endpoint.

OSS-clean

Public repo. Leak gate clean on this branch; no keys, no /home paths, no commercial or pricing content.

Port transactional Codex provider switch with delivery exclusion lock, quota telemetry observer, store, and UI indicators. Sanitize public fixtures and enforce clean-room boundaries.
@simon-inkie
Copy link
Copy Markdown
Owner Author

Test plan: disposable agent home exercise

This is the live exercise that must pass before this branch goes near main. Agreed with Io 2026-10-02. It is deliberately written to be run by hand on a throwaway home, not against a real agent.

The unit and shell suites cover argv construction, the lock, the token binding and the canary parsing. They do not cover killing a real pane and getting a real Codex back up. That is the whole risk, so it is the whole of this plan.

Safety rules

  • Run against a disposable home only. Never a real agent's home, never ~/.the-vault.
  • Take a copy of the real provider state file before starting; the exercise writes the authoritative state under PROVIDER_SWITCH_STATE_ROOT.
  • If the pane is left dead at any point, do not commit. Hit rollback (or abort before launch) and record what happened.
  • One agent, one worktree, no other agent homes touched.

Setup

export PROVIDER_SWITCH_AGENTS_ROOT=/tmp/ps-exercise/agents
export PROVIDER_SWITCH_STATE_ROOT=/tmp/ps-exercise/state
export PROVIDER_SWITCH_TMUX_SESSION=ps-exercise
mkdir -p "$PROVIDER_SWITCH_AGENTS_ROOT/psagent"

Provision the disposable home the same way a real one is provisioned: .runtime containing codex, an isolated .codex-home/ containing a valid config.toml, a gitignored .env.secrets holding the Azure key, and a .model file if the model path is to be exercised. Start a tmux session named ps-exercise with one window named psagent, and put a Codex process in it.

Acceptance table

# Step Exact check Pass condition
1 status psagent prints current provider names plus (the prepare precondition)
2 prepare psagent returns transaction id and token JSON with transaction_id, confirmation_token, expires_at
3 token binding re-run launch with a token for a different pane refuses: confirmation token binding does not match pane/provider
4 token expiry run launch after token_expires_at refuses: confirmation token expired
5 token reuse run launch twice with the same token second refuses: confirmation token was already used
6 launch watch the pane old Codex terminated, fresh Codex up, canary [provider-switch-canary nonce=...] Reply exactly: CANARY OK sent
7 status psagent before verify prints stage canary-sent, provider still plus
8 verify-canary finds the rollout in the azure home prints stage: verified and a thread_id
9 commit writes provider state prints stage: committed; state file names azure and the new thread
10 relaunch kill the pane, reboot the agent comes up on azure, resumes the committed thread, no resume of the old physical thread
11 reverse direction prepare then launch back to plus canary verifies, commit writes provider plus, codex_home null

Failure paths, each its own run

# Injected condition Expected behaviour
12 Canary never answers (kill the pane after launch) verify-canary refuses; rollback restores a fresh Plus pane; provider state ends plus
13 Wrong pane becomes active before verify verify-canary refuses: exact pane changed before canary verification
14 Stale lock from a killed run status reports the lock; rollback clears it and names the owner
15 Azure home missing config.toml prepare refuses: isolated Azure home is not provisioned
16 .runtime says claude prepare refuses: manual provider switch supports Codex agents only
17 Pane cwd is not the agent home prepare refuses: pane working directory is not the agent home
18 Provider state already azure but prepare called refuses: prepare requires current provider state to be plus

What "done" looks like

  • All of 1 to 11 pass on the disposable home, and each of 12 to 18 was observed to refuse or recover as stated.
  • The pane is alive at the end of every run, on the right provider.
  • No .env.secrets content appears in any audit record, transaction file or stdout.
  • The audit log (append_audit) shows one entry per stage, in order, for each run.

What I have not covered here

  • The crash-fallback half of the launch envelope (a Codex that dies mid-boot and is relaunched). Worth one extra manual run: kill the fresh Codex within the settle window and confirm wait_for_clean_codex either recovers or fails closed rather than reporting success on a dead pane.
  • Quota telemetry against a live endpoint. Separate, lower risk, does not gate this branch.

Post the results back here as a comment before asking for merge. If any row in 12 to 18 reports success instead of refusing, stop: that is the failure mode this branch exists to prevent.

@simon-inkie
Copy link
Copy Markdown
Owner Author 8000

Review, Sol 6.1, 2026-10-02

Reviewed SHA 115ad04.

  • Major — packages/base/bin/codex-provider-switch:464: The durable lock identifies the transaction but does not serialise commands within it. Two concurrent launch calls can both read prepared, validate the same unused token, then kill/relaunch the pane. A read-only mocked reproduction confirmed both launches succeeded. Token consumption needs atomic exclusion.

  • Major — packages/base/bin/codex-provider-switch:604: finalise persists finalised before releasing the lock. Interruption, audit failure or release failure leaves delivery and boot blocked, while retrying finalise rejects that stage. A mocked release failure confirmed this. Finalisation needs an idempotent recovery path; abort has the equivalent ordering problem.

  • Major — packages/base/bin/codex-provider-switch:559: Rollback treats successful send-keys as restoration of Plus, immediately persisting rolled-back. It never checks whether Codex starts successfully. Authentication/configuration failure can leave a bare shell, yet finalisation releases delivery and reports successful recovery. The rollback test mocks the send and therefore misses this failure.

  • Minor — packages/base/bin/boot-agents.sh:311: The launch envelope sources .env.secrets, but none of the repository’s ignore files excludes that filename. With the supported AGENTS_HOME override pointing inside the clone, a key-bearing file can be accidentally committed. Add an ignore rule or enforce/document an external, ignored agent root.

I found no credential-writing code or added private names, machine-specific paths or Inkie paid-feature disclosures. Azure-only credential ingress is not enforced: the launcher also inherits its environment and trusts the provisioned Codex configuration.

I did not run the supplied tests because they write fixtures, or test live tmux, real authentication, Azure canaries or App Server telemetry. Verification was source review plus read-only, in-memory reproductions.

@simon-inkie
Copy link
Copy Markdown
Owner Author

Review, Sonnet, 2026-10-03

PR 44, feat(base): transactional Codex provider switch and quota telemetry
Head SHA reviewed: 115ad04
Verdict: NEEDS CHANGES (0 Critical, 6 Major)

Summary

The staged design is sound (prepare, launch, verify-canary, commit, with rollback/abort/finalise), the confirmation token is hashed, single-use, expiring and pane-bound, inputs are regex-validated before they reach the shell, the Azure key is only read from the agent's .env.secrets and never written, and the canary requires a nonce in a user record followed by an assistant "CANARY OK" in a new rollout. Shell quoting in codex_launch_command uses %q for agent, role, home and session id. The gaps are in what the lock actually protects, whether the flow can start at all, one data-loss path in rollback, and two telemetry reliability and logic problems.

Critical

None found. No live switch was run, so the process-lifecycle paths are reasoned from the code only.

Major

M1. The lock only pauses the CLI push and boot, not the deck's own pane writers

packages/base/bin/inbox-push (new gate near line 348) and boot-agents.sh launch_agent check provider-switch-lock active. The deck process (packages/deck/app/lib/tmux.server.ts: wakeIdleAgent, pushMessageToIdleAgent, sendKeysToWindow, api.tmux-send) is not touched and never looks at the lock. A deck push during launch can type into the pane between the quiescence check and C-c, into the fresh shell between termination and relaunch, or into the new composer before the canary, corrupting the canary line or executing text at a shell. The PR text says delivery is serialised by the lock; it is for one of two delivery paths.

Fix: in the deck, refuse sends while <state_root>/<agent>/provider-switch.lock exists (a plain fs.existsSync mirroring the shell check, with the same PROVIDER_SWITCH_STATE_ROOT default), returning a typed not-attempted result. Coordinate with PR 43, which rewrites the same functions.

M2. prepare cannot start for any existing agent: nothing ever seeds provider.json

codex-provider-switch:411-418 requires an existing provider.json showing plus. The only writers are commit_switch (azure) and rollback (plus). No bootstrap, init subcommand or documentation creates the first record (the tests write it by hand, codex-provider-switch.test.py around the {"provider":"plus"} fixture). On a real fleet prepare fails with "requires authoritative provider state showing plus" for every agent until someone hand-writes JSON in the right place.

Fix: when provider.json is absent and the agent has no .codex-home pointer in use, treat it as plus and write the seed record inside prepare under the lock, or add an explicit init AGENT subcommand and document it in the README.

M3. Only plus to azure exists; the PR says both directions are supported

prepare hardcodes source_provider: plus / target_provider: azure, and validate_confirmation rejects any target but azure. After commit and finalise, prepare refuses (current provider is azure). The only way back is rollback, which needs a live non-finalised transaction. An agent that has been committed to azure and finalised cannot be returned to plus with this tool.

Fix: either generalise prepare AGENT --to plus|azure (target-specific home and canary), or correct the PR description and add a documented manual path.

M4. rollback after a failed terminate kills a healthy original Codex and discards its thread

launch marks the token used and stage launching before terminate_codex. If terminate_codex raises ("did not stop"), the original Plus Codex may still be running with its context. abort is only legal in prepared, so the only recovery is rollback, which (stage not prepared) calls terminate_codex and relaunches a FRESH Plus thread. A recoverable situation turns into loss of the running session.

Fix: launched_at is only set after a successful terminate, so use it as the discriminator. In rollback, if tx.get("launched_at") is None and the pane is still a quiescent Codex, restore state (write plus, mark rolled-back) without terminating or relaunching. Add a test for it.

M5. Quota observer never drains the child's stderr

codex-quota-observer.server.ts (StdioAppServerConnection): stdio: ["pipe","pipe","pipe"], nothing reads stderr. Once the OS pipe buffer (typically 64 KiB) fills, the app-server blocks on a stderr write and stops emitting rate-limit frames, so telemetry goes stale with no error. Fix: stdio: ["pipe","pipe","ignore"] (or drain and log it). Related: the child is not killed when the deck exits, and the module-level singleton is re-created on HMR, so dev restarts can leave orphan codex app-server processes; register process.once("exit", () => observer.stop()) and key the singleton on globalThis.

M6. Quota state machine cannot leave BLOCKED in production (unverified against live Codex)

codex-quota.ts reduceQuotaState, mergeSparseRateLimits: a sparse update with rateLimitReachedType: null is treated as "unavailable, not cleared" and merged to the previous non-null value, so a sparse update can never clear a block. clock, probe_started, probe_succeeded and probe_failed events are only dispatched from tests; the observer emits only full_read and sparse_update, and a full read happens once per connection. RECOVERY_CANDIDATE and PROBING are therefore unreachable except via reconnect, and a block latched at 100 percent stays displayed as blocked even after the window resets, with observedAt kept fresh by every sparse update so the stale-evidence warning never fires. I have not verified how the real Codex server encodes "cleared" in a sparse update, so the severity depends on that, but as written the UI can show "quota is blocke 8000 d" indefinitely.

Fix: wire a periodic clock event in the observer (setInterval tied to start/stop), re-issue account/rateLimits/read when the clock moves a block to RECOVERY_CANDIDATE, and add a test that drives the observer (not only the reducer) from blocked back to available.

Minor

  1. The PR description says the default launch path is unchanged. It is not: Codex launches now source the agent's .env.secrets into the pane shell, set AGENT_ROLE, wrap in env, and pass CODEX_HOME from provider state. Probably intended, but state it, since it changes which credentials the Codex process inherits for every Codex agent.
  2. boot-agents.sh launch_agent: model_flag="--model $(cat .model)" is unquoted and is typed into the pane shell by codex_launch_command; the Python side validates the model with a regex, the shell side does not. Apply the same regex (or printf %q) in launch_agent.
  3. codex_verify_resume_or_fallback now dispatches on $# -gt 4 to tell old 4-arg and new 6-arg calls apart. Fragile; prefer named positions or a single call shape.
  4. Canary is sent blind (send-keys -l then Enter) with the same swallowed-Enter exposure PR 43 fixes. The failure mode is safe (verify-canary never succeeds, rollback), but a swallowed Enter costs a full rollback.
  5. rollback writes the plus provider record and marks rolled-back without verifying the relaunched Plus Codex came up.
  6. No stage-level mutual exclusion: two concurrent launch calls for the same transaction both pass ensure_lock_owner (owner is the transaction id, not a process) before either writes launching, so both could terminate and relaunch. Add a launching compare-and-set via O_EXCL marker file.
  7. Lock has no staleness story. A crash leaves provider-switch.lock forever and boot and push stay paused; there is no force-release and the PR text only says it "can be inspected". Add provider-switch-lock break AGENT that requires rming a named transaction owner and logs to audit.
  8. PROVIDER_SWITCH_TOKEN, PROVIDER_SWITCH_NONCE, PROVIDER_SWITCH_NOW and PROVIDER_SWITCH_TMUX_BIN are test hooks honoured in the production binary; NOW can bypass token expiry and TMUX_BIN runs an arbitrary binary. Local-only, but gate them behind a single PROVIDER_SWITCH_TEST=1.
  9. The confirmation token is passed on argv (launch AGENT TX TOKEN), visible in ps for the call. Low value (single use, pane-bound, 5 minutes); read it from stdin or an env var instead.
  10. find_verified_rollout loads every candidate rollout fully into memory and p.stat() in the sort key can raise FileNotFoundError (uncaught, traceback instead of SwitchError). Stream the first lines and wrap the sort.
  11. Display reads the legacy mirror (provider-switch/current/<agent>.json) BEFORE the authoritative <agent>/provider-switch/provider.json; commit_switch writes authoritative first then the mirror non-atomically, so a crash between them shows the wrong provider. Read authoritative first. Also document that deck DATA_ROOT must equal the PROVIDER_SWITCH_STATE_ROOT used by the CLI, otherwise every agent reads as unknown.
  12. loadCodexQuotaDisplay starts the observer before the DATA_ROOT null check; with telemetry enabled and DATA_ROOT unset the default FileCodexQuotaStore calls resolveDataRoot(), which throws inside the shell loader. Wrap in try/catch or move after the check. Observer also reconnects every 5 s forever when codex is not on PATH; add backoff.
  13. The store persists credits.balance, individualLimit.limit/used and planType to DATA_ROOT/codex-quota.json. That is spend data on disk in plain JSON; the env comment calls it "redacted". Drop fields the display never reads.
  14. The shell loader now does two existsSync and a read per agent on every navigation; cache for a few seconds.

Nit

  • .env.example example path /home/you/.codex; use ~/.codex style or a $HOME placeholder.
  • codex-provider-switch.test.ts wraps the Python suite; a failing Python import would show as an opaque spawn failure.

Public-repo hygiene

  • Paid-plan wording in public code and copy: shell.tsx ("shared Plus account"), codex-quota-display.ts warningFor ("ChatGPT Plus quota is blocked." hardcoded regardless of planType), and the PR body ("Plus plan"). The PR claims no commercial or pricing content; this is third-party plan naming, not Inkie's, but it is plan wording and is wrong for any non-Plus account. Use neutral "Codex quota is blocked" and name providers default and azure if you want to avoid plan names.
  • No /home/<user> paths, secrets or internal agent names found in the diff (.the-vault already exists on main). Fixtures use alex.

Tests would fail without the change

Yes for the shell and Python suites (new functions and files). Gaps: no test of the happy path through terminate_codex and send_fresh, no concurrency test of the lock beyond the mkdir atomicity case, no test of rollback after terminate failure (M4), and no test driving the observer from blocked to available (M6).

Did not check

  • No live switch, no real tmux, no real Codex rollout files, no live quota endpoint.
  • Did not run the suites or typecheck.
  • Did not read the quota UI component (CodexQuotaStatus.tsx), AppShell.tsx changes or the Python and shell tests beyond their test names and fixtures.
  • Did not verify the Codex app-server wire contract (account/rateLimits/*) or that codex app-server --stdio is the correct invocation.

@simon-inkie
simon-inkie merged commit 433ebb2 into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0