Repository navigation
Description
Problem
zradar can answer "what is my agent's cost, latency, error rate and tool failure rate", but nothing tells a user when those cross a line. Agents fail quietly: a prompt change doubles token spend, a tool starts erroring at 20%, an eval score drops after a model swap. Today the user finds out from a bill, a customer, or by happening to open a page. The roadmap only promises "cost threshold and error rate alerts via webhook".
Proposal: scheduled, workspace-scoped alerting
A workspace defines alert rules that are evaluated on a schedule against the existing analytics layer and notify through webhook and Slack, with firing and recovery notifications, deduplication, silence and a visible evaluation history.
Capabilities (MVP)
Rules
- Fields: name, description, enabled, severity label,
frequency, rollingwindow, metric, optional filters (equality and in-list), optionalgroup_by(max 3 dimensions), operator, threshold,for_duration(pending period),silence_duration,evaluation_lag, destinations. - Metrics: count, sum, avg, p95, max, error rate, plus cost/token/latency fields; eval scores once a score read path exists (see Dependencies).
- Budget rules: calendar-period window (day/week/month-to-date) on cost or tokens with warning and critical percentages of a budget.
- Limits: minimum frequency, max rules per workspace, max groups evaluated per run (excess groups truncated and flagged).
Evaluation and state
- Per-rule and per-group state machine:
ok -> pending -> firing -> ok, plussilenced,error(evaluation failed) andinsufficient_data(below minimum count). Failures surface aserror, never as silence. - Multi-replica safe scheduling: due rules claimed with
FOR UPDATE SKIP LOCKED, a claim epoch so a stale worker cannot complete a claim it lost, and a reaper for stuck claims. - The window ends at
now - evaluation_lag, because the freshest data may still be in the WAL/write buffer and not yet in Parquet. Responses exposeevaluated_through. - Evaluation is accounted against the workspace query policy; throttling produces an
errorstate with a reason.
Notifications
- Destinations: webhook (URL, method, headers, optional HMAC signature) and Slack incoming webhook. Secrets are stored encrypted with a key supplied by configuration.
- Firing and recovery notifications include rule, workspace, severity, state, value, threshold, group labels, window and evaluated-through time.
- Retries with backoff, per-attempt records, and a visible failed/dead-letter state.
- SSRF guard: destinations resolving to loopback, link-local or private ranges are rejected by default, with a configurable allow-list for self-hosters.
Trust and operability
POST /api/v1/alerts/{id}/evaluatedry run: returns the value(s), would-be state and window without notifying or persisting state.POST /api/v1/alert-destinations/{id}/test: sends a clearly labelled test notification.- Evaluation and delivery history, audit-log entries for rule/destination changes, Prometheus metrics (evaluations, failures, duration, notification outcomes, claim contention), and a feature flag so the scheduler is off until enabled.
- New capabilities
ReadAlerts/WriteAlerts; REST under/api/v1/alertsand/api/v1/alert-destinations.
Later phases
Comparison rules (previous period / offset with min-count and baseline guardrails), email and PagerDuty, template variables, mute windows, routing quota-threshold events through the same dispatcher, UI editor.
Non-goals
Real-time (ingest-path) alerts, free-form SQL alert queries, anomaly detection, alert-on-alert composites, incidents/on-call, cross-workspace roll-ups.
Current state (grounded in code)
- Background jobs are spawned with a cancel token in
crates/runtime/zradar-runtime/src/builder.rs; a scheduler fits this pattern. ThresholdSink/ThresholdEvent(crates/core/zradar-policy) and thethreshold_dedupetable already notify on quota thresholds, with dedupe but no destinations.ParquetTelemetryReader::query_analytics(crates/core/zradar-parquet/src/telemetry_reader.rs) supports a whitelist of metrics (trace/span count, token counts, total cost, avg duration, error count) and dimensions (agent, model, session, service, span name/type, status, agent type), with equality filters only.- Gaps: no rate/ratio or percentile metric; no in-list or not-equal filters; evaluation scores are written (
insert_scores) but there is no read path; no system-health data source; no outbound HTTP client dependency, secret store or SSRF protection;Capabilityhas no alert entries; only a single squashed Postgres migration exists, so a migration convention for new tables needs to be settled; Parquet freshness lags ingest.
Proposed approach
- Extend the analytics layer (shared with the dashboards work): selectable bucket interval, error rate, percentiles, in-list/not-equal filters, scores read path.
- A dedicated
AlertQueryServiceover the typed analytics layer (not over HTTP DTOs). - A new
zradar-alertingcrate (evaluator, scheduler, state machine, dispatcher) and an additive Postgres migration:alerts,alert_states,alert_state_transitions,alert_destinations,alert_destination_links,alert_history,alert_delivery_attempts. Rule definitions stored as versioned JSONB. All rows carryworkspace_id. - REST API, capabilities, audit, metrics, feature flag; then a minimal UI.
Acceptance criteria
- Breach -> firing -> recovery works end to end with two replicas running, with zero duplicate notifications in a soak test and zero missed alerts across a replica restart.
- Dry-run returns the same value a real evaluation would for the same window.
- A broken rule (bad filter, query failure, throttled) is visible as
error, never asok. - Webhook destinations pointing at private ranges are rejected by default.
- p95 time from breach to notification <= frequency + evaluation lag + 30 s.
- Evaluating 500 rules adds < 5% to p95 query latency.
Open questions
- Single-workspace only for the MVP, with cross-workspace roll-ups left to downstream deployments?
- Encryption key source for destination secrets (config key, environment variable, KMS hook)?
- UI in the MVP, or API + config first?
- Default evaluation lag (proposal 60 s) and default minimum frequency / max rules per workspace?
- Recovery notification default on, or per rule?
- Route quota
ThresholdSinkevents through the dispatcher or keep them separate? - Slack as a first-class destination type or a templated webhook?
- gRPC admin proto in the MVP or HTTP only?
- Migration convention for new tables given the single squashed migration.
Dependencies
Analytics layer extensions and the scores read path (also required by the dashboards and evals work); outbound HTTP client dependency review; secret-encryption key decision.
Activity