8000 feat: scheduled alerting on cost, latency, error rate and tool failures · Issue #82 · zvectorlabs/zradar · GitHub
[go: up one dir, main page]

Skip to content

feat: scheduled alerting on cost, latency, error rate and tool failures #82

Open

Description

@Rushit

Problem

zradar can answer "what is my agent's cost, latency, error rate and tool failure rate", but nothing tells a user when those cross a line. Agents fail quietly: a prompt change doubles token spend, a tool starts erroring at 20%, an eval score drops after a model swap. Today the user finds out from a bill, a customer, or by happening to open a page. The roadmap only promises "cost threshold and error rate alerts via webhook".

Proposal: scheduled, workspace-scoped alerting

A workspace defines alert rules that are evaluated on a schedule against the existing analytics layer and notify through webhook and Slack, with firing and recovery notifications, deduplication, silence and a visible evaluation history.

Capabilities (MVP)

Rules

  • Fields: name, description, enabled, severity label, frequency, rolling window, metric, optional filters (equality and in-list), optional group_by (max 3 dimensions), operator, threshold, for_duration (pending period), silence_duration, evaluation_lag, destinations.
  • Metrics: count, sum, avg, p95, max, error rate, plus cost/token/latency fields; eval scores once a score read path exists (see Dependencies).
  • Budget rules: calendar-period window (day/week/month-to-date) on cost or tokens with warning and critical percentages of a budget.
  • Limits: minimum frequency, max rules per workspace, max groups evaluated per run (excess groups truncated and flagged).

Evaluation and state

  • Per-rule and per-group state machine: ok -> pending -> firing -> ok, plus silenced, error (evaluation failed) and insufficient_data (below minimum count). Failures surface as error, never as silence.
  • Multi-replica safe scheduling: due rules claimed with FOR UPDATE SKIP LOCKED, a claim epoch so a stale worker cannot complete a claim it lost, and a reaper for stuck claims.
  • The window ends at now - evaluation_lag, because the freshest data may still be in the WAL/write buffer and not yet in Parquet. Responses expose evaluated_through.
  • Evaluation is accounted against the workspace query policy; throttling produces an error state with a reason.

Notifications

  • Destinations: webhook (URL, method, headers, optional HMAC signature) and Slack incoming webhook. Secrets are stored encrypted with a key supplied by configuration.
  • Firing and recovery notifications include rule, workspace, severity, state, value, threshold, group labels, window and evaluated-through time.
  • Retries with backoff, per-attempt records, and a visible failed/dead-letter state.
  • SSRF guard: destinations resolving to loopback, link-local or private ranges are rejected by default, with a configurable allow-list for self-hosters.

Trust and operability

  • POST /api/v1/alerts/{id}/evaluate dry run: returns the value(s), would-be state and window without notifying or persisting state.
  • POST /api/v1/alert-destinations/{id}/test: sends a clearly labelled test notification.
  • Evaluation and delivery history, audit-log entries for rule/destination changes, Prometheus metrics (evaluations, failures, duration, notification outcomes, claim contention), and a feature flag so the scheduler is off until enabled.
  • New capabilities ReadAlerts / WriteAlerts; REST under /api/v1/alerts and /api/v1/alert-destinations.

Later phases

Comparison rules (previous period / offset with min-count and baseline guardrails), email and PagerDuty, template variables, mute windows, routing quota-threshold events through the same dispatcher, UI editor.

Non-goals

Real-time (ingest-path) alerts, free-form SQL alert queries, anomaly detection, alert-on-alert composites, incidents/on-call, cross-workspace roll-ups.

Current state (grounded in code)

  • Background jobs are spawned with a cancel token in crates/runtime/zradar-runtime/src/builder.rs; a scheduler fits this pattern.
  • ThresholdSink / ThresholdEvent (crates/core/zradar-policy) and the threshold_dedupe table already notify on quota thresholds, with dedupe but no destinations.
  • ParquetTelemetryReader::query_analytics (crates/core/zradar-parquet/src/telemetry_reader.rs) supports a whitelist of metrics (trace/span count, token counts, total cost, avg duration, error count) and dimensions (agent, model, session, service, span name/type, status, agent type), with equality filters only.
  • Gaps: no rate/ratio or percentile metric; no in-list or not-equal filters; evaluation scores are written (insert_scores) but there is no read path; no system-health data source; no outbound HTTP client dependency, secret store or SSRF protection; Capability has no alert entries; only a single squashed Postgres migration exists, so a migration convention for new tables needs to be settled; Parquet freshness lags ingest.

Proposed approach

  1. Extend the analytics layer (shared with the dashboards work): selectable bucket interval, error rate, percentiles, in-list/not-equal filters, scores read path.
  2. A dedicated AlertQueryService over the typed analytics layer (not over HTTP DTOs).
  3. A new zradar-alerting crate (evaluator, scheduler, state machine, dispatcher) and an additive Postgres migration: alerts, alert_states, alert_state_transitions, alert_destinations, alert_destination_links, alert_history, alert_delivery_attempts. Rule definitions stored as versioned JSONB. All rows carry workspace_id.
  4. REST API, capabilities, audit, metrics, feature flag; then a minimal UI.

Acceptance criteria

  • Breach -> firing -> recovery works end to end with two replicas running, with zero duplicate notifications in a soak test and zero missed alerts across a replica restart.
  • Dry-run returns the same value a real evaluation would for the same window.
  • A broken rule (bad filter, query failure, throttled) is visible as error, never as ok.
  • Webhook destinations pointing at private ranges are rejected by default.
  • p95 time from breach to notification <= frequency + evaluation lag + 30 s.
  • Evaluating 500 rules adds < 5% to p95 query latency.

Open questions

  1. Single-workspace only for the MVP, with cross-workspace roll-ups left to downstream deployments?
  2. Encryption key source for destination secrets (config key, environment variable, KMS hook)?
  3. UI in the MVP, or API + config first?
  4. Default evaluation lag (proposal 60 s) and default minimum frequency / max rules per workspace?
  5. Recovery notification default on, or per rule?
  6. Route quota ThresholdSink events through the dispatcher or keep them separate?
  7. Slack as a first-class destination type or a templated webhook?
  8. gRPC admin proto in the MVP or HTTP only?
  9. Migration convention for new tables given the single squashed migration.

Dependencies

Analytics layer extensions and the scores read path (also required by the dashboards and evals work); outbound HTTP client dependency review; secret-encryption key decision.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: corezradar-models, zradar-traitsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0