8000 feat: dashboards, sessions explorer and evals view with a typed panel-query API · Issue #83 · zvectorlabs/zradar · GitHub
[go: up one dir, main page]

Skip to content

feat: dashboards, sessions explorer and evals view with a typed panel-query API #83

Open

Description

@Rushit

Problem

zradar's data model is rich (sessions, agents, models, tools, tokens, cost, evaluation scores) but the product shows almost none of it:

  • The UI has a basic traces table and a stub trace detail page. The analytics, metrics, logs and spans pages are placeholders, there is no charting library, and there is no sessions or evals page.
  • The analytics endpoints are too narrow to drive useful charts: daily buckets only, equality-only filters, no percentiles or rates, a small dimension set.
  • Sessions can only be grouped client-side; there is no list or detail endpoint.
  • Evaluation scores are ingested and stored but cannot be read (no reader method, route or capability).

A developer cannot answer "where is my money going", "which tool is failing", "which sessions went badly" or "did quality improve after the prompt change" without writing code. The roadmap lists a Dashboard UI, cost dashboard and session explorer as not started.

Proposal

Ship three user-visible views on one foundation: a typed panel-query API and a chart/panel UI foundation.

1. Panel-query API (backend)

  • POST /api/v1/analytics/query with metric, group_by (max 3), filters (eq, ne, in), interval (minute | hour | day), time_range, limit, optional compare_to_previous.
  • Metrics: trace/span count, token fields, total_cost_usd, avg cost per trace, avg/p50/p95/p99 duration, error count and error rate, tool-call count, avg time-to-first-token, cache-hit rate, eval pass rate.
  • Dimensions: existing set plus tool_name, llm_provider, environment, user_id, prompt_name.
  • GET /api/v1/analytics/dimension-values for pickers and variables (scoped and retention-clamped).
  • Stat/KPI shape returning the current value plus delta vs the previous period.
  • Responses include evaluated_through (data-as-of) and truncated flags; point/group limits protect the server. All metric/dimension names are whitelisted; no user input is interpolated into SQL. No user-authored SQL here.

2. Sessions

  • GET /api/v1/sessions (paginated; filters: time range, agent, user, environment, error-only, min cost) returning per session: start, end, duration, trace count, span count, tokens, cost, error flag, user, agents.
  • GET /api/v1/sessions/{id}: ordered traces/spans summary for a timeline and drill-down.
  • Defined behaviour when session_id is empty (excluded from the list; explicit "no session" bucket in analytics).
  • UI: sortable/filterable list; detail timeline linking to the trace detail.

3. Evals

  • Reader methods plus GET /api/v1/scores (filter by name, source, data type, trace, session, time range).
  • GET /api/v1/analytics/scores: avg, pass rate, count, distribution, grouped by name/agent/model/prompt version/time bucket.
  • A ReadScores capability (or documented reuse of ReadDashboards).
  • UI: score trend per name, distribution, recent low scores linking to the owning trace.

4. UI foundation and built-in dashboards

  • Add a charting library; shared, URL-synced time-range and auto-refresh state; reusable stat, time-series, bar, pie/donut and table panels with loading/empty/error states and a minimum refresh interval.
  • Data hooks over the existing mode-aware API facade so the same code works in standalone and embedded (plugin) builds.
  • Finish the traces list and detail pages (all drill-downs land there).
  • Built-in dashboards defined declaratively (same schema as phase-2 user dashboards): Cost & Tokens, Latency & Errors, Tools & MCP, Quality (Evals), Sessions overview.

Phase 2: user-editable dashboards

  • Postgres dashboards table (id, workspace_id, title, description, data JSONB, schema_version, hash, owner, timestamps, soft delete), CRUD under /api/v1/dashboards, hash-checked optimistic updates (409 with latest on conflict), clone, JSON import/export, WriteDashboards capability, audit events.
  • Panel editor form (metric, group_by, filters, interval, chart type); variables (query-based via dimension-values, constant, custom); "add to dashboard" from sessions/evals; drill-down links; tabs and grid layout.

Non-goals

Raw-SQL panels (tracked separately), public/shared links, scheduled reports, per-dashboard RBAC, folders, annotations, eval datasets/experiments/LLM-judge execution, session replay.

Current state (grounded in code)

  • UI (ui/apps/zradar): Next.js static export, React 19, Tailwind, zustand, swr; pages traces (table), traces/detail (stub), analytics/metrics/logs/spans (placeholders), settings; single data hook use-traces.ts; mode-aware client facade (lib/api/client.ts); plugin entry (plugin/entry.tsx).
  • Backend: routes in crates/services/api/src/telemetry/router.rs, all gated by Capability::ReadDashboards and workspace-scoped. The generic /api/v1/analytics whitelist is in crates/core/zradar-parquet/src/telemetry_reader.rs (metrics: trace_count, span_count, token counts, total_cost_usd, avg_duration_ms, error_count; dimensions: agent_name, llm_model, session_id, service_name, span_name, span_type, status_code, agent_type).
  • Spans already carry first-class columns for what is needed: session_id, user_id, agent_name, llm_model, llm_provider, token and cost fields, tool_name, mcp_tool_name, time_to_first_token_ms, llm_cache_hit, evaluation_name, evaluation_passed, environment, prompt id/name/version.
  • EvaluationScore has its own Parquet schema and is written via insert_scores / score_extractor.rs; no read path.
  • Saved-object pattern exists (Postgres repository traits for settings and audit log); no dashboard tables.

Acceptance criteria

  • A new workspace sees useful built-in charts within 2 minutes of first ingest with zero configuration.
  • Every metric x interval x filter type in the panel-query API is covered by fixtures with numerically verified results (percentiles, rates, buckets, in-list).
  • Dashboard load p95 < 3 s (24 h range, 8 panels, warm); sessions list p95 < 2 s (7 d, 100k sessions).
  • A low eval score reaches its owning trace in at most 2 clicks.
  • One UI build serves standalone and embedded modes with no code forks.

Open questions

  1. Must user-editable dashboards work in standalone mode without Postgres (file-backed fallback)?
  2. Scores under ReadDashboards or a new ReadScores capability?
  3. Typed JSON panel query (recommended) vs a restricted SQL dialect?
  4. Session definition when session_id is empty: ignore, or derive from invocation/workflow id or a time gap?
  5. How is total_cost_usd derived (ingest-time vs SDK-supplied), and can parent/child spans double count in sums?
  6. Which score sources must work at launch (SDK, span events, log-extracted, LLM judge)?
  7. Recharts, or a library with heatmap/gauge/sankey for later?
  8. Result cache and minimum refresh interval in phase 1, given auto-refreshing dashboards?

Dependencies

Related existing work: metrics/logs analytics parity for non-trace panels. The panel-query extensions and scores read path are also prerequisites for alerting. Attribute indexing improves session and attribute filtering at scale but is not blocking.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: query-apiAdmin HTTP query/analytics APIenhancementNew feature or request

    Type

    No type
    3991

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0