Repository navigation
feat: dashboards, sessions explorer and evals view with a typed panel-query API #83
Description
Problem
zradar's data model is rich (sessions, agents, models, tools, tokens, cost, evaluation scores) but the product shows almost none of it:
- The UI has a basic traces table and a stub trace detail page. The analytics, metrics, logs and spans pages are placeholders, there is no charting library, and there is no sessions or evals page.
- The analytics endpoints are too narrow to drive useful charts: daily buckets only, equality-only filters, no percentiles or rates, a small dimension set.
- Sessions can only be grouped client-side; there is no list or detail endpoint.
- Evaluation scores are ingested and stored but cannot be read (no reader method, route or capability).
A developer cannot answer "where is my money going", "which tool is failing", "which sessions went badly" or "did quality improve after the prompt change" without writing code. The roadmap lists a Dashboard UI, cost dashboard and session explorer as not started.
Proposal
Ship three user-visible views on one foundation: a typed panel-query API and a chart/panel UI foundation.
1. Panel-query API (backend)
POST /api/v1/analytics/querywithmetric,group_by(max 3),filters(eq, ne, in),interval(minute | hour | day),time_range,limit, optionalcompare_to_previous.- Metrics: trace/span count, token fields,
total_cost_usd, avg cost per trace, avg/p50/p95/p99 duration, error count and error rate, tool-call count, avg time-to-first-token, cache-hit rate, eval pass rate. - Dimensions: existing set plus
tool_name,llm_provider,environment,user_id,prompt_name. GET /api/v1/analytics/dimension-valuesfor pickers and variables (scoped and retention-clamped).- Stat/KPI shape returning the current value plus delta vs the previous period.
- Responses include
evaluated_through(data-as-of) andtruncatedflags; point/group limits protect the server. All metric/dimension names are whitelisted; no user input is interpolated into SQL. No user-authored SQL here.
2. Sessions
GET /api/v1/sessions(paginated; filters: time range, agent, user, environment, error-only, min cost) returning per session: start, end, duration, trace count, span count, tokens, cost, error flag, user, agents.GET /api/v1/sessions/{id}: ordered traces/spans summary for a timeline and drill-down.- Defined behaviour when
session_idis empty (excluded from the list; explicit "no session" bucket in analytics). - UI: sortable/filterable list; detail timeline linking to the trace detail.
3. Evals
- Reader methods plus
GET /api/v1/scores(filter by name, source, data type, trace, session, time range). GET /api/v1/analytics/scores: avg, pass rate, count, distribution, grouped by name/agent/model/prompt version/time bucket.- A
ReadScorescapability (or documented reuse ofReadDashboards). - UI: score trend per name, distribution, recent low scores linking to the owning trace.
4. UI foundation and built-in dashboards
- Add a charting library; shared, URL-synced time-range and auto-refresh state; reusable stat, time-series, bar, pie/donut and table panels with loading/empty/error states and a minimum refresh interval.
- Data hooks over the existing mode-aware API facade so the same code works in standalone and embedded (plugin) builds.
- Finish the traces list and detail pages (all drill-downs land there).
- Built-in dashboards defined declaratively (same schema as phase-2 user dashboards): Cost & Tokens, Latency & Errors, Tools & MCP, Quality (Evals), Sessions overview.
Phase 2: user-editable dashboards
- Postgres
dashboardstable (id, workspace_id, title, description,dataJSONB,schema_version,hash, owner, timestamps, soft delete), CRUD under/api/v1/dashboards, hash-checked optimistic updates (409 with latest on conflict), clone, JSON import/export,WriteDashboardscapability, audit events. - Panel editor form (metric, group_by, filters, interval, chart type); variables (query-based via dimension-values, constant, custom); "add to dashboard" from sessions/evals; drill-down links; tabs and grid layout.
Non-goals
Raw-SQL panels (tracked separately), public/shared links, scheduled reports, per-dashboard RBAC, folders, annotations, eval datasets/experiments/LLM-judge execution, session replay.
Current state (grounded in code)
- UI (
ui/apps/zradar): Next.js static export, React 19, Tailwind, zustand, swr; pagestraces(table),traces/detail(stub),analytics/metrics/logs/spans(placeholders),settings; single data hookuse-traces.ts; mode-aware client facade (lib/api/client.ts); plugin entry (plugin/entry.tsx). - Backend: routes in
crates/services/api/src/telemetry/router.rs, all gated byCapability::ReadDashboardsand workspace-scoped. The generic/api/v1/analyticswhitelist is incrates/core/zradar-parquet/src/telemetry_reader.rs(metrics: trace_count, span_count, token counts, total_cost_usd, avg_duration_ms, error_count; dimensions: agent_name, llm_model, session_id, service_name, span_name, span_type, status_code, agent_type). - Spans already carry first-class columns for what is needed:
session_id,user_id,agent_name,llm_model,llm_provider, token and cost fields,tool_name,mcp_tool_name,time_to_first_token_ms,llm_cache_hit,evaluation_name,evaluation_passed,environment, prompt id/name/version. EvaluationScorehas its own Parquet schema and is written viainsert_scores/score_extractor.rs; no read path.- Saved-object pattern exists (Postgres repository traits for settings and audit log); no dashboard tables.
Acceptance criteria
- A new workspace sees useful built-in charts within 2 minutes of first ingest with zero configuration.
- Every metric x interval x filter type in the panel-query API is covered by fixtures with numerically verified results (percentiles, rates, buckets, in-list).
- Dashboard load p95 < 3 s (24 h range, 8 panels, warm); sessions list p95 < 2 s (7 d, 100k sessions).
- A low eval score reaches its owning trace in at most 2 clicks.
- One UI build serves standalone and embedded modes with no code forks.
Open questions
- Must user-editable dashboards work in standalone mode without Postgres (file-backed fallback)?
- Scores under
ReadDashboardsor a newReadScorescapability? - Typed JSON panel query (recommended) vs a restricted SQL dialect?
- Session definition when
session_idis empty: ignore, or derive from invocation/workflow id or a time gap? - How is
total_cost_usdderived (ingest-time vs SDK-supplied), and can parent/child spans double count in sums? - Which score sources must work at launch (SDK, span events, log-extracted, LLM judge)?
- Recharts, or a library with heatmap/gauge/sankey for later?
- Result cache and minimum refresh interval in phase 1, given auto-refreshing dashboards?
Dependencies
Related existing work: metrics/logs analytics parity for non-trace panels. The panel-query extensions and scores read path are also prerequisites for alerting. Attribute indexing improves session and attribute filtering at scale but is not blocking.
Activity