Repository navigation
[Feature] AI evidence, safety, and provider contract hardening #92
Description
Outcome
Turn the existing AI foundation into a production-suitable, evidence-bound contract before exposing AI output in the main product.
Current gaps
- Persisted insights contain free text and a broad time window but no evidence references.
- Model output is stored without structured validation.
- The Settings “Test with aggregates” action sends real health aggregates and persists an insight.
- The provider is described as Anthropic-compatible, but requests always use an OpenAI-style
/chat/completionsendpoint/body. Only Anthropic headers and response parsing exist. - There is no provider-strategy ADR, adversarial safety evaluation, data receipt, output retention, or insight deletion UI contract.
Scope
- Define versioned
MetabolicContextSnapshot,EvidenceRef, and typedObservationDraftcontracts. - Require every numeric/factual statement to reference deterministic evidence generated by the analytics layer.
- Add provider capability metadata: local/remote, structured output, availability reason, context size, locale, model/version, and resource limits.
- Separate a synthetic no-health-data connection test from explicit real generation.
- Show endpoint hostname and exact outgoing data categories before remote generation.
- Implement output/refusal validation and reject unsupported evidence IDs or inconsistent numbers.
- Rename cloud support to OpenAI-compatible or add a distinct Anthropic Messages serializer, endpoint, and contract tests.
- Record an AI provider/safety ADR.
- Define deletion, export, invalidation, prompt/model versioning, feedback, and retention behavior.
- Add medical/dosing/treatment/emergency/prompt-injection evaluation fixtures.
Provider strategy
- Deterministic local observations always work without AI.
- Apple on-device model when available.
- Curated, capability-qualified local GGUF/llama.cpp runtime.
- Explicit BYO cloud under Advanced.
- Never silently fall back from local to remote.
Parallelization
Ready now. This owns AI contracts, provider adapters, safety validation, ADR, and tests. It can run in parallel with #89, the design-system issue, and deterministic analytics, but should consume—not redefine—the evidence contract from the analytics lane.
Acceptance criteria
- Unsupported evidence or numeric claims are rejected before persistence/display.
- Sparse data never generates an observation.
- AI disabled means zero network activity.
- Synthetic connection testing sends no health data.
- Real remote generation shows recipient and data preview and requires explicit consent.
- Native Anthropic either works end-to-end with contract tests or is not claimed.
- All AI artifacts and keys have verified delete/export behavior.
- Prompt/model/OS versions are recorded for reproducibility.
- Representative unsafe prompts and journal prompt-injection fixtures fail closed.
- ADR, threat model, rollback plan, and R2 independent review are complete.
Out of scope
- Final Today insight card.
- Open-ended chat.
- Model download UI.
- Autonomous actions or alerts.
Repository-owner recreation of the legacy issue at #39. The original discussion remains available there.
Activity