8000 feat(data): preserve source-aware import identity by shroominic · Pull Request #68 · shroominic/OpenGlucose · GitHub
[go: up one dir, main page]

Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,18 @@ expectations are defined in [docs/compatibility.md](docs/compatibility.md).

## [Unreleased]

### Added

- Local health-context storage now preserves typed platform record identity and
provenance for deterministic repeated imports. It retains source deletions as
local-only tombstones; HealthKit and Health Connect readers remain separate
future work.
- The optional local Diary stores its manual meal, activity, and sleep entries
in an isolated, versioned SQLite protocol. It keeps the selected occurrence
time and can make one time-bounded, observational rise link without claiming
a cause. Existing v0.1.4 health-event data remains readable because diary
records are not written into the legacy health-events table.

## [0.1.4] - 2026-08-22

### Added
Expand Down
78 changes: 78 additions & 0 deletions docs/architecture/adr/0004-source-aware-import-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# ADR 0004: Preserve source-aware import identity locally

- Status: Accepted
- Date: 2026-08-24
- Owners: `@shroominic`

## Context

Health platforms can return the same activity, sleep, or heart-rate record in
more than one bounded read. They can also revise a record or report its
deletion after the initial import. The previous local repository appended every
sample, which made repeated imports duplicate data and discarded the source
details needed to explain or reconcile it.

The source record identifier, source app/package, device details, and revision
are restricted health metadata. They are necessary for correct local
reconciliation, but must not become logs, exports, analytics, or default UI.

## Decision

Add optional typed provenance to normalized activity, sleep, and heart-rate
samples. Provenance contains a platform-scoped stable external identity, source
application/package, source name/device, recording method, and source revision.
The normalized sample itself remains the authoritative observed interval.

The SQLite store uses a partial composite unique index per sample family:
`(platform, external ID)`. A provenance-bearing sample therefore replaces the
same source record. Manual and legacy samples have no import identity and keep
their append-only behavior. A batch containing the same imported identity more
than once is rejected before it can create order-dependent results.

Source deletions use a separate typed tombstone contract. Applying a tombstone
removes the matching visible sample and retains the platform identity and
revision locally, even if the platform did not return the original values or
interval. A later source record with that identity clears its matching
tombstone.

Schema version 2 adds nullable identity columns and the tombstone table without
rewriting or deleting schema-version-1 rows. The migration is performed by the
SQLite upgrade transaction. A prior schema-version-1 binary can still read the
JSON sample payloads after this upgrade, but it cannot preserve the new import
identity when it writes; do not continue imports on a downgraded binary. Roll
forward to a schema-version-2 build before importing again. The schema-two
migration probes existing columns and tables before adding them, because a
schema-one binary can lower SQLite's version marker while leaving the additive
schema-two shape in place. Rolling forward restores the marker and does not
repeat an `ALTER TABLE`; it does not restore provenance omitted by a
schema-one write. Schema-two binaries reject future unknown schema versions
instead of lowering their version marker.

## Alternatives considered

- Compare normalized values and timestamps for deduplication: rejected because
distinct records can legitimately share values and timestamps, while revised
records can change either.
- Append a deleted sample with fabricated values: rejected because a platform
deletion may contain only an identity and should not invent health data.
- Add native HealthKit or Health Connect import in this change: rejected. This
decision establishes the local contract first; permission UI, platform
readers, anchors, and source-overlap policy remain separate work.

## Consequences

- Repeated imports can be deterministic only when the platform provides a
stable external identity.
- Existing records remain readable and are not retroactively guessed into
identities.
- Importers must apply platform changes in source order and use the tombstone
API for deletions.
- Source identifiers remain local-only restricted metadata and are excluded
from default timelines and exports.

## Follow-up controls

- Persist one incremental anchor/cursor per platform and data type.
- Define and test source-priority and overlap rules before aggregate analytics.
- Add bounded native importers and permission UI with physical-device evidence.
- Add app-lifetime repository lifecycle and recovery ownership at composition.
110 changes: 110 additions & 0 deletions docs/architecture/adr/0005-bounded-apple-health-context-import.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# ADR 0005: Use bounded anchored Apple Health context import

- Status: Accepted
- Date: 2026-08-24
- Owners: `@shroominic`

## Context

OpenGlucose needs optional local context for sleep, workouts, and heart rate.
An initial full HealthKit read can be large, repeat records on every retry, and
miss source deletions. Apple does not disclose whether read access was granted;
an empty read and an unavailable read can look the same to an app.

The source-aware identity and tombstone contract from ADR 0004 is the required
storage dependency. HealthKit UUIDs, source app/device metadata, and anchors
are restricted health metadata. They must not enter logs, exports, analytics,
or default UI.

## Decision

Use an iOS-only native `HKAnchoredObjectQuery` channel for these exact types:

- sleep analysis;
- workouts; and
- heart rate.

The user must enable the separate context-import toggle and then press
**Import now**. Each request has a rolling 30-day range. Dart rejects ranges
over 31 days, and the native channel repeats that bound before querying.
No background observer or automatic import is started.

The native channel returns a versioned, fail-closed payload. It maps each
HealthKit UUID to a typed Apple Health import identity, retains local-only
source provenance, and returns deletion UUIDs as tombstones. There is one
opaque anchor per type in a separate versioned import-state file. The iOS
directory, file, and transaction artifacts are backup-excluded before use. A
binary that does not recognize a future import-state schema fails closed and
does not rewrite it. This keeps the import cursor separate from the strict
sensor/glucose restricted-state schema and permits a safe app downgrade.
Draft builds that wrote an import cursor into the strict v3 file discard that
cursor during upgrade and repeat a bounded source read; source-aware upserts
make that replay safe. They do not migrate opaque anchors across the two state
contracts.

The app persists samples/tombstones and then purges only Apple Health records
that have fallen before the current 30-day predicate before it persists that
type's next anchor. This explicit source/type-scoped expiry handles the fact
that an anchored query with a moving date predicate cannot later report every
deletion for an aged-out record. If the purge or local state write fails, that
type's anchor does not advance; a later user-triggered sync safely repeats the
bounded work.

The UI distinguishes an unavailable device, a locked device, a retryable
failure, a read-access request, a ready state, and **No accessible data**. It
must not claim that a read permission was granted. Missing
`HKMetadataKeyWasUserEntered` is retained as an unknown recording method, not
assumed to be automatic.

## Alternatives considered

- Use Flutter's generic Health package for all reads: rejected because this
scope requires iOS anchored changes and deletion records.
- Read every HealthKit record without a date predicate: rejected because the
user action would be unbounded.
- Treat an empty response as permission denial: rejected because HealthKit
intentionally does not reveal read authorization.
- Add Health Connect, historical glucose import, overlays/charts, or source
precedence here: rejected as separate product and safety decisions.

## Consequences

- Repeated source records replace local rows by stable identity rather than
duplicating them.
- Returned source deletions remove visible imported rows through tombstones.
- On each successful sync, the importer expires source/type records that fall
before that sync's rolling 30-day cutoff without fabricating a source
tombstone. Turning the toggle off does not itself erase retained context.
- A full native page can require another explicit sync; the UI states this
conservatively instead of starting a hidden long-running import.
- Turning the toggle off stops future app reads. It does not delete retained
local imported context; verified deletion remains separate work.

## Follow-up controls

- Establish source-overlap and display policy before rendering imported context
behind glucose data.
- Decide retention beyond the current 30-day bounded window and complete local
deletion for imported context.
- Add reviewed background-delivery policy only after explicit user controls and
physical-device evidence.
- Evaluate Health Connect separately; it must not reuse iOS authorization or
anchor assumptions.

## Required physical-device evidence before broader rollout

This partial implementation is not evidence-complete until a real iPhone with
Health data records each case below without copying values or identifiers into
test logs or issue comments:

- opt-in and off behavior, including the native authorization sheet;
- an empty or unreadable read shown only as **No accessible data**;
- HealthKit unavailable, a locked device, and a retryable native error mapped
to their safe public states;
- sleep, workout, and heart-rate import with a source revision/update and a
returned deletion;
- a record that ages out of the 30-day predicate removed before its next anchor
advances, including an interrupted/retried sync; and
- backup exclusion for the import-state directory, primary file, and staging
artifacts, plus an update/downgrade/roll-forward check that preserves an
unknown future state file.
2 changes: 2 additions & 0 deletions docs/architecture/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ enforcement detail is already implemented.
| [0001](0001-package-boundaries.md) | Accepted | Separate domain, BLE transport, vendor protocol, platform adapter, and app composition |
| [0002](0002-local-first-health-data.md) | Accepted | Keep health data local by default and make movement explicit |
| [0003](0003-platform-release-model.md) | Accepted | Build source-bound mobile artifacts and release only through fail-closed platform lanes |
| [0004](0004-source-aware-import-identity.md) | Accepted | Keep imported health-record identity and provenance local for deterministic reconciliation |
| [0005](0005-bounded-apple-health-context-import.md) | Accepted | Read only user-enabled Apple Health context through bounded, anchored iOS queries |

## Adding an ADR

Expand Down
31 changes: 31 additions & 0 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,37 @@ metadata rewrite resumes from either old or new names. If both names exist with
different contents, startup preserves both and fails closed for manual recovery
instead of guessing which glucose history is authoritative.

The local health SQLite repository schema two adds nullable source-platform and
external-record identity columns plus a local tombstone table. Schema-one rows
remain byte-for-byte JSON-compatible and are not retroactively assigned an
identity. The forward migration is additive and transaction-backed. A
schema-one binary can read the stored sample JSON after the upgrade, but cannot
preserve the identity columns when it writes new imports; do not continue
imports after a downgrade. Roll forward to a schema-two build before importing
again. A schema-one binary can lower SQLite's `user_version` while leaving the
additive schema-two columns in place. Schema two probes the table shape before
each additive migration, so a later schema-two launch restores the version
marker without duplicate-column failure. That recovery preserves existing rows;
it does not make schema-one writes source-aware.

The schema-two binary rejects an attempt to open a higher, unknown SQLite
schema version. It leaves that higher version marker unchanged rather than
silently relabelling an unrecognized schema as version two.

Schema three adds an isolated `fast_journal_entries` table for the optional
local diary. Its records use a separate strict fast-journal format one with an
explicit manual source, selected occurrence time, optional duration, and an
optional time-bounded observed-rise reference. The reference persists no
glucose value and is atomically claimed once by its exact rise-start instant.
It is not written into `health_events`: a v0.1.4 binary continues to decode
only its known health-event JSON and ignores the additive table. A v0.1.4
launch can lower SQLite's version marker while leaving the additive table in
place; a schema-three launch recreates its table/index declarations with
`IF NOT EXISTS` and preserves existing diary rows. Do not manually edit either
table. The current partial diary has no edit or delete flow, so its one-time
observational claim stays durable until a separately reviewed lifecycle policy
exists.

Downgrading to a schema-two app after filename migration is unsupported: the
older app cannot locate schema-three blobs even though their bytes remain on
disk. Roll forward to a schema-three build and preserve the original app
Expand Down
49 changes: 39 additions & 10 deletions docs/privacy/data-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ privacy owner before public distribution.

| Class | Examples | Required handling |
| ---------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Restricted health data | glucose values/times, calibrations, health events, derived metrics, AI inputs/outputs | Local by design; no automatic upload; native backup, logging, export, and deletion controls require the verification described below |
| Restricted health data | glucose values/times, calibrations, health events, derived metrics, imported source application/device identifiers, AI inputs/outputs | Local by design; no automatic upload; native backup, logging, export, and deletion controls require the verification described below |
| Sensitive device data | sensor serial/device ID, BLE advertisements, diagnostics, firmware | Minimize, keep on device, redact from filenames/logs/lock screen |
| Secret | API keys, signing keys, store credentials | Platform secret store or ephemeral release environment; never repository/preferences/logs |
| Operational | app version, non-sensitive error class, check result | May be retained only when documented and stripped of health/device identifiers |
Expand Down Expand Up @@ -41,6 +41,20 @@ relies on the application sandbox and operating-system at-rest protections;
their device-state behavior and threat-model sufficiency remain verification
work. Backup exclusion is not encryption.

The local health SQLite store can retain platform-owned import identities,
source application/package, source name/device, recording method, revision, and
deletion tombstones for deterministic reconciliation. These are restricted
health metadata. They are not included in the current archive export, default
timeline/UI, analytics, or production logs. The iOS Apple Health context reader
uses this contract only after explicit opt-in and a user-triggered import. It
reads sleep, workouts, and heart rate in a rolling 30-day bounded query with
one opaque anchor per type in a dedicated versioned import-state file. On iOS,
that directory and its transaction artifacts are verified backup-excluded; an
unknown future import-state version fails closed without being rewritten.
Health Connect, background delivery,
historical glucose import, source-overlap policy, and context display remain
unimplemented.

Display-only preferences may remain in platform preferences. In an actual web
build, `PreferencesHealthStateStore` delegates to `shared_preferences`, whose
web implementation uses origin-scoped browser `localStorage`; browser clearing,
Expand Down Expand Up @@ -77,15 +91,30 @@ clearing one selected sensor is not equivalent to complete erasure.

## Sharing and export

The iOS app implements an opt-in, write-only Apple Health integration. It sends
glucose values and their timestamps as blood-glucose samples only after the
user enables the integration and taps **Sync now**; simulated/demo data is
blocked. It does not read HealthKit data. Apple Health's privacy, retention,
backup, and deletion behavior applies once a sample is written. The local
contiguous export watermark and last-sync time use the restricted state store,
not backup-eligible preferences. Because a HealthKit write and the local cursor
cannot commit atomically, an interrupted sync can write a duplicate sample on
retry. The UI discloses this at-least-once behavior.
The iOS app implements two separate, opt-in Apple Health integrations.
Glucose export sends glucose values and their timestamps as blood-glucose
samples only after the user enables it and taps **Sync now**; simulated/demo
data is blocked. Its local contiguous export watermark and last-sync time use
the restricted state store, not backup-eligible preferences. Because a
HealthKit write and the local cursor cannot commit atomically, an interrupted
sync can write a duplicate sample on retry. The UI discloses this at-least-once
behavior.

Apple Health context import reads only sleep, workouts, and heart rate after
the user enables that separate control and taps **Import now**. Each sync is a
bounded rolling 30-day read. It retains source identity/provenance and opaque
per-type anchors in a separate versioned, backup-excluded iOS import-state
file for repeat reconciliation, and applies returned source deletions locally.
Before an anchor advances, it removes only matching Apple Health records that
have aged out of the rolling predicate; it does not fabricate source deletion
records. Apple does not disclose read permission to the app, so the
UI says **No accessible data** when a query returns no records; it does not
claim that access was granted or denied. It does not import blood glucose,
start background delivery, or show raw imported values in the default UI.
Disabling the toggle stops future reads but does not erase retained context;
the verified complete-delete flow remains unimplemented. Apple Health's
privacy, retention, backup, and deletion behavior applies after data enters
that platform or app storage.

Archived sensor details provide an explicit single-file export through the
platform share sheet. A confirmation preview displays the sensor-session date
Expand Down
Loading
0