8000 fix(cbio): read the raw payload word, not the empty packed field (G1) by shroominic · Pull Request #210 · shroominic/OpenGlucose · GitHub
[go: up one dir, main page]

Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ SHELL := /bin/sh
.PHONY: help bootstrap tooling-bootstrap tooling-check hooks format \
format-check lint typecheck test-unit test-integration test-e2e test \
cbio-gs1-evidence \
cbio-gs1-decode-comparison \
build build-android build-web build-ios build-macos test-ios-native \
test-macos-native \
verify-android-release-signing check
Expand Down Expand Up @@ -54,6 +55,9 @@ test-e2e: ## Report the explicitly deferred device end-to-end lane.
cbio-gs1-evidence: ## Run one GS1 session on a device and record a redacted artifact.
@./scripts/cbio-gs1-evidence.sh

cbio-gs1-decode-comparison: ## Replay one captured GS1 log as a side-by-side decode.
@./scripts/cbio-gs1-decode-comparison.sh

test: test-unit test-integration ## Run all locally configured automated tests.

build: build-android build-web ## Build the portable Android and web targets.
Expand Down
27 changes: 25 additions & 2 deletions docs/testing/cbio-gs1-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,11 @@ written:

| Key | Content |
|---|---|
| `schema` | `cbio.session-evidence/1` |
| `schema` | `cbio.session-evidence/2` |
| `identity` | harness path, harness revision, app package, app revision, platform string |
| `outcome` | `completed`, `aborted_no_target`, `aborted_missing_characteristics`, `aborted_authentication_failed`, `failed` |
| `writes` | command frames actually sent, per classified kind |
| `records` | glucose and raw record counts, first/last index, raw value range |
| `records` | glucose and raw record counts and first/last index, plus one `{count, minimum, maximum, nonZero}` block per decoded field (`rawPayload`, `processedGlucose`) |
| `notifications` | FF31 notifications observed |
| `startedAtUtc`, `endedAtUtc`, `durationMilliseconds` | bounded session timing |
| `errors` | counts per closed reason from a fixed vocabulary |
Expand All @@ -48,6 +48,29 @@ written:
The raw glucose field has no verified scale, so no artifact, log or document may
present it as a physical glucose value.

`records.rawPayload` and `records.processedGlucose` are kept as separate blocks
because they are separate fields on the wire: the `08` record's payload word at
offset 4 and the firmware's processed word at offset 6. A run whose payload
`nonZero` is 0 fails validation, and a payload count that disagrees with the raw
record count fails too. `nonZero` is what separates "this field is empty" from
"this decoder reads the wrong bytes" — an aggregate range cannot.

## Decode comparison

```sh
make cbio-gs1-decode-comparison
# equal to: CBIO_EVIDENCE_LOG=<log> dart run tool/replay_gs1_decode.dart
```

Replays one captured harness log through two decoders over the *same* window —
the frame parser the evidence path uses and `CbioHistoryArchive`, the decoder
behind the app's live readings — and writes
`gs1-decode-comparison-<utc>.json`. The artifact carries the byte arithmetic for
the first record, the payload and processed ranges, the hourly shape, and an
`appPathComparison` block with `compared`, `agreeing`, `missing`,
`disagreeing` and `agrees`. Gate G1 asks for exactly this: the same field, the
same indices, no unexplained disagreement between the two paths.

## Redaction

`cbioSessionEvidenceArtifactViolations` rejects an artifact that:
Expand Down
58 changes: 49 additions & 9 deletions docs/testing/cbio-gs1-glucose-live.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,40 +60,80 @@ zero. Two batch shapes matter:
| Path | Records | Layout | Content |
| --- | --- | --- | --- |
| `0A` | 1782, index 1..1782 | packed 10-bit field, `11 + 2n` bytes | **every field zero** |
| `08` | 9981, index 1..9981 | `temp, dump, current, extra`, `12 + 8n` bytes | one record per minute, values below |
| `08` | 9981, index 1..9981 | `temp, dump, payload, processed`, `12 + 8n` bytes | one record per minute, values below |

The two paths do not agree, and only `08` carries usable content on this sensor:

- `0A` ends at index 1782, `2026-09-12T01:38:00Z`, and is zero throughout.
- `08` continues to index 9981, `2026-09-17T18:17:00Z`, with no gaps.

So the processed field the vendor application displays is frozen and empty —
consistent with a sensor that is logging but not activated — while the raw
record path is live and current.
The `08` record is eight bytes and holds **two** candidate reading words. The
one the app renders is `payload`, the little-endian word at record offset 4.
The firmware's `processed` word at offset 6 — the same 10-bit packed field the
`0A` batch carries — is zero in **every record of every captured session**, and
that is the field the evidence path used to read. The "every raw glucose field
reads 0" verdict was a wrong-field bug in the evidence decoder, not a sensor
that reports nothing: the packed word is empty because the sensor is logging but
not activated, while the payload word is live and current.

### The arithmetic, on a real captured record

< 8000 /td> First `08` record of the 13:02 session (`plaintext 08 01 6a 0b 40 00 00 00`),
after the batch header:

| Offset | Bytes | LE16 | Field | Meaning |
| --- | --- | --- | --- | --- |
| 0 | `3a 01` | 314 | `temp` | 31.4 °C on the same unverified `/10` scale — the control that the stride is right |
| 2 | `6a 0b` | 2922 | `dump` | no established meaning |
| 4 | `40 00` | **64** | `payload` | the reading-bearing word: `raw / 10` → 6.4 |
| 6 | `00 00` | 0 | `processed` | packed `(b >> 6) \| (b1 << 2)` → 0 |

A zero here is zero *bytes*, not a shifted offset or a wrong width: the
temperature word decodes to a plausible value at offset 0, so the record stride
and the offsets are confirmed independently of the glucose question.

### The same window through both decoders

`make cbio-gs1-decode-comparison CBIO_EVIDENCE_LOG=<captured log>` replays one
captured session through the frame parser and through `CbioHistoryArchive`, the
decoder behind the app's live readings, and writes
`evidence/gs1-decode-comparison-<utc>.json`. For the 1520-record window
(index 1..1520, 2026-09-10T19:57Z to 2026-09-11T21:16Z, 25.3 h):

| Field | Range | Non-zero records | Hourly median |
| --- | --- | --- | --- |
| `payload` (offset 4) | 51..95 | 1520 / 1520 | 5.3 at 05:00Z to 8.6 at 12:00Z — dips overnight, rises through the day |
| `processed` (offset 6) | 0..0 | 0 / 1520 | 0 |

The two decoders agree on all 1520 payload indices (`agreeing` 1520,
`missing` 0, `disagreeing` 0), which is what gate G1 asks for: one field, one
scale, no unexplained disagreement between the app path and the evidence path.
A physiologically-shaped series is still not a calibrated one — it is one more
reason the `/10` scale is plausible, and nothing more.

## Derived values, and why they stay unverified

`CbioRawGlucoseRecord` divides `current` by 10 and `temp` by 10 because two
`CbioRawGlucoseRecord` divides `rawPayload` by 10 and `rawTemperature` by 10 because two
independent implementations of this protocol read the fields that way, and
because the result is plausible:

| Field | Raw range | Derived | Distribution |
| --- | --- | --- | --- |
| `current` | 39..97 | 3.9..9.7 mmol/L | p05 5.2, p50 6.4, p95 8.0 mmol/L |
| `temp` | 274..460 | 27.4..46.0 °C | 9961 of 9981 records in 20.0..42.0 °C |
| `payload` | 39..97 | 3.9..9.7 mmol/L | p05 5.2, p50 6.4, p95 8.0 mmol/L |
| `rawTemperature` | 274..460 | 27.4..46.0 °C | 9961 of 9981 records in 20.0..42.0 °C |

Agreement between two independent clients, and a glucose range that looks like
interstitial glucose, is not proof of scale. `isUnitVerified` stays `false` and
the raw integer is exposed alongside any converted number; a reference
measurement against a known index is what would settle it.

The newest record at the time of the run was index 9981,
`2026-09-17T18:17:00Z`, raw `current` 50 and raw `temp` 315 — 5.0 mmol/L and
`2026-09-17T18:17:00Z`, raw `payload` 50 and raw `rawTemperature` 315 — 5.0 mmol/L and
31.5 °C on the unverified scale.

## Reading timeline

| Run | Raw read from | Raw index captured | `current` range | Writes | Notifications |
| Run | Raw read from | Raw index captured | `payload` range | Writes | Notifications |
| --- | --- | --- | --- | --- | --- |
| 1 | 0 | 1..1520 (20 s window) | 51..95 | 5 | 127 |
| 2 | 0 | 1..1520 (20 s window) | 51..95 | 5 | 127 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,13 @@ const Set<CbioWriteKind> _requiredWrites = <CbioWriteKind>{
/// stored record instead of re-reading the whole archive.
const int _rawStartIndex = int.fromEnvironment('CBIO_RAW_START_INDEX');

/// Records carried in the emitted comparison. The window itself is not
/// truncated; only the artifact's per-record table is.
const int _comparisonRecords = int.fromEnvironment(
'CBIO_COMPARISON_RECORDS',
defaultValue: 1520,
);

String _hex(List<int> bytes) =>
bytes.map((b) => b.toRadixString(16).padLeft(2, '0')).join(' ');

Expand Down Expand Up @@ -128,6 +135,30 @@ void main() {
isTrue,
reason: 'the link produced no glucose and no raw history record',
);
// Gate G1: the reading-bearing field must carry content, and the app's
// live path and the harness path must report the same field for the same
// records. A run that reports an empty field is not a passing run.
final comparison = run.comparison;
expect(comparison, isNotNull, reason: 'no side-by-side decode');
expect(
comparison!.processedCount,
evidence.rawIndices.length,
reason:
'every raw record must contribute one payload and one '
'processed sample',
);
expect(
comparison.payloadNonZero,
greaterThan(0),
reason: 'the payload field the app renders decoded to zero everywhere',
);
expect(
run.appPathAgrees,
isTrue,
reason:
'the app path and the harness path disagree on the payload '
'word',
);
},
timeout: const Timeout(Duration(minutes: 8)),
);
Expand All @@ -141,11 +172,14 @@ final class _SessionRun {
final Map<CbioWriteKind, int> writeKinds = <CbioWriteKind, int>{};
final Map<int, int> glucoseByIndex = <int, int>{};
final Map<int, int> rawByIndex = <int, int>{};
final Map<int, int> processedByIndex = <int, int>{};
final Map<String, int> errors = <String, int>{};
int notifications = 0;
bool targetAcquired = false;
bool gattReleased = false;
bool authenticationObserved = false;
bool appPathAgrees = false;
CbioDecodeComparison? comparison;
CbioSessionOutcome outcome = CbioSessionOutcome.failed;

void wrote(CbioWriteKind kind) =>
Expand All @@ -171,9 +205,10 @@ final class _SessionRun {
allowedWrites: _allowedWrites,
glucoseIndices: glucoseByIndex.keys.toList()..sort(),
rawIndices: rawByIndex.keys.toList()..sort(),
rawGlucoseValues: [
rawPayloadValues: rawByIndex.values.toList(),
processedGlucoseValues: [
...glucoseByIndex.values,
...rawByIndex.values,
...processedByIndex.values,
],
errors: errors,
);
Expand Down Expand Up @@ -416,13 +451,16 @@ Future<void> _runSession(_SessionRun run) async {
final rawBatch = parseCbioRawDataFrame(plaintext);
rawRecords.addAll(rawBatch.records);
for (final record in rawBatch.records) {
run.rawByIndex[record.packed.index] = record.packed.rawGlucose;
run.rawByIndex[record.processed.index] = record.rawPayload;
run.processedByIndex[record.processed.index] =
record.processed.rawGlucose;
}
_emit(
'CBIO-A 08-batch count=${rawBatch.records.length} '
'first=${rawBatch.records.first.packed.index} '
'last=${rawBatch.records.last.packed.index} '
'raw=${rawBatch.records.map((r) => r.packed.rawGlucose).join(',')} '
'first=${rawBatch.records.first.processed.index} '
'last=${rawBatch.records.last.processed.index} '
'payload=${rawBatch.records.map((r) => r.rawPayload).join(',')} '
'processed=${rawBatch.records.map((r) => r.processed.rawGlucose).join(',')} '
'temp=${rawBatch.records.map((r) => r.rawTemperature).join(',')}',
);
} on CbioFrameException {
Expand All @@ -445,11 +483,35 @@ Future<void> _runSession(_SessionRun run) async {
}
if (rawRecords.isNotEmpty) {
_emit(
'CBIO-A raw first=${rawRecords.first.packed.index} '
'last=${rawRecords.last.packed.index} '
'raw=${rawRecords.map((r) => r.packed.rawGlucose).join(',')}',
'CBIO-A raw first=${rawRecords.first.processed.index} '
'last=${rawRecords.last.processed.index} '
'payload=${rawRecords.map((r) => r.rawPayload).join(',')}',
);
}
// Both decoders see the same bytes in the same session: the app's live path
// (the history archive, which is what the installed app renders) and the
// frame parser the evidence path uses. They must report the same field.
final comparison = compareCbioDecode(
plaintextFrames,
maxRecords: _comparisonRecords,
);
final appPath = CbioHistoryArchive();
plaintextFrames.forEach(appPath.ingest);
final agreement = compareCbioWithArchive(comparison, appPath.records);
run.appPathAgrees = agreement.agrees;
run.comparison = comparison;
_emit(
'CBIO-A compare appPath agrees=${agreement.agrees} '
'agreeing=${agreement.agreeing}/${agreement.compared} '
'missing=${agreement.missing} disagreeing=${agreement.disagreeing} '
'records=${comparison.recordCount} '
'payload=${comparison.payloadMinimum}..${comparison.payloadMaximum} '
'payloadNonZero=${comparison.payloadNonZero} '
'processed=${comparison.processedMinimum}..${comparison.processedMaximum} '
'processedNonZero=${comparison.processedNonZero} '
'index=${comparison.firstIndex}..${comparison.lastIndex}',
);
_emit('CBIO-COMPARISON ${jsonEncode(comparison.toJson())}');
run.outcome = CbioSessionOutcome.completed;
} on TestFailure {
// An in-flight assertion (an out-of-envelope write) is not a session
Expand Down
5 changes: 4 additions & 1 deletion openhealth/integration_test/cbio_glucose_query_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,10 @@ final class _QueryRun {
allowedWrites: _allowedWrites,
glucoseIndices: glucoseByIndex.keys.toList()..sort(),
rawIndices: const <int>[],
rawGlucoseValues: glucoseByIndex.values.toList(),
rawPayloadValues: const <int>[],
// This harness only reads the packed `0A` layout, which is the firmware's
// processed field. It carries no payload word at all.
processedGlucoseValues: glucoseByIndex.values.toList(),
errors: errors,
);
}
Expand Down
19 changes: 14 additions & 5 deletions packages/cgm_cbio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,14 @@ Separate offline entry points inspect `08` raw-data batches,
`F0/04` storage replies, and `F0/03` time replies:
`parseCbioRawDataFrame`, `parseCbioStorageFrame`, and `parseCbioTimeFrame`.
These require complete plaintext data frames and reject control ACKs. Raw
records retain temperature/current/dump integers plus packed fields; none is
converted to a physical unit. Time fields have no assigned epoch, and storage
status is not interpreted. The existing `CbioFrame` hierarchy and generic
parser acceptance remain unchanged. These entry points do not select firmware,
send queries, or authorize a live read.
records retain `rawTemperature`, `rawDump`, the reading-bearing `rawPayload`
word, and the firmware's `processed` word; `parseCbioRawDataFrame` is the single
owner of that eight-byte layout, so no other path duplicates the offsets. Only
`rawPayload` is converted, at `raw / 10`, and nothing is presented as a
physical unit (`isUnitVerified` is always false). Time fields have no assigned
epoch, and storage status is not interpreted. The existing `CbioFrame`
hierarchy and generic parser acceptance remain unchanged. These entry points do
not select firmware, send queries, or authorize a live read.

`parseCbioActivationFrame` separately reads the five-byte `F0/02` state reply
and preserves its raw byte without an active/inactive enum.
Expand All @@ -62,6 +65,12 @@ streams a contiguous raw archive up to the present; the `0A` packed field is
zero throughout, so only `08` carries usable content. See the
[live record](../../docs/testing/cbio-gs1-glucose-live.md).

`cbio_decode_comparison.dart` decodes one captured window through both the app's
live path and the evidence path and reports per-record agreement (`compared`,
`agreeing`, `missing`, `disagreeing`). It exists because a field-identity bug is
invisible in an aggregate range and obvious in the bytes: the evidence path once
read the empty `processed` word instead of the reading-bearing `rawPayload` word.

Run package checks from this directory with the pinned Dart SDK:

```sh
Expand Down
2 changes: 2 additions & 0 deletions packages/cgm_cbio/lib/cgm_cbio.dart
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ export 'src/cbio_frames.dart';
export 'src/cbio_history_archive.dart';
export 'src/cbio_vendor_frames.dart';
export 'src/cbio_crypto.dart';
export 'src/cbio_decode_comparison.dart';
export 'src/cbio_glucose_sync.dart';
export 'src/cbio_log_replay.dart';
export 'src/cbio_queries.dart';
export 'src/cbio_session_evidence.dart';
Loading
Loading
0