8000 merge: main into v3 (v2.0.1, #323, #381, #315, #389, #297) by adityamukho · Pull Request #412 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ patched unless a separate security advisory explicitly says otherwise.

| Version | Supported |
| ------- | --------- |
| latest 1.x minor line | ✅ |
| older 1.x minor lines | ❌ |
| 2.x (latest release) | ✅ |
| 1.x | ❌ |
| 0.x | ❌ |

## Reporting a Vulnerability
Expand Down
117 changes: 94 additions & 23 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ on:
permissions:
contents: read
issues: write
actions: read # read the previous night's job logs to confirm an alert repeats

jobs:
compile:
Expand Down Expand Up @@ -212,6 +213,7 @@ jobs:
id: bencher
continue-on-error: true
run: |
set -o pipefail
bencher run \
--project minigraf \
--token "${{ secrets.BENCHER_API_TOKEN }}" \
Expand All @@ -222,42 +224,111 @@ jobs:
--threshold-upper-boundary ${{ matrix.threshold }} \
--err \
--adapter rust_criterion \
--file ${{ matrix.file }}

- name: Open regression issue
--file ${{ matrix.file }} \
2>&1 | tee bencher_out.txt

# Shared CI runners are noisy, so one alert on its own is usually a slow
# runner, not a regression. Report only benchmarks that also alerted in
# this suite on the previous run. If a regression issue is already open,
# add a comment to it instead of skipping, so later alerts stay visible.
- name: Report regression
if: steps.bencher.outcome == 'failure'
uses: actions/github-script@v6
env:
SUITE: ${{ matrix.name }}
with:
script: |
const today = new Date().toISOString().slice(0, 10);
const title = `Benchmark regression - ${today}`;
const fs = require('fs');
const { owner, repo } = context.repo;
const suite = process.env.SUITE;
const jobName = `bench / ${suite}`;

// Bencher lists each alert as "- <benchmark> (Latency): <.../alerts/<id>>".
// The "View results" lines use /perf? URLs, so they do not match.
const alertRe = /- (\S+) \(Latency\): https:\/\/bencher\.dev\/console\/projects\/[^/\s]+\/alerts\//g;
const parseAlerts = text => new Set([...text.matchAll(alertRe)].map(m => m[1]));

const current = parseAlerts(fs.readFileSync('bencher_out.txt', 'utf8'));
const runUrl = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`;

let body;
if (current.size === 0) {
// Bencher failed without listing alerts (API or upload error).
// Report it at once so a broken setup does not go unnoticed.
body = [
`Bencher failed in the nightly benchmark run (${suite}) but listed no alerts. This is likely an API or upload error.`,
'',
`**Run:** ${runUrl}`,
].join('\n');
} else {
// Find this suite's job in the previous completed run and read its alerts.
let previous = new Set();
let previousUrl = null;
try {
const { data: runs } = await github.rest.actions.listWorkflowRuns({
owner, repo,
workflow_id: 'bench.yml',
branch: 'main',
status: 'completed',
per_page: 10,
});
const prevRun = runs.workflow_runs.find(r => r.id !== context.runId);
if (prevRun) {
previousUrl = prevRun.html_url;
const { data: jobs } = await github.rest.actions.listJobsForWorkflowRun({
owner, repo, run_id: prevRun.id, per_page: 100,
});
const prevJob = jobs.jobs.find(j => j.name === jobName);
if (prevJob) {
const { data: log } = await github.rest.actions.downloadJobLogsForWorkflowRun({
owner, repo, job_id: prevJob.id,
});
previous = parseAlerts(typeof log === 'string' ? log : Buffer.from(log).toString('utf8'));
}
}
} catch (err) {
core.warning(`Could not read the previous run's alerts: ${err.message}`);
}

const repeated = [...current].filter(b => previous.has(b)).sort();
console.log(`Alerts this run: ${[...current].join(', ')}`);
console.log(`Alerts in previous run: ${[...previous].join(', ') || 'none'}`);
if (repeated.length === 0) {
core.notice(`No benchmark in ${suite} alerted on two runs in a row. Not reporting.`);
return;
}

body = [
`Bencher flagged these benchmarks in the nightly run (${suite}) on two runs in a row:`,
'',
...repeated.map(b => `- \`${b}\``),
'',
`**Run:** ${runUrl}`,
`**Previous run:** ${previousUrl}`,
'',
'Please review the Bencher dashboard to confirm the regression.',
].join('\n');
}

const { data: openIssues } = await github.rest.issues.listForRepo({
owner: context.repo.owner,
repo: context.repo.repo,
owner, repo,
labels: 'performance',
state: 'open',
per_page: 100,
});
const existing = openIssues.filter(i =>
i.title.startsWith('Benchmark regression')
);
if (existing.length > 0) {
console.log(`Skipping: open regression issue already exists (#${existing[0].number})`);
const existing = openIssues.find(i => i.title.startsWith('Benchmark regression'));
if (existing) {
await github.rest.issues.createComment({
owner, repo, issue_number: existing.number, body,
});
console.log(`Commented on open regression issue #${existing.number}`);
return;
}

const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const today = new Date().toISOString().slice(0, 10);
await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: title,
body: [
`Bencher detected a performance regression in the nightly benchmark run (${{ matrix.name }}).`,
'',
`**Run:** ${runUrl}`,
'',
'Please review the Bencher dashboard for details on which benchmarks regressed.',
].join('\n'),
owner, repo,
title: `Benchmark regression - ${today}`,
body,
labels: ['performance'],
});
42 changes: 42 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Two values of the same attribute written in one transaction are both returned by every query path. Previously only one came back, and entity-bound and attribute-bound queries could return different ones (#371, #287).
- A retract that removes several values of one attribute in one call now hides all of them (#371).

## Unreleased on v2.x (main) — v2.0.2

These changes ship first in v2.0.2 from `main` and are merged into the `v3` branch.

### Performance

- **Bound-entity point queries no longer pay for other attributes' history (#323).** `[:e :attr ?v]` now range-scans only `(e, :attr)` in the EAVT index instead of every record the entity has ever written. Reading a rarely changed attribute of a heavily rewritten entity no longer slows down as that entity's history grows: with 2,000 retract/reassert cycles on another attribute of the same entity, it drops from 4.58 ms to 19.0 µs. Reading the heavily rewritten attribute itself, and attribute scans (`[?e :attr ?v]`), are about 1.35–1.4× faster (4.64 ms → 3.36 ms), from cheaper net-assert grouping and removing a redundant dedup pass. The file format is unchanged.
- **Attribute scans are bounded to exactly the queried attribute (#381).** `[?e :attr ?v]` computed the end of its AEVT range by incrementing the attribute's last byte. For attributes whose last byte is `0x7F` or `0xBF` — including about 1 in 64 non-ASCII characters, such as `:丿` or `:ÿ` — the result was not valid UTF-8, so the scan ran to the end of the whole AEVT index and read every later fact before filtering. It also read facts for prefix siblings (`:ab`, `:a/b` when scanning `:a`). The range now ends at `attribute + "\0"`, which covers exactly one attribute. Scanning 100 `:丿` facts in a checkpointed file with 40,000 facts on neighbouring attributes drops from 45.4 ms to 251 µs. Results are unchanged.

- **Checkpoints no longer re-encode the whole index (#315).** `checkpoint()` decoded and re-serialised every entry of all four covering indexes and re-read every page of the file for its checksum, so checkpointing after one new fact cost almost as much as after thousands. Index leaves that receive no new entries are now copied verbatim, only the leaves that do are decoded and re-split, and unchanged fact pages are no longer re-hashed. Leaves are still bounds-checked and the leaf chain is checked for cycles, so a damaged index fails the checkpoint instead of being copied forward. Checkpoint after one new fact drops from 246 ms to 26.3 ms on a 100k-fact file and from 23.9 ms to 4.0 ms at 10k. Cost still grows with graph size, because the index pages are copied on every checkpoint; the file format is unchanged. The `checkpoint()` and `wal_checkpoint_threshold` docs now state that a checkpoint is not a durability boundary: the WAL is already crash-durable.

### Fixed

- **A power loss could lose a new database or WAL file, or bring back a deleted WAL (#389).** On Linux and other POSIX systems, a created or deleted file survives a power loss only once its parent directory is fsynced; Minigraf fsynced file contents but never a directory. Creating the `.graph` file or the `<db>.wal` sidecar, and deleting the WAL after a checkpoint, now fsync the parent directory after the file operation. A resurrected WAL was already harmless, since replay skips entries at or below the last checkpointed transaction, but a lost `.graph` or WAL lost committed data. Windows needs no directory sync and is unchanged. Process kills were never affected, because they leave the OS page cache intact.
- **Recursive rules failed with a literal start when the recursion goes through another rule (#297).** `(query [:find ?y :where (chain :a ?y)])` over `(chain ?x ?y) :- (link ?x ?mid) (chain ?mid ?y)` failed with `Unbound variable in rule head: ?mid` (INT-022). The magic-sets rewrite, which runs when a rule argument is given as a literal, dropped earlier rule calls from the rules that pass bindings to the next call, so a variable bound only by such a call (here `?mid`, bound by `link`) was never bound. #300 fixed the case where a fact pattern binds it. Those calls are now kept, and results match the same query without a literal start.
- **Bound-entity queries could drop rows when one `WriteTransaction` wrote the same attribute in several valid-time windows (#323).** Both facts share a `tx_count`, and the selective lookup path de-duplicated on `(entity, attribute, tx_count, asserted)`, so one window's value was silently lost, while the same query via a full scan returned both. That de-duplication has been removed, and bound-entity queries now always match a full scan.

### Documentation

- **Unsafe WAL recovery advice removed from `docs/ERROR_REFERENCE.md`.** The WAL-001, WAL-002 and STG-011 resolutions said a `.wal` file could be deleted with no data loss, or that a database could be rebuilt from the WAL alone. Both are wrong: transactions committed since the last checkpoint exist only in the WAL, and the WAL holds nothing older than that checkpoint. The resolutions now say to copy both files first, to checkpoint with the version that wrote the WAL, and what is lost if the WAL is deleted.
- CLAUDE.md now matches the milestones (v2.0.2 final v2.x release, v3.0.0 format v8 and data integrity, v3.1.0 features) and documents `error.rs`, `magic_sets.rs`, `fault_inject.rs` and `src/browser/`. ROADMAP lists #405 and #407 in the v2.0.2 scope.
- README: MSRV (1.89) stated, v2.x known issue (#371) shown near the top, Maven coordinates corrected to `io.github.project-minigraf`, Android listed on Maven Central, and binding download locations point to the binding repos.
- `.github/SECURITY.md` lists 2.x as the supported line. About 50 broken wiki links in `docs/ERROR_REFERENCE.md` now use full wiki URLs.

### Notes

- On v2.x, reading a heavily rewritten attribute still costs time proportional to its history, because v7 index keys carry neither the value nor the assert/retract flag; the structural fix needs the v8 keys and is tracked for v3.0.0 in #379.
- Checkpoints on v2.x still copy every index page, so their cost grows with graph size. Copy-on-write index pages on the v3 branch, which also make `save()` crash-atomic, are needed for checkpoints proportional to the change alone (#374).

## v2.0.1 — 2026-09-25

Patch release on the v2.x line. File format is unchanged (v7); no API changes.

### Fixed

- **Entity-bound lookups could silently return nothing after an index rebuild on open (#370).** When the index checksum does not match on open (for example after a process was killed during `save()`), Minigraf rebuilds all four indexes from the fact pages. The rebuild computed each fact's location by re-packing all facts contiguously, but a file written by several checkpoints has partially filled pages that a contiguous re-pack does not reproduce. From the second checkpoint onward, rebuilt index entries pointed at the wrong page/slot: entity-bound queries (`[:some/ident ?a ?v]`) returned `[]` or errored while attribute-driven queries still worked, and `checkpoint()` copied the bad entries forward. The rebuild now reads each fact's real on-disk location (#372).

### Known issues

- Files whose indexes were already rebuilt with wrong locations by v2.0.0 are not repaired by this release. A public integrity check and rebuild-indexes-from-fact-pages operation is tracked in #373; a crash-atomic `save()` in #374.
- Two values of the same attribute for one entity written in a single `transact` (or retracted in a single `retract`) can read back as one value, differently per query path (#371, #287). The fix changes the index key layout (file format v8) and ships in v3.0.0. Workaround on v2.x: write or retract each value of a multi-valued attribute in its own call.

## v2.0.0 — 2026-08-26

This was originally slated as v1.3.0, matching the GitHub milestone name under
Expand Down
23 changes: 16 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,15 +101,17 @@ cargo run < demos/demo_negation.txt

1. **`src/graph/`** — EAV fact store with bi-temporal support
- `types.rs`: `Fact`, `Value`, `EntityId`, `TxId`, `VALID_TIME_FOREVER`
- `storage.rs`: `FactStorage` — in-memory store, `transact_batch`, `retract`, `get_facts_as_of`, `get_facts_valid_at`, `net_asserted_facts`
- `storage.rs`: `FactStorage` — in-memory store, `transact_batch`, `retract`, `get_facts_as_of`, `get_facts_valid_at`, `net_asserted_facts`, `get_facts_by_entity_attribute_indexed`

2. **`src/storage/`** — Persistence layer
- `mod.rs`: `StorageBackend` trait, `FileHeader` v8 (84 bytes), `CommittedFactReader` / `CommittedIndexReader` traits
- `backend/file.rs`: Single `.graph` file backend (4KB pages, cross-platform)
- `backend/memory.rs`: In-memory backend for testing
- `backend/fault_inject.rs`: `FaultInjectingBackend` — injects I/O errors for durability tests (test builds only)
- `index.rs`: EAVT / AEVT / AVET / VAET key types, `FactRef`, `encode_value`
- `btree_v6.rs`: On-disk B+tree (`build_btree`, `OnDiskIndexReader`, `MutexStorageBackend`)
- `btree_v6.rs`: On-disk B+tree (`build_btree`, `rebuild_btree_incremental`, `OnDiskIndexReader`, `MutexStorageBackend`)
- `cache.rs`: LRU page cache (`PageCache`, default 256 pages)
- `dir_sync.rs`: `sync_parent_dir` — fsyncs the parent directory after creating the `.graph`/WAL or deleting the WAL (no-op off Unix)
- `packed_pages.rs`: Packed fact pages (~25 facts/4KB page), `MAX_FACT_BYTES`
- `persistent_facts.rs`: `PersistentFactStorage` — v8 save/load, auto-migration v7→v8

Expand All @@ -123,6 +125,7 @@ cargo run < demos/demo_negation.txt
- `types.rs`: `EdnValue`, `Pattern`, `DatalogQuery`, `AsOf`, `ValidAt`, `WhereClause` (incl. `Not`, `NotJoin`); `PseudoAttr` enum, `AttributeSpec` wrapper
- `optimizer.rs`: Selectivity-based join reordering; disabled under `wasm` feature
- `prepared.rs`: `BindValue`, `PreparedQuery` — parse-once/execute-many with named `$slot` bind slots; `prepare_query`, `substitute`
- `magic_sets.rs`: Magic-sets rewriting for demand-driven recursive rule evaluation (not applied to rules with `not`/`not-join`)

4. **`src/temporal.rs`** — UTC-only timestamp parsing (avoids chrono CVE GHSA-wcg3-cvx6-7396)

Expand All @@ -132,6 +135,10 @@ cargo run < demos/demo_negation.txt

7. **`src/wal.rs`** — Fact-level sidecar WAL, CRC32-protected entries, crash recovery

8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`

9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking), `indexeddb.rs` (IndexedDB persistence)

### Data Model

```rust
Expand Down Expand Up @@ -166,18 +173,20 @@ Auto-migrates v7 → v8 on open (index rebuild). v1–v6 are rejected (STG-028).

## Test Coverage

**1157 tests passing** (1149 passing, 8 ignored; unit + integration + doc).
**1206 tests passing** (1198 passing, 8 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.

## Current Maintainer Context

**v2.0.0 released** — it introduced kernel file locking (#317, #304), structured runtime error codes (#277), `OpenOptions` `#[non_exhaustive]`, and an MSRV of Rust 1.89. See `CHANGELOG.md` for the full rationale and release history.
**v2.0.1 released** (2026-09-25) — patch fixing index rebuild on open (#370). **v2.0.0** introduced kernel file locking (#317, #304), structured runtime error codes (#277), `OpenOptions` `#[non_exhaustive]`, and an MSRV of Rust 1.89. See `CHANGELOG.md` for the full rationale and release history.

Relevant areas for the planned query-profiler work (see `ROADMAP.md`):
- `src/query/datalog/` — query executor where profiling hooks will go (#185)
- `docs/BENCHMARKS.md` — post-1.0 performance baseline updates
Current plan (see `ROADMAP.md` and the milestones):
- **v2.0.2** (final planned v2.x release, format v7, no API change) ships from `main`.
- **v3.0.0** (format v8 and data integrity, tracker #383) is developed on the long-lived `v3` branch. Merge `main` into `v3` regularly.
- **v3.1.0** holds new features (query profiler #185, `:limit` #306/#310, lag/lead #182, UniFFI additions).
- Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0.

## Testing Conventions

Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ resolver = "2"

[package]
name = "minigraf"
version = "2.0.0"
version = "2.0.1"
edition = "2024"
rust-version = "1.89"
description = "Zero-config, single-file, embedded graph database with bi-temporal Datalog queries"
Expand Down
Loading
0