8000 docs: README envelope on the streaming engine; :limit (#306) is v3.0.0 scope by adityamukho · Pull Request #728 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,8 +223,8 @@ See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

Current plan (see `ROADMAP.md` and the milestones):
- **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release.
- **v3.0.0** (format v8 and data integrity, tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **v3.next** (the release after v3.0.0) holds new features (query profiler #185, `:limit` #306, lag/lead #182, UniFFI additions).
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **v3.next** (the release after v3.0.0) holds new features (query profiler #185, lag/lead #182, UniFFI additions).
- Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0. All v2.x known issues are listed in pinned issue #421 (`known-issue` label); `data-integrity`/`corruption`/`durability` issues fixed in a v2.x patch stay open until that patch is published; issues fixed on the unreleased `v3` branch close on merge (closing keywords do not fire on `v3`, so close them with `gh issue close`) and stay listed in #421 (CONTRIBUTING.md).
- Support policy and tiers: PHILOSOPHY.md §7 and §10. v2.x gets data-integrity and security fixes for 12 months after v3.0.0. Tier 1 = Rust + Python + browser WASM (from v3.0.0); other bindings are Tier 2.

Expand Down
19 changes: 10 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,18 +247,19 @@ See [ROADMAP.md](ROADMAP.md) for planned work and current direction.

## Performance

Performance envelope on the `v3` branch (format v8), measured up to 10M facts on a laptop (Intel i7-1065G7, SSD, page cache warm; see [BENCHMARKS.md](docs/BENCHMARKS.md#performance-envelope-v300-394) for every shape, RSS and the method):
Performance envelope on the `v3` branch (format v8), measured up to 10M facts on an otherwise idle server, two rounds (see [BENCHMARKS.md](docs/BENCHMARKS.md#executor-removal-551) for every shape, RSS and the method):

| Workload | 100K facts | 1M facts | 10M facts |
|---|---:|---:|---:|
| Point lookup (bound entity + attribute, incl. `:as-of` / `:valid-at`), p50 / p99 | 13 µs / 16 µs | 12 µs / 48 µs | 12 µs / 46 µs |
| Entity with 1,000 versions, p50 / p99 | 0.26 / 0.30 ms | 0.26 / 0.29 ms | 0.27 / 0.30 ms |
| Open a file (RSS after open: 3 MiB) | 0.12 ms | 0.14 ms | 0.12 ms |
| Checkpoint after 1 new fact, p50 / p99 | 3.1 / 5.1 ms | 2.9 / 4.3 ms | 3.8 / 21 ms |
| Attribute scan (N/6 rows), p50 | 43 ms | 0.46 s | 4.8 s |
| Selective 2-clause join (1 row), p50 | 54 ms | 0.61 s | 6.5 s |

Point lookups and time travel on one entity stay flat at every tested size. Scans, aggregations and joins grow linearly with the facts of the attributes they touch, because a query materializes them before joining (#473). They are interactive up to about 100K facts. Rule queries read the whole database (#529): 2.7 s at 100K facts and 40 s at 1M. Graphs over 10M facts are not tested.
| Point lookup (bound entity + attribute, incl. `:as-of` / `:valid-at`), p50 / p99 | 18–21 µs / 25–39 µs | 18–20 µs / 22–37 µs | 18–20 µs / 22–27 µs |
| Entity with 1,000 versions, p50 / p99 | 0.29–0.30 / 0.31–0.35 ms | 0.29–0.30 / 0.30–0.36 ms | 0.29–0.30 / 0.32–0.37 ms |
| Open a file (RSS after open: 3 MiB) | 0.12–0.16 ms | 0.25–0.27 ms | 0.13–0.23 ms |
| Checkpoint after 1 new fact, p50 / p99 | 0.7–0.9 / 1.4–1.6 ms | 0.9–1.8 / 2.7–2.8 ms | 1.1–1.5 / 2.2–2.9 ms |
| Attribute scan (N/6 rows) drained in batches of 1,000, p50 | 23 ms | 0.25 s | 2.5 s |
| Selective 2-clause join (1 row), p50 | 38 µs | 37 µs | 37 µs |
| Rule query with a constant (magic sets, 5–7 rows), p50 | 1.1 ms | 1.5 ms | 1.9 ms |

Point lookups, time travel on one entity, and joins and rule queries that a constant makes selective stay flat at every tested size. Queries that read a whole attribute (scans, aggregates, joins with nothing selective) grow linearly with the facts they read, in bounded memory: the 1.67M-row scan at 10M facts holds one batch, 7 MiB. Windows and whole transitive closures hold their rows, and `OpenOptions::query_memory_limit` stops a query that passes it with `QRY-012`. Graphs over 10M facts are not tested.

File-backed databases limit string values to 4,068 bytes (`WAL-003`). In-memory databases have no limit.

Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,8 @@ Scope — format and storage integrity:

Scope — query API:

- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`). The binding cursor objects follow. Behind the cursor, the streaming engine (#473) runs every query and produces rows as they are pulled: patterns, expressions and pseudo-attributes (#544), joins and `:db/valid-at` (#545), `not` and `or` (#546), aggregates and windows (#547), rules (#548), and transaction reads (#550); the executor's query path is gone (#551; done on `v3`). `:limit` (#306) is the one #473 acceptance item left, in v3.next. Operators do not spill: memory follows operator state, with an opt-in per-query limit (`QRY-012`, #549).
- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`). The binding cursor objects follow. Behind the cursor, the streaming engine (#473) runs every query and produces rows as they are pulled: patterns, expressions and pseudo-attributes (#544), joins and `:db/valid-at` (#545), `not` and `or` (#546), aggregates and windows (#547), rules (#548), and transaction reads (#550); the executor's query path is gone (#551; done on `v3`). Operators do not spill: memory follows operator state, with an opt-in per-query limit (`QRY-012`, #549).
- Native `:limit N` / `:offset` for plain `:find` queries (#306), with the 10M-fact acceptance test moved from #473.
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log pins its generation: checkpoints run but do not reuse its pages (#543; `API-013` deprecated). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462.
- Read-only open: `OpenOptions::read_only` takes a shared lock, so several readers can hold a file at once, writes nothing to the file or its WAL (a WAL is applied in memory, a v7 file is read without migrating), and refuses writes with `API-014` (#429; Rust API done on `v3`). The bindings' `open_with_options` follows.
- Raw log writer: `LogWriter` builds a new database from `FactRecord`s with their `tx_count`, `tx_id`, valid-time windows and assertion flags kept, gaps allowed (a purged transaction stays a hole for `:as-of`), at `<path>.partial` renamed into place by `finish` (#431; Rust API done on `v3`). Completes extract (#429, #430), transform, load. The binding objects follow.
Expand All @@ -119,7 +120,6 @@ Additive features, kept out of v3.0.0 so the #371 fix is not held up by them:

- Rule persistence — design first (#241)
- Query profiler (#185)
- Native `:limit N` / `:offset` for plain `:find` queries (#306)
- Set membership predicates (#316)
- `lag`/`lead` window functions (#182)
- Sliding row frames — `:rows N preceding` (#183; builds on #182)
Expand Down
Loading
0