8000 docs(perf): verify committed net-assert is superseded (#532) by adityamukho · Pull Request #775 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ cargo run < demos/demo_negation.txt
- `btree.rs`: On-disk B+tree over byte keys (`build_btree`, `cow_update` (copy-on-write inserts and deletes; an emptied leaf or node leaves its parent, an emptied tree is root 0; #550) and `cow_insert`, `LeafCursor` (with `seek`), `prefix_scan`, `get`, `MutexStorageBackend`)
- `dict.rs`: `DictReader` (id ↔ UUID/ident, tx timestamps, long values; `uuids_of` walks dense id batches with `LeafCursor` and uses exact scalar reads for sparse/small batches, #530) and `Encoder` (checkpoint-time id assignment and key building)
- `value_pages.rs`: append-only value pages for strings over 64 bytes and sets whose element encoding is over 64 bytes (#316; DICT dedup compares bytes, `long_bytes`/`find_long_bytes`)
- `reader.rs`: `OnDiskReader` — covering reads of committed facts (`CommittedReader`, `KeySource`); `live_facts` and the resumable `live_keys` decide net-assert on index keys (#379) among the versions a `TxBound` keeps; `tx_order` (cached per reader), `last_tx_at`, `tx_runs` over the DICT's transactions (#551)
- `reader.rs`: `OnDiskReader` — covering reads of committed facts (`CommittedReader`, `KeySource`); query scans use the resumable `live_keys` to decide net-assert on index keys (#379) among the versions a `TxBound` keeps; `live_facts` is test-only, and production fact-level net-assert handles pending records (#532, superseded by #551); `tx_order` (cached per reader), `last_tx_at`, `tx_runs` over the DICT's transactions (#551)
- `verify.rs`: integrity walk for `Minigraf::verify` (tree order, index digests, DICT, page accounting) and the source choice for `rebuild_indexes` (#373)
- `meta.rs` / `page.rs` / `freelist.rs`: meta pages A/B, the common page header and allocator (skips quarantined ids), free-list chain
- `pins.rs`: `Pins` / `GenerationPin` — generations held by open snapshots and the quarantine of pages checkpoints freed after them (#543)
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Diff line change
Original file line number Diff line number
Expand Up @@ -120,7 +120,7 @@ Scope — release-gate testing and documentation:
- Persistent fuzz corpus and an operation-sequence target (#387; done on `v3`: corpus caching, weekly `cmin`, the v8 page targets, and the `ops_sequence` and `corrupt_file` targets)
- Fault injection across save/WAL/recovery (#390)
- Long-running soak test at scale (#392; done on `v3`: `tests/soak_test.rs`, 10M facts and a 5-hour churn weekly, alternating `main` and `v3`)
- Benchmarks at 100K–10M facts and a published performance envelope (#394; on `v3`: `examples/perf_envelope.rs`, results in `docs/BENCHMARKS.md`; follow-ups #528–#532)
- Benchmarks at 100K–10M facts and a published performance envelope (#394; on `v3`: `examples/perf_envelope.rs`, results in `docs/BENCHMARKS.md`; follow-ups #528–#532, with #532 superseded by #551's removal of repeated committed-fact net-assert; [audit and measurements](docs/benchmarks/532-net-assert-audit.md))
- Higher storage/WAL coverage gates and branch-coverage gates (#396; on `v3`: line and branch floors per file in `coverage-thresholds.toml`, checked on every PR)
- Durability, recovery and operations guide (#398; done on `v3`: `docs/DURABILITY.md`, also on the docs site; found #535)

Expand Down
2 changes: 1 addition & 1 deletion docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ On the executor, before #544 (the executor's query path is gone since #551). `pe
- **Up-front materialization (50–60%).** Before matching, the executor turns every live fact of each pattern's attribute into a `Fact` (`selective_fact_fetch` → `live_scan`). A pattern without a bound entity is read in full, so a selective join costs as much as scanning its widest attribute. The streaming engine (#473: `Scan` and `IndexNestedLoopJoin` operators) replaces this.
- **DICT lookup per row (26–35%).** `uuid_of` does a B+tree `get` for every row's entity id (#530).
- **Bindings as `HashMap<String, Value>` (about 25%).** SipHash insert, lookup and drop per row; #473's columnar batches replace them.
- **`net_asserted_facts` over committed facts that `live_facts` already resolved (about 10%)** (#532).
- **`net_asserted_facts` over committed facts that `live_facts` already resolved (about 10%)** (#532). Superseded by #551's removal of the executor's query path; see the [source audit and fresh envelope/profiles](benchmarks/532-net-assert-audit.md). Pending records still need fact-level net-assert.
- **Keyword constants hashed to a UUID per fact (about 8% of the old join)** (#531). The streaming engine resolves pattern constants while planning; see the [post-#551 audit and profiles](benchmarks/531-keyword-constants.md). Expression keyword/ref equality retains a separate per-row conversion described there.
- `attribute_value` reads the whole attribute instead of seeking AVET (#518). Fixed by the streaming engine's `Scan` (#544): see below.

Expand Down
65 changes: 65 additions & 0 deletions docs/benchmarks/532-check-rows.rs
99C9
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
//! Independent counts for the two unchecked envelope shapes (#532).
//! Run against the pinned source; this reads the fact log, not Datalog.
use minigraf::{FactFilter, FactOrder, OpenOptions};
use serde_json::{Value, json};
use std::path::Path;

fn main() -> anyhow::Result<()> {
let args: Vec<String> = std::env::args().collect();
let dir = Path::new(&args[1]);
let report: Value = serde_json::from_str(&std::fs::read_to_string(&args[2])?)?;
let mut checks = Vec::new();
for scale_report in report["scales"].as_array().unwrap() {
let scale = scale_report["label"].as_str().unwrap();
let fixture = dir.join(format!("fixture-v1-{scale}.graph"));
let meta: Value =
serde_json::from_str(&std::fs::read_to_string(fixture.with_extension("json"))?)?;
let entities = meta["entities"].as_u64().unwrap();
let tx_half = meta["tx_half"].as_u64().unwrap();
let db = OpenOptions::new().read_only(true).path(&fixture).open()?;
let mut log = db.fact_log(
&FactFilter::new()
.tx_range(tx_half..=tx_half)
.order(FactOrder::Storage),
)?;
let cutoff = log.next_batch(1)?.unwrap()[0].tx_id;
drop(log);
let mut ages = db.fact_log(
&FactFilter::new()
.attributes([":age"])
.order(FactOrder::Storage),
)?;
let mut age_count = 0_u64;
let mut as_of_count = 0_u64;
while let Some(batch) = ages.next_batch(1000)? {
for record in batch {
anyhow::ensure!(record.asserted, "fixture age unexpectedly retracted");
age_count += 1;
as_of_count += u64::from(record.tx_id <= cutoff);
}
}
anyhow::ensure!(age_count == entities, "fixture age count mismatch");
let join_count = (0..entities)
.filter(|i| ((i * 7919 + 1) % entities) % 100 == 7)
.count() as u64;
for (name, expected) in [("join_3_scan", join_count), ("as_of_ts_scan", as_of_count)] {
let query = scale_report["queries"]
.as_array()
.unwrap()
.iter()
.find(|q| q["name"] == name)
.unwrap();
let observed = query["rows"].as_u64().ok_or_else(|| {
anyhow::anyhow!("{scale} {name}: no rows in report (error or cap)")
})?;
anyhow::ensure!(
expected == observed,
"{scale} {name}: expected {expected} rows, observed {observed}"
);
checks
.push(json!({"scale":scale,"shape":name,"expected":expected,"observed":observed}));
}
}
println!("{}", serde_json::to_string_pretty(&checks)?);
Ok(())
}
Loading
Loading
0