8000 feat(engine): or and or-join on the streaming engine (#546 PR 2) by adityamukho · Pull Request #626 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CHANGELOG.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,11 +135,11 @@ cargo run < demos/demo_negation.txt
- `stratification.rs`: `DependencyGraph`, `stratify()` — negative edges + cycle detection
- `rules.rs`: `RuleRegistry` — thread-safe rule management
- `types.rs`: `EdnValue`, `Pattern`, `DatalogQuery`, `AsOf`, `ValidAt`, `WhereClause` (incl. `Not`, `NotJoin`); `PseudoAttr` enum, `AttributeSpec` wrapper
- `optimizer.rs`: Selectivity-based join reordering on every build (the `wasm` feature is a deprecated no-op). `reorder()` gates every reordering site; tests turn it off per thread (`set_reorder`, `minigraf::__reorder` under `test-oracle`, #596); `plan_seeded` orders a body with its imported variables counted as bound, and `subplans()` lets tests force the engine's negation bodies hashed or correlated (`set_subplans`, `minigraf::__subplans`, #546)
- `optimizer.rs`: Selectivity-based join reordering on every build (the `wasm` feature is a deprecated no-op). `reorder()` gates every reordering site; tests turn it off per thread (`set_reorder`, `minigraf::__reorder` under `test-oracle`, #596); `plan_seeded` orders a body with its imported variables counted as bound, and `subplans()` lets tests force the engine's negation bodies and `or` branches hashed or correlated (`set_subplans`, `minigraf::__subplans`, #546)
- `prepared.rs`: `BindValue`, `PreparedQuery` — parse-once/execute-many with named `$slot` bind slots; `prepare_query`, `substitute`
- `magic_sets.rs`: Magic-sets rewriting for demand-driven recursive rule evaluation (not applied to rules with `not`/`not-join`)

3a. **`src/query/engine/`** — Streaming query engine (#473), beside the executor until #551: `plan.rs` (which queries it runs: patterns, expression clauses and `not`/`not-join` without rule invocations, in `optimizer::plan` order (deferred `not`s last); a per-fact pseudo-attribute pattern reads the metadata column its source's scan fills (the hidden variable the parser named, #594), and `:db/valid-at` joins as `[e _ _]`; plain `:find` variables, today), `scan.rs` (one pattern over committed key ranges via `CommittedReader::key_source`/`live_keys`, AVET ranges for cross-kind constants, then the pending range by whole runs; one entity per probe for a join), `join.rs` (`IndexNestedLoopJoin` when the pattern's entity is bound, `HashJoin` otherwise; `JoinKey`: floats by IEEE `==`, a keyword equals a ref to the entity it names), `filter.rs` (`Filter`/`Bind` and pseudo-attribute steps over batches; a `Bind` on a bound slot keeps the rows where the result is join-equal, #596), `anti.rs` (`AntiJoin`, #546: a body with the same rows for every outer row runs once into a hash set of its imported-variable keys; any other body, or one whose first pattern's entity is imported, runs per outer row from `seed.rs`'s one-row `Seed`, via `Operator::rewind`; never deduplicates), `cell.rs`/`batch.rs` (id cells in columnar batches of 1,024), `ids.rs` (DICT and provisional ids, `keyword_eid`), `project.rs` (ids → values for returned rows only), `oracle.rs` (with `cfg(test)` or the `test-oracle` feature, every engine query also runs on the executor and must agree). `mod.rs`: `Operator` (`next_batch`/`rewind`/`close`/`counters`), `Stream` (the operator tree over an owned `Snapshot`, pulled one batch at a time). `DatalogExecutor::open` / `execute_query` try it first: a cursor gets the `Stream` (`Answer::Stream`), `execute` drains it.
3a. **`src/query/engine/`** — Streaming query engine (#473), beside the executor until #551: `plan.rs` (which queries it runs: patterns, expression clauses, `not`/`not-join` and `or`/`or-join` without rule invocations, in `optimizer::plan` order, then the `or`s in text order, then the deferred `not`s; a per-fact pseudo-attribute pattern reads the metadata column its source's scan fills (the hidden variable the parser named, #594), and `:db/valid-at` joins as `[e _ _]`; plain `:find` variables, today), `scan.rs` (one pattern over committed key ranges via `CommittedReader::key_source`/`live_keys`, AVET ranges for cross-kind constants, then the pending range by whole runs; one entity per probe for a join), `join.rs` (`IndexNestedLoopJoin` when the pattern's entity is bound, `HashJoin` otherwise; `JoinKey`: floats by IEEE `==`, a keyword equals a ref to the entity it names), `filter.rs` (`Filter`/`Bind` and pseudo-attribute steps over batches; a `Bind` on a bound slot keeps the rows where the result is join-equal, #596), `anti.rs` (`AntiJoin`, #546: a body with the same rows for every outer row runs once into a hash set of its imported-variable keys; any other body, or one whose first pattern's entity is imported, runs per outer row from `seed.rs`'s one-row `Seed`, via `Operator::rewind`; never deduplicates), `union.rs`/`distinct.rs` (`Union` and `Distinct`, #546: each `or` branch planned as a `not` body is, hashed into a table or run per outer row; then one row per distinct binding, as the executor's `Value`s compare), `cell.rs`/`batch.rs` (id cells in columnar batches of 1,024), `ids.rs` (DICT and provisional ids, `keyword_eid`), `project.rs` (ids → values for returned rows only), `oracle.rs` (with `cfg(test)` or the `test-oracle` feature, every engine query also runs on the executor and must agree). `mod.rs`: `Operator` (`next_batch`/`rewind`/`close`/`counters`), `Stream` (the operator tree over an owned `Snapshot`, pulled one batch at a time). `DatalogExecutor::open` / `execute_query` try it first: a cursor gets the `Stream` (`Answer::Stream`), `execute` drains it.

4. **`src/temporal.rs`** — UTC-only timestamp parsing (avoids chrono CVE GHSA-wcg3-cvx6-7396)

Expand Down Expand Up @@ -204,7 +204,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1578 tests** (1568 passing, 10 ignored; unit + integration + doc).
**1588 tests** (1578 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1578 tests
cargo test # run 1588 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
5 changes: 3 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

21 changes: 21 additions & 0 deletions docs/superpowers/plans/2026-10-18-546-negation-disjunction.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,27 @@ The exclusion-set paths (`apply_not_clause`, `apply_not_join_clause`) already co
replacing a cell could not change an answer; this matches the executor, whose body
paths pass `replace = false`.

11. **Which `or`s the engine runs (PR 2).** A branch is planned by the same code as a
`not` body (`Builder::sub`), with its `not`s, placed or deferred, as in a query. The
executor keeps:
- a branch holding a rule invocation or a nested `or` / `or-join` (the executor
evaluates a nested one unseeded inside the branch, where a join variable bound only
outside it is an `INT-031`);
- a branch with a predicate the function registry may lack: the executor reports
`QRY-007` only when a row reaches it;
- an `or` whose branches bind different hidden metadata variables (each branch's own
sources): the executor's rows keep them;
- a branch whose `not` reads a variable only the branch's expression binds: the
executor applies a branch's `not`s before its expressions.

Three more executor answers depended on the path; the per-row path, taken when a
branch reads the outer row, gave the spec's answer. (a) and (b) came up while
comparing the two paths for this plan, (c) from the oracle. PR 2 fixes them in the
executor: (a) a branch naming different outer variables than another met no outer
row on the joined path; (b) an `or-join` branch that does not bind every join
variable lost its rows on the joined path; (c) `:db/valid-at` in a branch read the
wall clock, not the query's "now".

## Steps

### 1. Executor: seeded bodies (PR 1)
Expand Down
18 changes: 18 additions & 0 deletions examples/perf_envelope.rs
Original file line number Diff line number Diff line change
Expand Up @@ -518,6 +518,24 @@ fn shapes(e: usize, tx_half: u64, hot_tx_half: u64) -> Vec<Shape> {
.count()
},
},
// #546: an `or-join` of two closed branches over different
// groups, each read once and keyed on a value.
Shape {
name: "or_2",
query: "[:find ?e :where [?e :group 7] [?e :age ?a] \
(or-join [?a] (and [?f :group 17] [?f :age ?a] [(< ?a 30)]) \
(and [?g :group 27] [?g :age ?a] [(> ?a 60)]))]"
.into(),
rows: {
let ages: std::collections::HashSet<usize> = (0..e)
.filter(|i| (i % 100 == 17 && i % 90 < 30) || (i % 100 == 27 && i % 90 > 60))
.map(|i| i % 90)
.collect();
(0..e)
.filter(|i| i % 100 == 7 && ages.contains(&(i % 90)))
.count()
},
},
Shape {
name: "recursive_ancestors",
query: format!("[:find ?b :where (boss :p{leaf} ?b)]"),
Expand Down
2 changes: 1 addition & 1 deletion src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ pub fn __reorder(on: bool) -> ReorderGuard {
query::datalog::optimizer::set_reorder(on)
}

// Plan-independence tests (#546): force how negation bodies run.
// Plan-independence tests (#546): force how negation bodies and `or` branches run.
#[cfg(feature = "test-oracle")]
#[doc(hidden)]
pub use query::datalog::optimizer::{Subplans, SubplansGuard};
Expand Down
Loading
Loading
0