8000 fix(query): run an expression reading an or-bound variable after the ors (#625) by adityamukho · Pull Request #629 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A `:db/valid-at` pattern in a `not-join` body binds the query's valid time on every path** (#615). When the body also held an expression, sat in an `or` branch, or ran in a query that invokes a rule, the body bound `nil`, so `(not-join [?e] [?e :db/valid-at ?t] [(> ?t 0)])` never matched and excluded nothing; without the expression the same body excluded every entity. It now binds the query's `:valid-at` value (or the current time) as the other paths do. Present since v0.13.1; v2.x is affected too.
- **A `:db/valid-at` pattern in a rule definition binds the query's valid time** (#617). The rule evaluators never received it, so in a rule body the pattern bound `nil` and the query failed with `INT-022`; in a rule's `not` / `not-join` body it bound `nil`, so the negation never matched on it; and in a rule's `or` branch it bound the current time instead of the query's `:valid-at`. Rule bodies, their negation bodies and `or` branches now bind the query's `:valid-at` value, the current time without one, and `nil` under `:any-valid-time`, as patterns outside rules do. v2.x is affected too.
- **An `or` / `or-join` branch whose expression reads an outer variable matches** (#620). A branch such as `[(> ?v 1)]` in `[?e :n ?v] (or [(> ?v 1)] [(< ?v 0)])` was evaluated on its own from an empty binding and joined back afterwards, so its expression saw `?v` unbound and the branch matched nothing: the `or` silently dropped rows it should keep, in queries and in rule bodies. A branch whose expression reads a variable the branch does not bind itself is now seeded with each incoming row, as branches with `not` / `not-join` already were.
- **An expression reading a variable only an `or` binds keeps its rows** (#625). The planner places expressions among the patterns and `not`s, and `or` / `or-join` run after them, so an expression such as `[(> ?x 1)]` in `[?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(> ?x 1)]` ran before `?x` was bound and dropped every row, on the executor, on the streaming engine and in rule bodies. Such an expression now runs after the `or`s, before the `not`s that wait for them, as those `not`s already did.
- **An `or` that shares no variable with the earlier clauses returns its cross product** (#610). Such an `or` — `[?x :edge ?z] (or [?p :edge :b] [?p :edge :c])` — returned no rows when no branch had `not`, `not-join` or a pseudo-attribute, because the incoming rows were replaced by the branch results. Each incoming row now joins every branch row, in queries and rule bodies. `or-join` still refuses a join variable no earlier clause binds (INT-018): it filters, it does not bind.
- **A prepared query keeps a literal `:as-of`, `:valid-at` or `:any-valid-time`** (#603). Binding a prepared query's slots dropped any `:as-of` or `:valid-at` that was not itself a slot, so `PreparedQuery::execute` and `PreparedQuery::query` ran at the current transaction time and valid now: `(query [:find ?v :as-of 1 :where [:a :n ?v]])` prepared returned the current rows, not those at transaction 1, and a prepared per-fact pseudo-attribute query with `:any-valid-time` failed with `INT-026`. Slot forms (`:as-of $tx`, `:valid-at $t`) were not affected. Present since 2026-04-04; fixed on v2.x too.
- **The wasm32 builds are tested again, and the browser layer no longer panics or drops pages.** The repo split (#231) moved the browser and WASI CI jobs to minigraf-wasm, which only builds the package, so core's own wasm tests stopped running and the wasm32 test build broke unnoticed. A new `wasm.yml` workflow runs the `wasm-bindgen` tests in headless Firefox and Chrome, runs the unit and integration tests on `wasm32-wasip1` under Wasmtime, and lints both targets with `clippy -D warnings`. Fixes found on the way: `BrowserDb` skipped any dirty page it could not read when flushing to IndexedDB and still reported success, and now returns the error (also fixed in v2.0.4, #470). The IndexedDB upgrade callback, the IndexedDB backend handle and the page-buffer lock no longer `unwrap()` (on wasm32 a panic aborts the whole instance). IndexedDB keys that are not whole page numbers are rejected instead of truncated. The nine `BrowserBufferBackend` tests used `#[test]`, which the wasm test runner never runs, and now run. File-only modules (`wal`, `storage::verify`, `storage::dir_sync`) are no longer compiled for wasm32, so the wasm32 builds have no warnings.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1588 tests** (1578 passing, 10 ignored; unit + integration + doc).
**1592 tests** (1582 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1588 tests
cargo test # run 1592 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

< 8000 a href="#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5" id="expand-down-link--diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5" class="js-expand directional-expander single-expander" aria-label="Expand Down" data-url="/project-minigraf/minigraf/blob_excerpt/4398d66647a307341a378f6eb0ccde1372a9d53b?context=pull_request&diff=unified&direction=down&in_wiki_context&last_left=137&last_right=137&left=287&left_hunk_size&mode=100644&path=README.md&pull_request_id=4921440490&right=287&right_hunk_size" data-left-range="138-286" data-right-range="138-286"> Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

15 changes: 12 additions & 3 deletions src/query/datalog/evaluator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -824,9 +824,10 @@ impl StratifiedEvaluator {
);

// Plan: assigns index hints + pushes Expr to earliest binding position.
// Not/NotJoin are excluded from plan_clauses above, so `deferred` is
// always empty here — rule bodies keep their existing not-filter path.
let (planned, _deferred) = crate::query::datalog::optimizer::plan(plan_clauses);
// Not/NotJoin are excluded from plan_clauses above, so `deferred` holds
// only expressions whose variables an Or/OrJoin binds (#625); they run
// after the or pass. Rule bodies keep their existing not-filter path.
let (planned, deferred) = crate::query::datalog::optimizer::plan(plan_clauses);

// Process planned clauses in order: Pattern → join, Expr → filter/extend.
let fn_guard = self
Expand Down Expand Up @@ -890,6 +891,14 @@ impl StratifiedEvaluator {
drop(registry_guard);
expanded
};
let candidates = {
let fn_guard = self
.functions
.read()
.map_err(|_| err_coded!(ErrorCode::Qry008))?;
let exprs: Vec<&WhereClause> = deferred.iter().collect();
apply_expr_clauses_in_evaluator(candidates, &exprs, &fn_guard)
};

// Build temp_eval once per rule (outside the binding loop);
// instantiate_head_public only uses storage, not the registry.
Expand Down
31 changes: 26 additions & 5 deletions src/query/datalog/executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -727,15 +727,26 @@ impl DatalogExecutor {
)?;
drop(rules_guard);

// Apply Not/NotJoin clauses the planner couldn't place inline (#248) — their
// variables are only bound by the Or/OrJoin pass above, so they must run here,
// after it. Sorted cheapest-first, same as the pre-#248 post-filter ordering.
// Apply Not/NotJoin clauses the planner couldn't place inline (#248), and
// expressions (#625) — their variables are only bound by the Or/OrJoin pass
// above, so they must run here, after it. Sorted cheapest-first, same as the
// pre-#248 post-filter ordering; expressions cost 0 and come first, so a
// stable sort keeps them ahead of the nots, in text order.
let mut deferred_not_clauses = deferred_not_clauses;
if optimizer::reorder() {
deferred_not_clauses.sort_by_key(optimizer::clause_cost);
}
for clause in deferred_not_clauses {
bindings = match clause {
WhereClause::Expr { expr, binding: out } => {
let replace = out
.as_deref()
.is_some_and(|var| binders.expression(&expr, var));
bindings
.into_iter()
.filter_map(|b| apply_expr(&expr, out.as_deref(), b, &registry, replace))
.collect()
}
WhereClause::Not(body) => apply_not_clause(
bindings,
&body,
Expand Down Expand Up @@ -978,8 +989,9 @@ impl DatalogExecutor {
)?;
drop(rules_guard);

// Apply Not/NotJoin clauses the planner couldn't place inline (#248) — their
// variables are only bound by the Or/OrJoin pass above. (The StratifiedEvaluator
// Apply Not/NotJoin clauses the planner couldn't place inline (#248), and
// expressions (#625) — their variables are only bound by the Or/OrJoin pass
// above. Expressions come first, as above. (The StratifiedEvaluator
// handles `not`/`not-join` in rule bodies; this handles them appearing directly
// in the query body alongside rule invocations.)
let mut deferred_not_clauses = deferred_not_clauses;
Expand All @@ -988,6 +1000,15 @@ impl DatalogExecutor {
}
for clause in deferred_not_clauses {
bindings = match clause {
WhereClause::Expr { expr, binding: out } => {
let replace = out
.as_deref()
.is_some_and(|var| binders.expression(&expr, var));
bindings
.into_iter()
.filter_map(|b| apply_expr(&expr, out.as_deref(), b, &registry, replace))
.collect()
}
WhereClause::Not(body) => apply_not_clause_derived(
bindings,
&body,
Expand Down
77 changes: 57 additions & 20 deletions src/query/datalog/optimizer.rs
10000
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,11 @@ pub(crate) fn clause_ref_vars(clauses: &[WhereClause]) -> std::collections::Hash
/// clause, which the executor applies after this plan) and must be evaluated by the
/// executor's post-filter instead, to avoid running it before its dependency is bound.
///
/// An `Expr` reading a variable that no pattern in the list and no `Expr` placed
/// before it binds is deferred the same way (#625), ahead of the deferred
/// `Not`/`NotJoin` clauses and in text order, since a deferred `not` may read what
/// it binds.
///
/// Returns `(interleaved, deferred)` where `interleaved` pairs each Pattern with
/// `Some(hint)` and each Expr/Not/NotJoin with `None`.
pub fn plan(
Expand Down Expand Up @@ -361,16 +366,31 @@ pub(crate) fn plan_seeded(
}

// Push each Expr to the earliest position where all its variables are bound.
// An Expr reading a variable that no pattern here and no Expr placed before
// it binds depends on a source outside this list (an Or/OrJoin, #625): it is
// deferred, ahead of the deferred nots, which may read what it binds.
let mut deferred: Vec<WhereClause> = Vec::new();
let mut placeable: std::collections::HashSet<String> = seeded.clone();
placeable.extend(result.iter().flat_map(|(c, _)| match c {
WhereClause::Pattern(p) => pattern_bound_vars(p),
_ => Vec::new(),
}));
for expr_clause in exprs {
let vars: std::collections::HashSet<String> =
if let WhereClause::Expr { expr, .. } = &expr_clause {
expr_vars(expr).into_iter().collect()
let (vars, out): (std::collections::HashSet<String>, Option<String>) =
if let WhereClause::Expr { expr, binding } = &expr_clause {
(expr_vars(expr).into_iter().collect(), binding.clone())
} else {
Default::default()
};
if !vars.is_subset(&placeable) {
deferred.push(expr_clause);
continue;
}
placeable.extend(out);

let mut bound = seeded.clone();
// Default: append at end (covers no-var Exprs and vars never bound by any pattern).
// Default: append at end (covers no-var Exprs and Exprs reading what an
// Expr placed earlier binds).
let mut insert_pos = result.len();

if !vars.is_empty() && vars.is_subset(&bound) {
Expand Down Expand Up @@ -399,7 +419,6 @@ pub(crate) fn plan_seeded(
not_clauses.sort_by_key(|c| std::cmp::Reverse(clause_cost(c)));
}

let mut deferred: Vec<WhereClause> = Vec::new();
for nc in not_clauses {
let required: std::collections::HashSet<String> = match &nc {
// `?_`-prefixed vars are the wildcard idiom (e.g. `(not [?e :attr ?_x])`)
Expand Down Expand Up @@ -860,24 +879,42 @@ mod tests {
"no-var Expr must be last"
);
}
#[test]
fn test_expr_unbound_var_goes_to_end() {
// ?x is never bound by any pattern
let p1 = WhereClause::Pattern(make_pattern(var("e"), kw(":val"), var("v")));
let expr = WhereClause::Expr {
fn gt(v: &str, n: i64, binding: Option<&str>) -> WhereClause {
WhereClause::Expr {
expr: Expr::BinOp(
BinOp::Gt,
Box::new(Expr::Var("?x".to_string())),
Box::new(Expr::Lit(Value::Integer(0))),
Box::new(Expr::Var(v.to_string())),
Box::new(Expr::Lit(Value::Integer(n))),
),
binding: None,
};
let (planned, _deferred) = plan(vec![p1, expr]);
assert_eq!(planned.len(), 2);
assert!(
matches!(planned[1].0, WhereClause::Expr { .. }),
"Expr with unbound var must be last"
);
binding: binding.map(str::to_string),
}
}
#[test]
fn test_expr_unbound_var_is_deferred() {
// ?x is never bound by any pattern: an Or binds it, after the plan (#625).
let p1 = WhereClause::Pattern(make_pattern(var("e"), kw(":val"), var("v")));
let expr = gt("?x", 0, None);
let (planned, deferred) = plan(vec![p1, expr.clone()]);
assert_eq!(planned.len(), 1, "the Expr must not be planned");
assert_eq!(deferred, vec![expr]);
}
#[test]
fn test_expr_reading_placed_expr_output_is_planned() {
// ?y comes from an Expr whose input a pattern binds: both are planned.
let p1 = WhereClause::Pattern(make_pattern(var("e"), kw(":val"), var("v")));
let (bind, read) = (gt("?v", 0, Some("?y")), gt("?y", 0, None));
let (planned, deferred) = plan(vec![p1, bind, read]);
assert!(deferred.is_empty());
assert_eq!(planned.len(), 3);
}
#[test]
fn test_deferred_exprs_come_before_deferred_nots_in_text_order() {
let p1 = WhereClause::Pattern(make_pattern(var("e"), kw(":val"), var("v")));
let not_clause = WhereClause::Not(vec![gt("?y", 0, None)]);
let (bind, read) = (gt("?x", 0, Some("?y")), gt("?y", 0, None));
let (planned, deferred) = plan(vec![p1, not_clause.clone(), bind.clone(), read.clone()]);
assert_eq!(planned.len(), 1);
assert_eq!(deferred, vec![bind, read, not_clause]);
}

// ── Not/NotJoin push-down (#248) ─────────────────────────────────────
Expand Down
4 changes: 2 additions & 2 deletions src/query/engine/plan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -307,8 +307,8 @@ pub(crate) fn plan(query: &DatalogQuery) -> Option<Plan> {
b.clause(c.clone())?;
}
}
// A `not` whose variables only an `or` binds runs last, as on the
// executor.
// An expression or a `not` whose variables only an `or` binds runs
// last, expressions first, as on the executor (#625).
for c in sorted(deferred) {
b.clause(c)?;
}
Expand Down
38 changes: 38 additions & 0 deletions tests/disjunction_test.rs
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -754,3 +754,41 @@ fn test_or_cross_product_after_pseudo_attribute() {
.unwrap();
assert_eq!(result_count(&r), 2, "two edges, one tx-count each, one ?p");
}

// ── #625: an expression reading a variable only an `or` binds ────────────────

fn issue_625_db() -> Minigraf {
let db = db();
db.execute("(transact [[:a :n 1] [:b :n 2] [:c :ok true]])")
.unwrap();
db
}

#[test]
fn test_expression_after_or_reads_or_bound_variable() {
let db = issue_625_db();
let r = db
.execute("(query [:find ?e ?x :where [?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(> ?x 1)]])")
.unwrap();
assert_eq!(result_count(&r), 1, "only :b has ?x over 1");
let r = db
.execute(
"(query [:find ?y :where [?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(+ ?x 10) ?y] (not [(> ?y 11)])])",
)
.unwrap();
assert_eq!(result_count(&r), 1, "the not sees the expression's ?y");
}

#[test]
fn test_expression_after_or_reads_or_bound_variable_with_rules() {
let db = issue_625_db();
db.execute("(rule [(num ?e) [?e :n _]])").unwrap();
db.execute("(rule [(big ?e ?x) [?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(> ?x 1)]])")
.unwrap();
let r = db
.execute("(query [:find ?x :where (num ?e) (or [?e :n ?x] [?e :ok ?x]) [(> ?x 1)]])")
.unwrap();
assert_eq!(result_count(&r), 1, "query calling a rule");
let r = db.execute("(query [:find ?x :where (big ?e ?x)])").unwrap();
assert_eq!(result_count(&r), 1, "rule body");
}
57 changes: 57 additions & 0 deletions tests/plan_independence_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -435,3 +435,60 @@ fn or_branch_expression_reads_outer_variable_in_rule_body() {
let r = same(&db, "(query [:find ?w :where (pick ?e ?w)])");
assert_eq!(ints(&r), vec!["i:1", "i:2"], "or-and in a rule body");
}

// ── #625: an expression reading a variable only an `or` binds ────────────────

fn issue_625_data(db: &Minigraf) {
db.execute("(transact [[:a :n 1] [:b :n 2] [:c :ok true]])")
.unwrap();
}

#[test]
fn expression_reads_variable_bound_only_by_or() {
let db = Minigraf::in_memory().unwrap();
issue_625_data(&db);
let r = same(
&db,
"(query [:find ?e ?x :where [?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(> ?x 1)]])",
);
assert_eq!(r.len(), 1, "only :b has ?x over 1");
// Mixed: one variable from a pattern, the other from the or.
let r = same(
&db,
"(query [:find ?x :where [?e :n ?v] (or [?e :n ?x] [?e :ok ?x]) [(= ?x ?v)]])",
);
assert_eq!(ints(&r), vec!["i:1", "i:2"], "?x equals ?v on both rows");
}

#[test]
fn expression_chain_on_variable_bound_only_by_or() {
let db = Minigraf::in_memory().unwrap();
issue_625_data(&db);
let r = same(
&db,
"(query [:find ?y :where [?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(+ ?x 10) ?y] [(> ?y 11)]])",
);
assert_eq!(ints(&r), vec!["i:12"], "?y from ?x, then filtered");
// A not reading the expression's output runs after it.
let r = same(
&db,
"(query [:find ?y :where [?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(+ ?x 10) ?y] (not [(> ?y 11)])])",
);
assert_eq!(ints(&r), vec!["i:11"], "the not sees ?y");
}

#[test]
fn expression_reads_variable_bound_only_by_or_with_rules() {
let db = Minigraf::in_memory().unwrap();
issue_625_data(&db);
db.execute("(rule [(num ?e) [?e :n _]])").unwrap();
let r = same(
&db,
"(query [:find ?x :where (num ?e) (or [?e :n ?x] [?e :ok ?x]) [(> ?x 1)]])",
);
assert_eq!(ints(&r), vec!["i:2"], "query calling a rule");
db.execute("(rule [(big ?e ?x) [?e :n _] (or [?e :n ?x] [?e :ok ?x]) [(> ?x 1)]])")
.unwrap();
let r = same(&db, "(query [:find ?x :where (big ?e ?x)])");
assert_eq!(ints(&r), vec!["i:2"], "rule body");
}
Loading
0