8000 perf: unstable sorts for unique query metadata (#795) by adityamukho · Pull Request #799 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Performance

- **Unstable sorts for unique query metadata** (#795). Prepared-query slot names, an `or`'s branch variables and a subplan's imported variables sort with `sort_unstable`: their keys are unique, or, for a join variable named twice, the tied entries are identical, so order and plans are unchanged. Release builds shrink by 1,008 bytes (native CLI), 1,752 bytes (WASI) and 1,886 bytes (browser `--lib`); prepare and planning time is unchanged within noise. No API, dependency or file-format change.

- **Release certified aggregate groups** (#781). Ordered aggregation now includes ordinary `sum`/`avg`, fixed-scalar `min`/`max`, and Integer/nil `sum-distinct`. Effective grouping tuples may include hidden `:with` keys, constants and repetitions when they reduce to one ascending Integer/Boolean dimension; existing single numeric/Boolean keys remain supported. Arguments are certified from constants, metadata or raw attribute history using the pinned reader, with pending/staged data and unsupported origins retaining full-input evaluation. Every `:with` Sort is retained. Limits count emitted groups after min/max drops; a retained Sort still drains its input. Completed groups are released, but one growing distinct group remains linear. Small selective grouped queries pay proof startup cost. Common result-buffer memory failures can precede later unread source failures, as on the existing ordered-count path; registered finalizers retain full-drain behavior. No aggregate semantics, syntax, public API, dependency or storage format changes.

- **Resolve expression keyword literals once per operator** (#774). `=` and `!=` reuse a literal keyword's entity UUID across input rows, including nested expressions, rule bodies and prepared slots. The cache contains only literals from that expression and is discarded with the operator; dynamic operands retain the existing comparison semantics. No API, dependency or file-format change. See the [measurements](docs/benchmarks/774-expression-keywords.md).
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**2056 tests** (2037 active, 19 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**2057 tests** (2038 active, 19 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

The #760 deterministic storage suite adds three native unit tests in a separate `--cfg minigraf_shuttle` build, outside the default count. Run only `graph::storage::deterministic::` with that cfg; ordinary unit tests do not enter its scheduler. `deterministic-schedules.yml` runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See `docs/testing/deterministic-schedules.md` for scope, controls and findings.
Expand Down
8 changes: 5 additions & 3 deletions docs/TEST_COVERAGE.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@

| | Default features | With `--features test-hooks` |
|---|---|---|
| **Active** | 2,037 | 2,072 |
| **Active** | 2,038 | 2,073 |
| **Ignored** | 19 | 20 |
| **Total** | 2,056 | 2,092 |
| **Total** | 2,057 | 2,093 |

Counts above are the current Cargo test inventory (active versus ignored),
including #777, #789 and #781. Verification results below identify which suites ran.
including #777, #789, #781 and #795. Verification results below identify which suites ran.

**#781 verification (2026-10-29)**: 24 active tests and two ignored measurement
probes are added. On implementation `90cb88924`, the complete default suite
Expand Down Expand Up @@ -110,6 +110,8 @@ execution was not repeated; its 2,066-test inventory was verified.

## Covered Areas

- **Unique query metadata sorts** (#795; `src/query/engine/tests.rs`, one test): a join variable named twice in a `not-join` or `or-join`, the only equal-key tie in the planner's import sort, returns the rows of the same clause with it named once, under the chosen, hashed and correlated body forms.

- **Certified ordered aggregates** (#781; `src/query/engine/aggregate_order_tests.rs`, `order/tests.rs`, `order/arguments.rs`, `tests/input_order_test.rs`): sum/avg, scalar extrema, Integer/nil distinct sums, complete effective tuples, hidden `:with` keys, output remapping and raw-history argument domains. The original #641 count-only eligibility coverage is extended with explicit scalar/numeric answers, retained Sort, dropped groups, source-error/resource-error competition, pending/staged invalidation, optional-proof rollback, portable committed readers, and separately bounded Aggregate/IdSpace controls. Unproved tuples/arguments and registered finalizers retain full-input behavior.

- **Aggregate value handling** (#789; `tests/aggregate_value_handling_test.rs`, seven default tests and one with `test-hooks`): all value variants, numeric-only sum/distinct sum, mixed-type extrema, existing nil/nonnumeric groups versus empty queries, grouped and ungrouped queries, hidden `:with` groups and join multiplicity, prepared binds/cursors, recursive rules, transaction-time views and staged rollback, checkpoint/reopen, and default/bounded partition windows over identical populations. Three added accumulator tests check adversarial permutations, signed-zero/NaN bits, infinities, integer boundaries, the explicit total order and distinct membership prediction. Two added engine tests check retained numeric-member accounting, duplicates, QRY-012 boundaries, upstream failures and release on rewind/close. Rewritten error-precedence tests use real UDF finalizer failures under Limit/Offset; ordered-aggregate eligibility and frozen answers are unchanged.
Expand Down
2 changes: 1 addition & 1 deletion src/query/datalog/prepared.rs
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ fn collect_slot_names(query: &DatalogQuery) -> Vec<String> {
}
collect_slots_from_clauses(&query.where_clauses, &mut names);
let mut result: Vec<String> = names.into_iter().collect();
result.sort();
result.sort_unstable();
result
}

Expand Down
6 changes: 4 additions & 2 deletions src/query/engine/plan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1396,7 +1396,7 @@ impl Builder {
.filter(|v| !self.slots.by_var.contains_key(*v))
.cloned()
.collect();
own.sort();
own.sort_unstable();
if k == 0 {
names = own;
} else if own != names {
Expand Down Expand Up @@ -1438,7 +1438,9 @@ impl Builder {
.into_iter()
.filter_map(|v| self.slots.by_var.get(&v).map(|&s| (v, s)))
.collect();
imports.sort_by_key(|&(_, s)| s);
// Every name has its own slot, so equal slots are one variable named
// twice (`[?x ?x]`): ties are identical.
imports.sort_unstable_by_key(|&(_, s)| s);
let seeded: HashSet<String> = imports.iter().map(|(v, _)| v.clone()).collect();
let (seeded_order, _) = optimizer::plan_seeded(body.to_vec(), &seeded);
// A probe per outer row holds nothing: correlated when the first
Expand Down
44 changes: 44 additions & 0 deletions src/query/engine/tests.rs
63AF
Original file line number Diff line number Diff line change
Expand Up @@ -1466,6 +1466,50 @@ fn planner_chooses_the_body_form() {
assert_eq!(forms(constant), ["correlated"]);
}

/// A join variable named twice imports one slot twice, the only tie in the
/// planner's import sort (#795): each body form answers as with it named
/// once.
#[test]
fn repeated_join_variable_matches_single() {
use crate::query::datalog::optimizer::{Subplans, set_subplans};
let db = Minigraf::in_memory().unwrap();
db.execute("(transact [[:p1 :a 1] [:p2 :a 2] [:p3 :a 3] [:q1 :b 2] [:q2 :b 3] [:q2 :c 3]])")
.unwrap();
let sorted = |q: &str| {
let QueryResult::QueryResults { results, .. } = db.execute(q).expect(q) else {
panic!("expected query results");
};
let mut out: Vec<String> = results.iter().map(|r| format!("{r:?}")).collect();
out.sort();
out
};
let pairs = [
(
"(query [:find ?e :where [?e :a ?x] (not-join [?x ?x] [?f :b ?x])])",
"(query [:find ?e :where [?e :a ?x] (not-join [?x] [?f :b ?x])])",
1,
),
(
"(query [:find ?e ?x :where [?e :a ?x] (not-join [?e ?x ?e] [?f :b ?x] [?f :c ?x])])",
"(query [:find ?e ?x :where [?e :a ?x] (not-join [?e ?x] [?f :b ?x] [?f :c ?x])])",
2,
),
(
"(query [:find ?e :where [?e :a ?x] (or-join [?x ?x] [?f :b ?x] [?f :c ?x])])",
"(query [:find ?e :where [?e :a ?x] (or-join [?x] [?f :b ?x] [?f :c ?x])])",
2,
),
];
for mode in [Subplans::Choose, Subplans::Hash, Subplans::Correlated] {
let _m = set_subplans(mode);
for (twice, once, n) in pairs {
let rows = sorted(twice);
assert_eq!(rows.len(), n, "{twice}");
assert!(rows == sorted(once), "{twice}");
}
}
}

/// Which negations the engine runs (#546): every `not` / `not-join`, with
/// a rule invocation too since rule queries run here (#548).
#[test]
Expand Down
Loading
0