8000 feat(rules): one total max_derived_facts limit; max_results a deprecated alias (#553) by adityamukho · Pull Request #663 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
8000
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- A string with no closing quote fails with the new `PRS-109` (it ran to the end of the input), and a `:` with no name after it, such as `:+`, with the new `PRS-110` (it was the keyword `:`).

A lone `-` stays the subtraction operator; the grammar now lists it with the other operators. The verifier's #651 entries are removed.
- **One rule limit, `max_derived_facts`, on the total facts a rule query derives** (#553). `OpenOptions::max_derived_facts` and `:max-derived-facts` now limit every fact the rules of one query derive, across iterations and strata (the database's own facts do not count); they used to limit the new facts of one iteration. `OpenOptions::max_results` and `:max-results`, which already limited the total (#528), are deprecated aliases of it (`#[deprecated]` on the Rust field and builder method); the lower of the two applies, and a per-query value still replaces the database's. `OpenOptions::max_results` now defaults to no limit of its own, so it can lower the limit but not raise it: a caller who raised `max_results` above 1,000,000 must raise `max_derived_facts` instead. Neither ever limited the rows a query returns, whatever the name said; `:limit` (#306) will. A caller who lowered `max_derived_facts` to cap each iteration now caps the total, so a long recursion whose iterations each stayed under the limit can now fail with `INT-020`. The `INT-020` text now reads `evaluator: iteration or derived-fact limit exceeded: Max derived facts (N) exceeded. …`. The 1,000-iteration limit and the 1,000,000 default are unchanged.

### Added

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1672 tests** (1662 passing, 10 ignored; unit + integration + doc).
**1680 tests** (1670 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1672 tests
cargo test # run 1680 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
27 changes: 17 additions & 10 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ with no `CodedError` anywhere in its chain.
| INT-017 | Pseudo-attribute not valid in this position | Internal |
| INT-018 | Variable not bound by any outer/earlier clause | Internal |
| INT-019 | Datalog parser internal error | Internal |
| INT-020 | Evaluator iteration/result limit exceeded | Internal |
| INT-020 | Evaluator iteration/derived-fact limit exceeded | Internal |
| INT-021 | Evaluator: unsupported where-clause in evaluate_rule | Internal |
| INT-022 | Datalog evaluator internal error | Internal |
| INT-023 | Packed page: fact exceeds slot capacity | Internal |
Expand Down Expand Up @@ -3591,20 +3591,27 @@ matches on `INT-019` for those errors must match the new codes.
**Resolution**:
- Report the input that raised it as a bug.

### INT-020 Evaluator iteration/result limit exceeded
### INT-020 Evaluator iteration/derived-fact limit exceeded

**Error text**: `evaluator: iteration or result limit exceeded: {}`
**Error text**: `evaluator: iteration or derived-fact limit exceeded: {}`

**Cause**: The recursive rule evaluator (semi-naive evaluation) exceeded its
maximum iteration count, maximum derived facts per iteration, or maximum
number of facts derived in total (`max_results`; the database's own facts do
not count) — guards against infinite recursion/cycles or
runaway-generating rules.
**Cause**: The recursive rule evaluator exceeded its maximum iteration count
(1,000), or its rules derived more facts in total than `max_derived_facts`
allows (`OpenOptions::max_derived_facts` or `:max-derived-facts`, default
1,000,000; the database's own facts do not count) — guards against infinite
recursion/cycles or runaway-generating rules. `max_results` / `:max-results`
are deprecated aliases of that limit; when both are given, the lower applies.
Neither limits the rows a query returns.

Before v3.0.0, `max_derived_facts` limited the facts derived in one iteration,
`max_results` limited the total, and the text read `iteration or result limit
exceeded`. From v3.0.0 there is one limit, on the total (#553).

**Resolution**:
- Check the rule for an unintended cycle (e.g. `(ancestor ?x ?y) [?x
:parent ?y]` combined with a cyclic `:parent` graph). Raise the relevant
limit via query options if the workload is legitimately large.
:parent ?y]` combined with a cyclic `:parent` graph). Raise
`max_derived_facts` (or `:max-derived-facts` on the query) if the workload
is legitimately large.

### INT-021 Evaluator: unsupported where-clause in evaluate_rule

Expand Down
2 changes: 1 addition & 1 deletion docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/cursor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
//! Queries the streaming engine runs (#473) produce rows as they are pulled,
//! one internal batch at a time: closing or dropping the cursor stops the
//! work. Other queries compute the whole answer when the cursor opens, so
//! `max_results` and `max_derived_facts` apply exactly as they do to
//! `max_derived_facts` applies exactly as it does to
//! [`crate::Minigraf::execute`]. Either way a cursor is owned (it borrows
//! nothing from the database handle), and [`Cursor::next_batch`] can fail.

Expand Down
187 changes: 149 additions & 38 deletions src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ use crate::graph::FactStorage;
use crate::graph::storage::Snapshot;
use crate::graph::types::Value;
use crate::query::datalog::evaluator::DEFAULT_MAX_DERIVED_FACTS;
use crate::query::datalog::evaluator::DEFAULT_MAX_RESULTS;
use crate::query::datalog::executor::DatalogExecutor;
use crate::query::datalog::executor::QueryResult;
use crate::query::datalog::functions::{
Expand Down Expand Up @@ -137,11 +136,18 @@ pub struct OpenOptions {
/// [`read_only`](Self::read_only) handle. A tool that reads the whole file
/// can keep it resident with `file_size / 4096` pages.
pub page_cache_size: usize,
/// Maximum facts that can be derived per recursive rule iteration.
/// Defaults to 1_000_000. Use to prevent runaway recursive rules.
/// Maximum facts the rules of one query may derive in total; the
/// database's own facts do not count. Defaults to 1_000_000. Stops a
/// runaway recursive rule with `INT-020`; it never limits returned rows.
pub max_derived_facts: usize,
/// Maximum facts a rule query may derive. Defaults to 1_000_000.
/// The database's own facts do not count against it.
/// Deprecated alias of [`max_derived_facts`](Self::max_derived_facts):
/// the lower of the two applies (#553). Defaults to `usize::MAX`, no
/// limit of its own, so it can lower the limit but not raise it; raise
/// `max_derived_facts` instead. It never limited returned rows.
#[deprecated(
since = "3.0.0",
note = "use `max_derived_facts`; the lower of the two applies"< 10000 /td>
)]
pub max_results: usize,
/// Open even when the filesystem cannot lock files at all.
///
Expand Down Expand Up @@ -181,13 +187,14 @@ pub struct OpenOptions {
pub read_only: bool,
}

#[allow(deprecated)] // `max_results`, kept as an alias until it is removed
impl Default for OpenOptions {
fn default() -> Self {
OpenOptions {
wal_checkpoint_threshold: 1000,
page_cache_size: 256,
max_derived_facts: DEFAULT_MAX_DERIVED_FACTS,
max_results: DEFAULT_MAX_RESULTS,
max_results: usize::MAX,
allow_unlocked: false,
synchronous: SyncMode::Full,
read_only: false,
Expand Down Expand Up @@ -220,25 +227,38 @@ impl OpenOptions {
self
}

/// Set the maximum facts that can be derived per recursive rule iteration.
/// Set the maximum facts the rules of one query may derive in total.
///
/// Defaults to 1_000_000. Use lower values to prevent runaway recursive rules
/// from consuming excessive memory. Can be overridden per-query using
/// `:max-derived-facts N` in the query vector.
/// Defaults to 1_000_000. The database's own facts do not count against it,
/// and queries without rules are not limited by it. A query that derives
/// more fails with `INT-020`, which stops a runaway recursive rule. Can be
/// overridden per query with `:max-derived-facts N` in the query vector.
pub fn max_derived_facts(mut self, n: usize) -> Self {
self.max_derived_facts = n;
self
}

/// Set the maximum number of facts a rule query may derive.
///
/// Defaults to 1_000_000. The database's own facts do not count against it,
/// and queries without rules are not limited by it.
/// Deprecated alias of [`max_derived_facts`](Self::max_derived_facts):
/// the lower of the two applies (#553), so it can lower the limit but not
/// raise it above `max_derived_facts`. It never limited the rows a query
/// returns.
#[deprecated(
since = "3.0.0",
note = "use `max_derived_facts`; the lower of the two applies"
)]
#[allow(deprecated)]
pub fn max_results(mut self, n: usize) -> Self {
self.max_results = n;
self
}

/// The limit rule queries run with: the lower of `max_derived_facts` and
/// its deprecated alias `max_results` (#553).
#[allow(deprecated)]
fn derived_fact_limit(&self) -> usize {
self.max_derived_facts.min(self.max_results)
}

/// Open even when the filesystem cannot lock files. See
/// [`OpenOptions::allow_unlocked`] for when this is appropriate.
#[must_use]
Expand Down Expand Up @@ -786,10 +806,7 @@ impl Minigraf {
self.inner.rules.clone(),
self.inner.functions.clone(),
);
executor.set_limits(
self.inner.options.max_derived_facts,
self.inner.options.max_results,
);
executor.set_limits(self.inner.options.derived_fact_limit());
executor.set_snapshot(snapshot);
executor
}
Expand All @@ -800,8 +817,8 @@ impl Minigraf {
/// while it is open do not change it. The cursor owns what it needs, so it
/// can outlive this handle and move to another thread. Queries the
/// streaming engine runs produce rows as they are pulled; others compute
/// the whole answer here, and `max_results` and `max_derived_facts` apply
/// exactly as they do to [`Minigraf::execute`].
/// the whole answer here, and `max_derived_facts` applies exactly as it
/// does to [`Minigraf::execute`].
///
/// # Errors
///
Expand Down Expand Up @@ -1491,10 +1508,7 @@ impl<'a> WriteTransaction<'a> {
self.inner.rules.clone(),
self.inner.functions.clone(),
);
executor.set_limits(
self.inner.options.max_derived_facts,
self.inner.options.max_results,
);
executor.set_limits(self.inner.options.derived_fact_limit());
return executor.execute(cmd);
}

Expand All @@ -1505,10 +1519,7 @@ impl<'a> WriteTransaction<'a> {
self.inner.rules.clone(),
self.inner.functions.clone(),
);
executor.set_limits(
self.inner.options.max_derived_facts,
self.inner.options.max_results,
);
executor.set_limits(self.inner.options.derived_fact_limit());
executor.execute(cmd)
}

Expand All @@ -1521,10 +1532,7 @@ impl<'a> WriteTransaction<'a> {
self.inner.rules.clone(),
self.inner.functions.clone(),
);
executor.set_limits(
self.inner.options.max_derived_facts,
self.inner.options.max_results,
);
executor.set_limits(self.inner.options.derived_fact_limit());
executor.execute(DatalogCommand::Rule(rule))
}

Expand Down Expand Up @@ -2277,6 +2285,7 @@ mod tests {
// ── file-backed: open_with_options custom threshold ───────────────────────

#[test]
#[allow(deprecated)] // every field, `max_results` too
fn test_open_with_options_custom_threshold() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("test.graph");
Expand Down Expand Up @@ -2755,9 +2764,7 @@ mod tests {
fn test_max_derived_facts_limit_enforced() {
// Recursive rule will derive many facts
// Low limit should trigger error
let low_opts = OpenOptions::default()
.max_derived_facts(5)
.max_results(1_000_000);
let low_opts = OpenOptions::default().max_derived_facts(5);
let db_low = Minigraf::in_memory_with_options(low_opts).unwrap();

// Add base edges in a chain
Expand All @@ -2784,9 +2791,7 @@ mod tests {
);

// Same query with higher limit should succeed
let high_opts = OpenOptions::default()
.max_derived_facts(100_000)
.max_results(1_000_000);
let high_opts = OpenOptions::default().max_derived_facts(100_000);
let db_high = Minigraf::in_memory_with_options(high_opts).unwrap();

db_high.execute("(transact [[:a :edge :b] [:b :edge :c] [:c :edge :d] [:d :edge :e] [:e :edge :f]])").unwrap();
Expand Down Expand Up @@ -2847,6 +2852,112 @@ mod tests {
);
}

/// A chain a → b → c → d → e → f and transitive `reachable`: 15 pairs
/// derived, never more than 5 new in one iteration.
fn chain_db(opts: OpenOptions) -> Minigraf {
let db = Minigraf::in_memory_with_options(opts).unwrap();
db.execute(
"(transact [[:a :edge :b] [:b :edge :c] [:c :edge :d] [:d :edge :e] [:e :edge :f]])",
)
.unwrap();
db.execute("(rule [(reachable ?x ?y) [?x :edge ?y]])")
.unwrap();
db.execute("(rule [(reachable ?x ?y) [?x :edge ?z] (reachable ?z ?y)])")
.unwrap();
db
}

const CHAIN_QUERY: &str = "(query [:find ?x ?y :where (reachable ?x ?y)])";

fn chain_code(db: &Minigraf, query: &str) -> Option<String> {
db.execute(query).err().map(|e| e.code().to_string())
}

#[test]
fn max_derived_facts_limits_the_total_not_one_iteration() {
// #553: the limit counts every fact derived, across iterations.
let db = chain_db(OpenOptions::default().max_derived_facts(15));
assert_eq!(chain_code(&db, CHAIN_QUERY), None, "15 derived facts fit");
let db = chain_db(OpenOptions::default().max_derived_facts(14));
let err = db.execute(CHAIN_QUERY).unwrap_err();
assert_eq!(err.code(), "INT-020");
assert!(
err.to_string().contains("Max derived facts (14) exceeded"),
"INT-020 names derived facts"
);
}

#[test]
#[allow(deprecated)]
fn max_results_is_an_alias_and_the_lower_limit_applies() {
let int020 = Some("INT-020".to_string());
let db = chain_db(OpenOptions::default().max_results(14));
assert_eq!(chain_code(&db, CHAIN_QUERY), int020, "alias alone");
let db = chain_db(OpenOptions::default().max_results(15));
assert_eq!(chain_code(&db, CHAIN_QUERY), None, "alias alone, enough");
let db = chain_db(
OpenOptions::default()
.max_derived_facts(14)
.max_results(1_000),
);
assert_eq!(
chain_code(&db, CHAIN_QUERY),
int020,
"max_derived_facts lower"
);
let db = chain_db(
OpenOptions::default()
.max_derived_facts(1_000)
.max_results(14),
);
assert_eq!(chain_code(&db, CHAIN_QUERY), int020, "max_results lower");
}

#[test]
#[allow(deprecated)]
fn max_derived_facts_alone_raises_the_limit_above_the_default() {
// The alias has no limit of its own unless set (#553).
let opts = OpenOptions::default().max_derived_facts(2_000_000);
assert_eq!(opts.derived_fact_limit(), 2_000_000);
assert_eq!(OpenOptions::default().derived_fact_limit(), 1_000_000);
assert_eq!(
OpenOptions::default()
.max_results(2_000_000)
.derived_fact_limit(),
1_000_000,
"the alias does not raise the limit"
);
}

#[test]
fn per_query_limits_replace_the_database_limit_and_the_lower_applies() {
let int020 = Some("INT-020".to_string());
let q = |opts: &str| format!("(query [:find ?x ?y :where (reachable ?x ?y) {opts}])");
let db = chain_db(OpenOptions::default().max_derived_facts(14));
assert_eq!(
chain_code(&db, &q(":max-derived-facts 15")),
None,
"replaces"
);
assert_eq!(
chain_code(&db, &q(":max-results 15")),
None,
"alias replaces"
);
let db = chain_db(OpenOptions::default());
assert_eq!(chain_code(&db, &q(":max-results 14")), int020, "alias");
assert_eq!(
chain_code(&db, &q(":max-derived-facts 15 :max-results 14")),
int020,
":max-results lower"
);
assert_eq!(
chain_code(&db, &q(":max-derived-facts 14 :max-results 15")),
int020,
":max-derived-facts lower"
);
}

// ── read-only handle drop must not modify the file ────────────────────────

#[test]
Expand Down
2 changes: 1 addition & 1 deletion src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1636,7 +1636,7 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[
(
ErrorCode::Int020,
"INT-020",
"evaluator: iteration or result limit exceeded: {}",
"evaluator: iteration or derived-fact limit exceeded: {}",
ErrorCategory::Internal,
),
(
Expand Down
Loading
Loading
0