8000 feat: :query-memory-limit, limit bind slots and PreparedQuery limit setters (#870) by adityamukho · Pull Request #874 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/binary-size.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
name: Check distribution binary size
runs-on: ubuntu-latest
env:
SIZE_LIMIT_BYTES: 1263616 # 1234 KiB — #277 error codes, #306 top-k, #723 ancestor join planning; see PHILOSOPHY.md §4
SIZE_LIMIT_BYTES: 1269760 # 1240 KiB — #277 error codes, #306 top-k, #723 ancestor join planning, #870/#861 query limit keywords; see PHILOSOPHY.md §4
steps:
- name: Checkout code
uses: actions/checkout@v4
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- **`OpenOptions::allow_overrides`** (#869, part of #862): the embedder lists, as `Overridable` variants, the query keywords that may override the database's settings. The default, `Overridable::DEFAULTS` (`MaxDerivedFacts`, `MaxResults`), allows every keyword that could do so before, so no query changes. A query using a keyword the list leaves out is refused with the new **AP 10000 I-025** before it runs, from `execute`, `query`, `prepare` and `WriteTransaction::execute`; the list replaces the defaults rather than adding to them, and is not stored in the file. `BrowserDb` takes no options and uses the defaults. `Overridable` is `#[non_exhaustive]`.

- **`:query-memory-limit`, limit bind slots and `PreparedQuery` limit setters** (#870, part of #862). `:query-memory-limit N` replaces `OpenOptions::query_memory_limit` for one query, up or down; `0` is allowed and lets only a query that holds no state run (QRY-012 otherwise). It is gated by the new `Overridable::QueryMemoryLimit`, which is **not** in `Overridable::DEFAULTS`, so a database refuses it with API-025 unless the embedder lists it; `BrowserDb` always refuses it. `:max-derived-facts`, `:max-results` and `:query-memory-limit` take a `$slot` filled with `BindValue::Val(Value::Integer(n))` (INT-014/INT-015 out of range as for a literal, API-024 for another type, API-022 when missing). `PreparedQuery::with_max_derived_facts` and `with_query_memory_limit` replace the database values for one statement; they are not gated, and an allowed keyword still overrides them. See [docs/CONFIGURATION.md](docs/CONFIGURATION.md). The v3 binary-size gate rises from 1,234 to 1,240 KiB (+672 B measured here; the rest is room for #861).

### Changed

- **INT-015 text** (#870): `{} must be a positive integer` is now `{} must be a non-negative integer`, because `:query-memory-limit` accepts 0. The code and when `:max-derived-facts`/`:max-results` raise it are unchanged.

- **Ordinary average** (#777): `(avg ?v)` no longer raises PRS-027. Grouped and ungrouped queries use the same numeric population and arithmetic as window `avg`: skip nil and nonnumeric values, sum exactly, round the total once to Float, then divide by the numeric count. Integer totals are not saturated. Results are Float, or nil for an existing group without numbers; no matching rows yields no rows. The total can overflow before division, and division rounds separately. Existing window behavior is unchanged. Retained exact-sum allocations for ordinary `sum` and `avg` are now charged to the query memory limit.

- **Align ordinary aggregate value handling with windows** (#789, v3.0.0).
Expand Down
12 changes: 6 additions & 6 deletions CLAUDE.md

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions PHILOSOPHY.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,10 @@ SQLite's success comes from a clear philosophy: be a library, not a server. Be s
The approved #723 ancestor-join optimization raises it to 1,234 KiB:
the measured increase is 2,720 bytes, avoiding a whole-attribute value
map for selective in-memory recursion without adding a storage index.
The #862 limit keywords raise it to 1,240 KiB: `:query-memory-limit`,
limit bind slots and the `PreparedQuery` setters (#870) measured +672
bytes, 312 over the old gate, and the limit leaves room for
`:max-iterations` (#861), which is built the same way.
The gate builds the x86_64 Linux CLI with cargo-dist's `dist` profile,
the one releases ship. From v3.0.0 `dist` keeps release's full LTO
rather than thin LTO, 7–31% smaller per target (#796; sizes per target
Expand Down
61 changes: 56 additions & 5 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ code are not gated.

| `OpenOptions` field | Default | Also changed by | Semantics of an override |
|---|---|---|---|
| `max_derived_facts` | 1,000,000 | query text: `:max-derived-facts N` | Replaces the database value for that query, up or down |
| `max_results` (deprecated) | no limit of its own | query text: `:max-results N` | Alone, replaces the database value; with `:max-derived-facts`, the lower applies |
| `query_memory_limit` | no limit | — (query keyword and `PreparedQuery` setter planned, #870) | — |
| `max_derived_facts` | 1,000,000 | query text: `:max-derived-facts N` or `$slot`; code: `PreparedQuery::with_max_derived_facts` | Replaces the database value for that query, up or down; a keyword overrides the setter |
| `max_results` (deprecated) | no limit of its own | query text: `:max-results N` or `$slot` | Alone, replaces the database value; with `:max-derived-facts`, the lower applies |
| `query_memory_limit` | no limit | query text: `:query-memory-limit N` or `$slot` (not in `DEFAULTS`); code: `PreparedQuery::with_query_memory_limit` | Replaces the database value for that query, up or down; a keyword overrides the setter |
| round limit | 1000, fixed | — (`OpenOptions::max_iterations` and `:max-iterations` planned, #861) | — |
| `transaction_chunk_facts` | 65,536 | — (per-transaction `TxOptions` planned, #871) | — |
| `synchronous` | `SyncMode::Full` | — (per-transaction `TxOptions` planned, #871) | — |
Expand All @@ -41,6 +41,7 @@ database's settings, as `Overridable` variants:
|---|---|---|
| `Overridable::MaxDerivedFacts` | `:max-derived-facts` | yes |
| `Overridable::MaxResults` | `:max-results` | yes |
| `Overridable::QueryMemoryLimit` | `:query-memory-limit` | no |

The default, `Overridable::DEFAULTS`, allows every keyword that could
override a setting before the list existed, so existing queries keep
Expand Down Expand Up @@ -78,5 +79,55 @@ let opts = OpenOptions::new().allow_overrides(
```

`Overridable` is `#[non_exhaustive]`: later releases add variants for new
keywords, such as `:query-memory-limit` (#870, not in `DEFAULTS`) and
`:max-iterations` (#861, in `DEFAULTS`).
keywords, such as `:max-iterations` (#861, in `DEFAULTS`).

`QueryMemoryLimit` is left out of `DEFAULTS` because the keyword can raise the
limit that keeps a query from exhausting memory. List it to let query text
change the limit:

```rust
use minigraf::{OpenOptions, Overridable};

let db = OpenOptions::new()
.query_memory_limit(64 << 20)
.allow_overrides(Overridable::DEFAULTS.iter().copied().chain([Overridable::QueryMemoryLimit]))
.open_memory()?;
db.execute("(query [:find ?x :where [?e :v ?x] :query-memory-limit 1048576])")?;
```

## Limit keywords

`:max-derived-facts N`, `:max-results N` and `:query-memory-limit N` take an
integer or a `$slot` that a prepared query fills with
`BindValue::Val(Value::Integer(n))`:

| Keyword | Range | Out of range | Other value |
|---|---|---|---|
| `:max-derived-facts`, `:max-results` | `n >= 1` | `INT-014` | `INT-015` (literal), `API-024` (bind value) |
| `:query-memory-limit` | `n >= 0` (bytes) | `INT-015` | `INT-015` (literal), `API-024` (bind value) |

A missing bind value is `API-022`. `Minigraf::execute` and `Minigraf::query`
refuse a query with an unbound slot (`API-010`).

`:query-memory-limit 0` lets a query run only if it holds no state: no hash
join, aggregate, sort, distinct set or rule relation, and no entity ids to
resolve. Uncheckpointed facts and returned entities need ids, so in practice
only a query over checkpointed facts that returns values runs under 0; any
other query fails at once with `QRY-012`.

## `PreparedQuery` setters

`PreparedQuery::with_max_derived_facts(n)` and
`PreparedQuery::with_query_memory_limit(bytes)` replace the database values
that were copied when the query was prepared, for that statement. They are
application code, so `allow_overrides` does not gate them. An allowed
keyword in the prepared text, literal or slot, still overrides the setter.
`usize::MAX` means no limit.

```rust
let pq = db
.prepare("(query [:find ?x ?y :where (reach ?x ?y)])")?
.with_max_derived_facts(10_000)
.with_query_memory_limit(16 << 20);
let rows = pq.execute(&[])?;
```
27 changes: 16 additions & 11 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ with no `CodedError` anywhere in its chain.
| INT-012 | Duplicate query option | Internal |
| INT-013 | Query option requires a value | Internal |
| INT-014 | Query option must be >= 1 | Internal |
| INT-015 | Query option must be a positive integer | Internal |
| INT-015 | Query option must be a non-negative integer | Internal |
| INT-016 | (or)/(or-join) nested inside (not)/(not-join) | Internal |
| INT-017 | Pseudo-attribute not valid in this position | Internal |
| INT-018 | Variable not bound by any outer/earlier clause | Internal |
Expand Down Expand Up @@ -3848,8 +3848,9 @@ clause, `not`/`not-join`/`or`/`or-join`, a rule invocation, or `and`). The

**Error text**: `duplicate {}`

**Cause**: A query specified `:max-derived-facts`, `:max-results`, `:limit`,
`:offset`, `:order-by` or `:distinct` more than once.
**Cause**: A query specified `:max-derived-facts`, `:max-results`,
`:query-memory-limit`, `:limit`, `:offset`, `:order-by` or `:distinct` more
than once.

**Resolution**:
- Specify each query option at most once.
Expand All @@ -3858,31 +3859,35 @@ clause, `not`/`not-join`/`or`/`or-join`, a rule invocation, or `and`). The

**Error text**: `{} requires a value`

**Cause**: A query option keyword (`:max-derived-facts` or `:max-results`)
was present with no following value.
**Cause**: A query option keyword (`:max-derived-facts`, `:max-results` or
`:query-memory-limit`) was present with no following value.

**Resolution**:
- Supply a positive integer value after the option keyword.
- Supply an integer value or a `$slot` after the option keyword.

### INT-014 Query option must be >= 1

**Error text**: `{} must be >= 1`

**Cause**: `:max-derived-facts` or `:max-results` was given a value less
than 1.
than 1, as a literal or as a prepared query's bind value.

**Resolution**:
- Use a value of 1 or greater.

### INT-015 Query option must be a positive integer
### INT-015 Query option must be a non-negative integer

**Error text**: `{} must be a positive integer`
**Error text**: `{} must be a non-negative integer`

**Cause**: `:max-derived-facts` or `:max-results` was given a non-integer
value.
value, or `:query-memory-limit` was given a negative or non-integer value, as
a literal or as a prepared query's bind value. (Before #870 the text said
"positive integer".)

**Resolution**:
- Supply a positive integer literal.
- Supply an integer literal or a `$slot`: at least 1 for
`:max-derived-facts` and `:max-results`, at least 0 for
`:query-memory-limit`.

### INT-016 (or)/(or-join) nested inside (not)/(not-join)

Expand Down
72 changes: 37 additions & 35 deletions docs/GRAMMAR.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,29 @@ EBNF has no lookahead, so a pest lookahead appears as a comment: `(* not followe

```ebnf
(* ── WORD-BOUNDARY HELPER ── *)
word-cont ::= letter | numeral | "?" | "_" | "-" | "/"
kw-find ::= ":find" (* not followed by word-cont *) (* token *)
kw-where ::= ":where" (* not followed by word-cont *) (* token *)
kw-with ::= ":with" (* not followed by word-cont *) (* token *)
kw-as-of ::= ":as-of" (* not followed by word-cont *) (* token *)
kw-valid-at ::= ":valid-at" (* not followed by word-cont *) (* token *)
kw-any-valid-time ::= ":any-valid-time" (* not followed by word-cont *) (* token *)
kw-max-derived-facts ::= ":max-derived-facts" (* not followed by word-cont *) (* token *)
kw-max-results ::= ":max-results" (* not followed by word-cont *) (* token *)
kw-limit ::= ":limit" (* not followed by word-cont *) (* token *)
kw-offset ::= ":offset" (* not followed by word-cont *) (* token *)
kw-distinct ::= ":distinct" (* not followed by word-cont *) (* token *)
kw-over ::= ":over" (* not followed by word-cont *) (* token *)
kw-rows ::= ":rows" (* not followed by word-cont *) (* token *)
kw-partition-by ::= ":partition-by" (* not followed by word-cont *) (* token *)
kw-order-by ::= ":order-by" (* not followed by word-cont *) (* token *)
kw-desc ::= ":desc" (* not followed by word-cont *) (* token *)
kw-asc ::= ":asc" (* not followed by word-cont *) (* token *)
kw-valid-from ::= ":valid-from" (* not followed by word-cont *) (* token *)
kw-valid-to ::= ":valid-to" (* not followed by word-cont *) (* token *)
reserved-head ::= ("not-join" | "not" | "or-join" | "or") (* not followed by word-cont *) (* token *)
and-head ::= "and" (* not followed by word-cont *) (* token *)
word-cont ::= letter | numeral | "?" | "_" | "-" | "/"
kw-find ::= ":find" (* not followed by word-cont *) (* token *)
kw-where ::= ":where" (* not followed by word-cont *) (* token *)
kw-with ::= ":with" (* not followed by word-cont *) (* token *)
kw-as-of ::= ":as-of" (* not followed by word-cont *) (* token *)
kw-valid-at ::= ":valid-at" (* not followed by word-cont *) (* token *)
kw-any-valid-time ::= ":any-valid-time" (* not followed by word-cont *) (* token *)
kw-max-derived-facts ::= ":max-derived-facts" (* not followed by word-cont *) (* token *)
kw-max-results ::= ":max-results" (* not followed by word-cont *) (* token *)
kw-query-memory-limit ::= ":query-memory-limit" (* not followed by word-cont *) (* token *)
kw-limit ::= ":limit" (* not followed by word-cont *) (* token *)
kw-offset ::= ":offset" (* not followed by word-cont *) (* token *)
kw-distinct ::= ":distinct" (* not followed by word-cont *) (* token *)
kw-over ::= ":over" (* not followed by word-cont *) (* token *)
kw-rows ::= ":rows" (* not followed by word-cont *) (* token *)
kw-partition-by ::= ":partition-by" (* not followed by word-cont *) (* token *)
kw-order-by ::= ":order-by" (* not followed by word-cont *) (* token *)
kw-desc ::= ":desc" (* not followed by word-cont *) (* token *)
kw-asc ::= ":asc" (* not followed by word-cont *) (* token *)
kw-valid-from ::= ":valid-from" (* not followed by word-cont *) (* token *)
kw-valid-to ::= ":valid-to" (* not followed by word-cont *) (* token *)
reserved-head ::= ("not-join" | "not" | "or-join" | "or") (* not followed by word-cont *) (* token *)
and-head ::= "and" (* not followed by word-cont *) (* token *)

(* ── EDN PRIMITIVES ── *)
boolean-lit ::= ("true" | "false") (* not followed by word-cont *) (* token *)
Expand Down Expand Up @@ -80,8 +81,8 @@ query-cmd ::= "(" "query" query-vector ")"
query-vector ::= "[" query-section+ "]"
query-section ::= find-section | where-section | as-of-section | valid-at-section
| any-valid-time-section | with-section | max-derived-facts-section
| max-results-section | limit-section | offset-section
| order-by-section | distinct-section
| max-results-section | query-memory-limit-section | limit-section
| offset-section | order-by-section | distinct-section

(* ── FIND SECTION ── *)
find-section ::= kw-find find-spec+
Expand Down Expand Up @@ -123,17 +124,18 @@ expr-arg ::= expr | set-lit | boolean-lit | nil-lit | float-lit | intege
| string-lit | bind-slot | keyword | variable

(* ── TEMPORAL / WITH SECTIONS ── *)
as-of-section ::= kw-as-of (integer-lit | string-lit | bind-slot)
valid-at-section ::= kw-valid-at (string-lit | kw-any-valid-time | bind-slot)
any-valid-time-section ::= kw-any-valid-time
with-section ::= kw-with variable+
max-derived-facts-section ::= kw-max-derived-facts integer-lit
max-results-section ::= kw-max-results integer-lit
limit-section ::= kw-limit (integer-lit | bind-slot)
offset-section ::= kw-offset (integer-lit | bind-slot)
distinct-section ::= kw-distinct
order-by-section ::= kw-order-by "[" order-entry+ "]"
order-entry ::= find-spec (kw-asc | kw-desc)?
as-of-section ::= kw-as-of (integer-lit | string-lit | bind-slot)
valid-at-section ::= kw-valid-at (string-lit | kw-any-valid-time | bind-slot)
any-valid-time-section ::= kw-any-valid-time
with-section ::= kw-with variable+
max-derived-facts-section ::= kw-max-derived-facts (integer-lit | bind-slot)
max-results-section ::= kw-max-results (integer-lit | bind-slot)
query-memory-limit-section ::= kw-query-memory-limit (integer-lit | bind-slot)
limit-section ::= kw-limit (integer-lit | bind-slot)
offset-section ::= kw-offset (integer-lit | bind-slot)
distinct-section ::= kw-distinct
order-by-section ::= kw-order-by "[" order-entry 6141 + "]"
order-entry ::= find-spec (kw-asc | kw-desc)?

(* ── RULE ── *)
rule-cmd ::= "(" "rule" rule-vector ")"
Expand Down
Loading
Loading
0