Repository navigation
fix(engine): probe joins on a bound value through AVET (#721) - #722
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
A join whose inner pattern shares only its value with the rows so far, under a constant attribute, was a HashJoin over every fact of the attribute. In a rule's delta run it was rebuilt every round: since #700 the ancestors shape (boss :leaf ?b) took 1.8 s at 1M facts instead of 2 ms. It is now an IndexNestedLoopJoin on the value: per outer row, the AVET ranges of the values that join-equal it (a keyword and the ref to the entity it names, either way; 0.0 and -0.0), and the attribute's pending facts with its join key, read once per scan with the shadow set. The join checks every key, so answers are the hash join's. A pattern that reads a relation, or has a constant entity, keeps its hash join. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
…ty's keyword is unknown (#721) The pending facts and shadow set of a value probe are read once per query, not per scan, so a rule run in every round reads them once. An entity cell must also find the keyword that names it, and a keyword's entity is a hash of its name: no index leads back. The query records the keywords it makes entities from (rule heads, and the keyword constants its rule calls name, the magic seeds); a probe by an entity without one, on an attribute with committed keyword values, turns the join into the hash join the planner chose before, on the side it named. recursive_ancestors (bench machine, p50): 1.1 ms at 100K, 1.4 ms at 1M, 1.8 ms at 10M (v3: 101 ms, 1.8 s, 35 s). join_3_selective 145 ms -> 0.2 ms at 1M; join_3_scan unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
The nightly recursion group gains the ancestors shape, whose rounds join on a value, next to fanout. CLAUDE.md and TEST_COVERAGE describe value probes; the plan records what measuring changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
…e list (#721) The attribute's pending facts are keyed by encode_value, so building the per-query map hashes no keyword names; each probe looks up the values that join-equal its cell, the same list that gives the committed AVET ranges. An entity cell with no recorded keyword falls back to the hash join when the attribute has pending keyword values too. CHANGELOG entry with the envelope and Criterion numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW
This was referenced Oct 23, 2026
Closed
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the recursion regression in #721 (issue closed by hand after merge; closing keywords do not fire on
v3). Plan:docs/superpowers/plans/2026-10-23-721-value-probes.md.Problem
Since #700, the delta run of
(boss :leaf ?b)(after the magic-set rewrite:(boss ?y ?z)then[?x :manager ?y]) joins:manageron its value. That join was aHashJoinover every:managerfact, rebuilt every round:recursive_ancestors1.8 s at 1M facts and 35 s at 10M instead of 2–3 ms.Change
JoinKind::ProbeValue { key, build_inner }: a join whose inner pattern has a constant attribute, a non-constant entity and a bound value (and reads no relation) probes per outer row: the AVET ranges of the values that join-equal the row's value, and the attribute's uncheckpointed facts with those values. The join checks every key, so answers are the hash join's.scan::candidates): a keyword and a ref to the entity it names, either way;0.0and-0.0; nothing for NaN.IdSpace::probes): the attribute's pending facts (byencode_value) and the committed triples they shadow, so a rule run in every round, or a body run per row, reads them once.recursion/ancestorsjoins the nightlyrecursiongroup.Two designs were measured and dropped (plan, deviations 5–8): per-scan reads (reread every round) and a reverse keyword map read from the attribute's keyword range (
join_3_scan15.6 s and 681 MiB at 10M).Measurements (bench machine,
v3dfa2041 vs this branch, parallel on separate sockets, two rounds)#394 envelope, p50:
recursive_ancestorsjoin_3_selectiveThe point shapes (
point_entity,attribute_value,join_2) are bimodal per process (about 18 or 54 µs) on both builds; six fresh processes per build gave the same distributions. Onerecursive10M run hit the 60 s cap (both builds take 56–59 s).Criterion
recursion(in memory):ancestors/w10_d497.7 → 15.1 ms,chain/depth_10041.4 → 20.8 ms,chain/depth_100.68 → 0.53 ms,fanout/w10_d322.7 → 20.8 ms. In memory every fact is pending, and the first probe reads the attribute's pending facts once per query (12 ms of the 15 ms for 11,110 facts); before #700ancestorswas 0.27 ms. A pending value index would remove that read; #545 (decision 3) left it out. Tracked in #723.Release binary: 1,154 KiB (gate 1,200).
Tests
rule_delta_probes_a_bound_value: the ancestors delta run probes by value and hashes no stored attribute; answers in memory and on a half-checkpointed file.value_probe_reads_only_the_probed_keys: operator counters, through a pending retraction of a committed fact.value_probe_matches_as_joins_do: keyword and ref from either side, an entity cell with no recorded keyword (the fallback),0.0/-0.0, long strings,:as-of,:valid-at; in memory, checkpointed, half pending.value_probe_falls_back_mid_batch: a fallback after probes by keyword, mid-batch, over several batches: every row joins once.planner_chooses_the_joinupdated (a value join is now a probe).tests/answers/), plan independence and model-based tests unchanged. Bench machine: 1,701 passed / 12 ignored; withtest-hooks1,717 / 13.Codex review, two rounds: (1) probe cache charge kept after
IdSpace::release— not a defect,Heldreleases on drop; (2) fallback loses rows — not a defect (rows before the miss were emitted by their probes; the table or the stream covers the rest and later batches), now covered byvalue_probe_falls_back_mid_batch.🤖 Generated with Claude Code
https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW