Lore evals measure whether memory changes agent outcomes, not just whether
search returns plausible rows. The default harness is the retrieval runner:
it reads committed YAML fixtures, loads them into fixture-backed services,
runs the production loadWakeUpData wake-up retrieval composition, scores
surfaced memory IDs, and writes a JSON artifact. The retrieval runner never
reads or writes a live Notion vault -- its fixture adapter replaces Notion
search.
The retrieval runner protects seven wake-up memory-surfacing sections:
taskMemories (the default), taskOnly, memories (recents),
relatedMemories, full context, pinnedContext, and inheritedMemories.
Each task selects the surface under test via the surface field. The fixture
adapter replaces Notion search, so the retrieval runner does not claim to
benchmark Notion vector ranking or the wake-up debug metrics emitted by the
observability path.
A second mode, the Notion-backed runner (--runner notion), points the
same suite YAML at an operator-configured sandbox project and exercises the
real retrieval stack -- Notion's rate limiter, hybrid search composition,
contains/semantic fusion, ranking. It performs live Notion reads scoped to
the project named via --project, but does not write to the vault. Use it
nightly or on PRs that touch retrieval composition; don't use it as the per-PR
hot-path gate.
| Need | Read |
|---|---|
| Runner overview, quick start, sandbox-vault discipline, shared vault registry | This file |
| Retrieval YAML schema, wake-up surfaces, task-eval suites, metrics, baseline refresh/drift rules | evals-suite-format.md |
| LongMemEval bench setup, env vars, ingestion/retrieval strategies, profile suites, safety gates, cleanup | evals-longmemeval.md |
| Phase 0/A/B decisions, rejected alternatives, historical benchmark selection evidence | archive/evals-phase-decisions.md |
npm run build
node dist/cli.js eval run evals/suites/lore-core.yamlThe starter retrieval suite writes a JSON artifact under evals/results/ by
default. Pass --out <path> to choose a deterministic artifact path for CI or
comparison runs:
node dist/cli.js eval run evals/suites/lore-core.yaml \
--out evals/results/lore-core-latest.json \
--jsonNine runners ship today. retrieval and notion share the same suite YAML
format and surface registry; task, bench, profile,
retrieval-quality, skill-retrieval, skill-use, and skill-agent each
have their own suite shape and scoring path because they score
agent-produced workspace state, LongMemEval-style multi-session recall, profile
taxonomy quality, ranked live-search quality, public corpus ranking, or
context-enabled answerability rather than wake-up surfaced memory ids:
| Runner | What it exercises | Where to use it |
|---|---|---|
retrieval (default) |
Fixture-backed loadWakeUpData with deterministic token-overlap search. No Notion calls. |
Per-PR CI; the inner-loop fast feedback. |
notion |
Real loadWakeUpData against LoreServices initialized from .lore.yaml. Hits Notion. |
Nightly CI; PRs that touch retrieval composition or ranking. |
task |
End-to-end agent run against a synthetic workspace, scored by deterministic verifiers. Shells out to codex exec. |
Nightly CI; opt-in PRs. Slow + model-cost; not the per-PR hot path. |
bench |
End-to-end LongMemEval bench: per-example ingest + recall through Lore MCP + judge. Hits Notion + OpenAI. | Operator-dispatched only (workflow_dispatch). A narrow conversation-memory reference, not the headline SkillRet/Lore or formation-transfer measurement. |
profile |
Deterministic profile-owned extraction artifacts scored against taxonomy, required-field, and hallucination expectations. No Notion or model calls. | Per-profile support suites and PRs that change profile manifests, schemas, or extraction contracts. |
retrieval-quality |
Labeled live-vault query → expected-memory checks through the real MemoryService.searchWithExplain path. Computes target rank, recall@1/5/10, NDCG@10, harmful@k, and MRR across product, RunTool AI, and REST keyword lanes. |
PRs and release checks that touch search transport or ranking. Read-only but live-vault-backed, so run deliberately against an operator-approved vault. |
skill-retrieval |
SkillRet memory-as-skill corpus ranking. The keyword lane is offline; the notion-ai lane imports SkillRet into a persistent Lore eval vault and scores Notion AI search through MemoryService.searchWithExplain. |
Public-scale substrate/candidate-recall diagnostic for memory-as-skill ranking. Does not measure representative Lore agent use. |
skill-use |
Offline no-context / oracle-context / retrieved-context / harmful-context answerability over labeled support sets. Computes success, answer accuracy, context sufficiency, harmful context rate, and retrieval gap to oracle. | Deterministic context-use proxy for memory-as-skill evidence. Does not measure live Notion behavior or a powered agent unless a separate answerer is plugged in. |
skill-agent |
SkillRet tasks as read-only seeded-vault Lore use. The agent receives Lore read instructions plus read-only search/expand tools, writes are blocked, and scoring separates tool use, target surfaced, target selected, and answer applied. | Representative SkillRet/Lore eval for fixed vaults. It measures whether an instructed agent can use an existing Lore vault, not whether Lore forms or mutates memories. |
The longitudinal lore-full-loop task condition is not a headline
formation-transfer metric when Phase A only inspects a workspace. A defensible
formation-transfer measurement needs separate gates for prior-task success,
formed-memory quality, retrieval, and Phase B use. Keep public skill
retrieval, verified skill use, production-vault retrieval, and
formation-transfer reports separate so a strong ranker-only result cannot be
misread as agent outcome lift.
Pass --runner notion --project <SandboxProject> to route a run through the
production retrieval stack (rate limiter, hybrid search, contains/semantic
fusion, ranking) against an operator-maintained sandbox vault. Per-task results
collapse to a single synthetic live-vault scenario; the runner uses the
helpful-memory expectation as the live-vault assertion because the live vault
IS the "memory is present" state by definition. The other ablations
(no-lore, empty-lore, noisy-memory, stale-memory) cannot be replayed
against a single live vault state without seed/cleanup infrastructure, so
notion-mode is structurally one-scenario-per-task. Lift / harm metrics therefore
stay null in notion mode. shouldNotSurface lists from the helpful-memory
expectation are forwarded into the live-vault assertion but are typically empty
in retrieval-mode helpful scenarios; if non-empty, those ids almost certainly
don't exist in the live vault and the assertion silently passes.
Suite syntax, task-mode verifier details, retrieval metrics, and baseline drift
rules live in evals-suite-format.md. LongMemEval
bench operations and the temporal-fidelity caveat that must travel with any
reported LongMemEval numbers live in evals-longmemeval.md.
retrieval-quality suites live under evals/retrieval-quality/. They are
live-vault, read-only suites: each case labels a natural-language paraphrase,
the expected memory id or primary/acceptable memory id sets, optional harmful
memory ids, and the project name that owns the memory. The runner resolves each
project by name, runs the real search path, and emits one result per lane.
npm run build
LORE_CONFIG_ROOT=~/Developer/Notion/Mail \
node dist/cli.js eval run evals/retrieval-quality/mail.yamlThe default lanes are:
| Lane | What it measures |
|---|---|
product |
The shipped lore search / lore-query action='search' path: semantic mode with the process's resolved feature flags. |
runtool-ai |
Semantic mode with RunTool search forced on, proving Notion AI search can rank the target. |
rest-keyword |
Contains-mode dataSources.query keyword search, retained as a comparison lane rather than a required pass condition. |
product and runtool-ai are required lanes by default. A required result
passes only when the target ranks #1 and the mechanism trace proves the
expected transport fired. For multi-target cases, a primary id must rank #1;
acceptable ids contribute to NDCG but do not satisfy the rank #1 gate. Harmful
ids surfaced in the top-10 fail the result and are reported as harmful@k. For
AI lanes, tools/run search must fire and REST client.search must not fire.
For rest-keyword, dataSources.query must fire and neither RunTool search
nor client.search may fire.
Suite limit must be at least 10, matching the largest reported recall@k
cutoff.
skill-retrieval suites live under evals/skill-retrieval/ and target
SkillRet-shaped public corpora. The committed skillret-smoke suite uses a
tiny in-repo corpus for deterministic checks; skillret-test targets the full
public SkillRet test split fetched from the pinned checksum manifest. The
keyword lane is an offline lexical control. The notion-ai lane is a
substrate recall diagnostic: it imports each SkillRet skill as a procedure memory in a
persistent Notion-backed eval vault, adds SkillRet taxonomy topics, topic keys,
tags, keywords, and body metadata, stores the skill-id to memory-id mapping in
a local manifest, and then scores returned memories from Notion AI search.
It does not measure an LLM agent deciding to use Lore or applying a retrieved
memory.
npm run build
node dist/cli.js eval skill-retrieval fetch skillret
node dist/cli.js eval run evals/skill-retrieval/skillret-test.yamlRun the Notion AI lane against the registered persistent SkillRet eval vault:
npm run build
node dist/cli.js eval skill-retrieval fetch skillret
mkdir -p /tmp/lore-eval-vaults/skillret-dev-eval
node dist/cli.js eval vaults show skillret-dev-eval --config \
> /tmp/lore-eval-vaults/skillret-dev-eval/.lore.yaml
NOTION_ENV=dev \
LORE_CONFIG_ROOT=/tmp/lore-eval-vaults/skillret-dev-eval \
node dist/cli.js eval skill-retrieval import \
evals/skill-retrieval/skillret-notion-ai.yaml \
--yes \
--create-project \
--parallel 3
NOTION_ENV=dev \
LORE_CONFIG_ROOT=/tmp/lore-eval-vaults/skillret-dev-eval \
LORE_EVAL_SKILLRET_REAL=1 \
node dist/cli.js eval run evals/skill-retrieval/skillret-notion-ai.yamlThe committed skillret-notion-ai suite is bound to the registered
skillret-dev-eval vault page id, so import and run fail if the active
.lore.yaml points at a different vault. The suite imports rows into taxonomy
topics while leaving search project-scoped (searchTopicScoped: false) so the
AI lane can retrieve across the full SkillRet corpus. Full-corpus live runs are
paced by queryDelayMs to avoid intentionally saturating the Notion rate
limit. The import command is idempotent against both the generated manifest path
declared by
suite.notion.importManifestPath and the live vault's Promotion Source Key
values. Re-running it skips already-imported skills whose content and metadata
still match only after confirming the live Notion row still exists, repairs
matching-source rows whose rendered metadata changed, and rebuilds entries from
existing live memories before creating anything if the local manifest is
missing. It fails if the suite points at a manifest for a different corpus
revision, split, document field set, project, or root topic.
The import path accepts older document transforms so it can repair them in
place, but limited imports refuse to mark a legacy manifest current. The
notion-ai run path requires the manifest to match the current transform and
cover every skill required by the selected queries' qrels before scoring.
SkillRet source fields longer than 60,000 characters are capped in the
rendered Notion memory body with an explicit truncation marker. The manifest
keeps the full-source hash, the rendered-content hash, and per-skill
truncatedFields metadata so the fidelity tradeoff is auditable. The manifest
contains live Notion page ids and stays under
evals/skill-retrieval/manifests/, which is intentionally ignored.
Smoke-suite shape:
version: 1
runner: skill-retrieval
name: skillret-smoke
corpus:
kind: skillret
root: fixtures/skillret-mini
split: test
document:
textFields: [name, description, skill_md]
queries:
limit: 200
seed: smoke-v1
lanes:
- keyword
k: [1, 5, 10]
retrieval:
limit: 10The notion-ai lane requires a retrieval window of at most 25, matching the
RunTool search return cap. It also records a mechanism trace and counts
mechanism failures when RunTool AI search did not dispatch as expected. Report
this runner as public-scale candidate recall only. It does not show that the
agent used the memory correctly, that Lore formed the memory autonomously, or
that the normal Lore instructions caused an agent to query the vault.
skill-agent suites live under evals/skill-agent/. They are the
representative SkillRet/Lore eval path for fixed evaluation vaults. The vault is
a read-only seeded fixture: agents may search, recall, and expand existing
memories, but they may not create, update, archive, approve, reject, promote,
mine, autosave, or otherwise mutate Lore state. Any write attempt is blocked by
the harness and counted in the artifact.
npm run build
NOTION_ENV=dev \
LORE_CONFIG_ROOT=/tmp/lore-eval-vaults/skillret-dev-eval \
LORE_EVAL_BENCH_REAL=1 \
node dist/cli.js eval run evals/skill-agent/skillret-readonly-agent.yamlThe runner reuses the SkillRet import manifest declared by the linked
skill-retrieval suite. It validates that the active Lore config points at the
suite's registered vault, that the manifest covers every selected query's qrels,
and that the selected agent conditions run against the same fixed corpus
revision.
The default committed suite uses:
no-lore: baseline model/task prior with no Lore tools.tool-driven-lore: headline condition. The agent receives read-only Lore instructions and read-only search/expand tools. Search uses the agent-facinglore-query action='search'behavior, including planned semantic query variants unless query planning is disabled.oracle-context: diagnostic ceiling where the target procedure body is injected directly.
Artifacts score separate layers so a retrieved-but-unused memory is not counted as Lore success:
toolUse: a successful read-only Lore tool call happened.targetSurfaced: an expected SkillRet memory appeared in tool results.targetExpanded: the agent read the expected memory body.targetSelected: the final structured answer cited the expected memory or SkillRet id.answerApplied: the answer used the selected target evidence.writeAttemptsBlocked: attempted write-path calls rejected by the harness.
Keep skill-agent reports separate from skill-retrieval reports. Raw Notion
search recall is useful for diagnosing the substrate, but the headline Lore
claim is read-only agent use over a stable seeded vault.
skill-use suites live under evals/skill-use/. They measure whether selected
memory evidence is sufficient to answer a task under explicit conditions:
no-context, oracle-context, retrieved-context, and optionally
harmful-context. Each task defines one or more support sets; satisfying any
support set makes the context sufficient. Harmful evidence ids model stale or
contradictory memories that should not enter retrieved context.
npm run build
node dist/cli.js eval run evals/skill-use/smoke.yamlSmoke-suite shape:
version: 1
runner: skill-use
name: skill-use-smoke
corpus:
documentsPath: fixtures/basic/documents.jsonl
retrieval:
lane: keyword
limit: 1
k: [1]
thresholds:
minOracleAccuracy: 1
maxNoContextAccuracy: 0
minRetrievedOracleRatio: 1
maxHarmfulContextRate: 0
conditions:
- no-context
- oracle-context
- retrieved-context
requiredConditions:
- oracle-context
- retrieved-context
tasks:
- id: calendar-retention
prompt: What retention period should the calendar export use?
answer:
accepted: [30 days]
mode: contains
supportSets:
- id: primary
documentIds: [mem-calendar-retention]
harmfulEvidenceIds:
- mem-calendar-retention-staleThe default answerer is a deterministic evidence proxy: it answers only when the supplied context satisfies a support set and contains no harmful evidence. That makes the suite a stable context-sufficiency gate, not a powered-agent benchmark. A future powered answerer can reuse the same suite contract, but its results must be reported separately because model choice, prompting, retries, and judge behavior become part of the measurement.
Notion-mode suites typically live alongside the fixture suite but reference
real Notion page ids in shouldSurface. The committed lore-core suite
uses synthetic ids (e.g. decision/auth-model) and will not match anything on
a real vault -- author a parallel suite under evals/suites/<name>-sandbox.yaml
whose shouldSurface lists the actual ids from the operator's sandbox vault.
The CLI rejects project names that don't word-boundary-match one of
sandbox, eval, test, scratch, staging, dev, or playground unless
LORE_EVAL_NOTION_ALLOW_PRODUCTION=1 is set in the environment. Notion-mode
reads are technically read-only, but pointing the runner at a production vault
still hammers per-token rate limits and surfaces misleading "drift" against
fixture-shaped expectations -- the env-var gate forces an explicit decision.
The boundary is a regex \b word boundary, so Widget-staging, Eval-Project,
and dev-vault all pass, but camel-case names without a separator
(EvalProject, TestVault) reject -- the regex sees those as embedded
substrings rather than standalone markers. If your sandbox uses a camel-case
convention, either add a separator (Eval-Project) or set the env var to opt
in.
Shared live eval vault locators live in
evals/vaults.yaml. The registry may commit Notion
workspace ids and vault page ids because they are access locators, not bearer
credentials; operators still need matching Notion auth and page access. Never
commit tokens or generated .lore.yaml files.
List the registered vaults:
npm run build
node dist/cli.js eval vaultsMaterialize a local config root for the seeded dev sandbox:
mkdir -p /tmp/lore-eval-vaults/lore-dev-sandbox
node dist/cli.js eval vaults show lore-dev-sandbox --config \
> /tmp/lore-eval-vaults/lore-dev-sandbox/.lore.yaml
node dist/cli.js eval vaults show lore-dev-sandbox --envLive evals that are measuring cost should opt into Lore's local cost ledger in that sandbox config root before running:
costTracking:
enabled: true
ledgerPath: /tmp/lore-eval-vaults/lore-dev-sandbox/eval-costs.jsonlRun the live longitudinal suite against that vault by pointing
LORE_EVAL_LONGITUDINAL_CONFIG_ROOT at the generated config root, exporting the
vault selector env, and opting into the live task runner:
NOTION_ENV=dev \
NOTION_WORKSPACE_ID=415fc269-e68f-4da0-b3e3-b1273b741a7f \
LORE_EVAL_LONGITUDINAL_CONFIG_ROOT=/tmp/lore-eval-vaults/lore-dev-sandbox \
LORE_EVAL_TASK_REAL=1 \
LORE_EVAL_LONGITUDINAL_REAL=1 \
LORE_EVAL_LONGITUDINAL_S
5AC5
ANDBOX_PROJECT="Eval Sandbox" \
node dist/cli.js eval run --runner task evals/task-suites/longitudinal.yamlThe initial seeded-vault validation recorded in the registry is a smoke run,
not a statistically powered benchmark: it ran all 4 scenarios in the committed
longitudinal suite across both conditions (8 condition runs total). On
2026-05-14, no-memory passed 3/4 scenarios, lore-full-loop passed 4/4
scenarios, and the observed success-rate lift was +25 percentage points.
An OSS direction-setting pilot lives at
evals/task-suites/longitudinal-github-cli-pilot.yaml. It materializes GitHub
CLI (cli/cli) at pinned commit
9a593ce81b593dee752cc11737d1a3ef768e52b3 and runs five scenarios across
no-memory, seeded-lore, and lore-full-loop. Treat this suite as pilot
instrumentation only: it is meant to estimate verifier stability, paired
discordance, token cost, and runtime before choosing the powered sample size for
the 15 percentage point MDE benchmark. It is not a statistically powered result.
Its source-controlled seed corpus lives at
evals/vault-seeds/github-cli-pilot.yaml; it models GitHub CLI as one project
with component-level topics, not as a monorepo.
A larger PR-derived and generalized candidate bank lives at
evals/task-suites/longitudinal-github-cli-powered-candidates.yaml, backed by
evals/vault-seeds/github-cli-powered.yaml. It is a scenario pool, not a fixed
balanced dataset. As model capability moves, scenarios can be demoted from hard
to medium or medium to easy, so the pool is expected to have more easy scenarios
than medium scenarios and fewer hard scenarios than medium scenarios. Powered
runs should draw a deterministic random sample from that pool and record the
seed rather than assuming every authored scenario belongs in the headline
measurement. Those candidate scenarios are not independent statistical units:
many intentionally cluster around the same source memory or source pull
request. Do not publish it as the powered benchmark result until candidates
have been validated for prompt bounds, no-op baseline failure, verifier
stability, runtime, and cost.
The primary powered comparison is lore-full-loop vs. no-memory, because it
measures Lore's end-to-end formation/retrieval/use loop. seeded-lore is an
ablation for separating vault availability from memory formation quality.
The 2026-05-27 GitHub CLI checkpoint is recorded in
docs/evals-github-cli-powered-20260527.md.
lore-full-loop scenarios should validate durable cross-session learnings that
Lore is designed to capture, such as decisions, conventions, gotchas,
workarounds, and explicit future follow-ups. They should not require generic
project facts or Phase B-only feature details to be captured unless Phase A
makes that durable follow-up explicit. Before a powered run, pre-register the
cluster/capping rule for repeated source memories and PRs, the multiplicity
treatment for secondary comparisons, and the culling rules used to promote
candidates. Candidate culling must be blind to condition deltas: remove tasks
only for objective validity failures such as no-op pass, verifier ambiguity,
prompt out-of-bounds behavior, flake rate, runtime, or cost.
Phase B starts from a clean rematerialization of the original workspace source,
not the Phase A workspace. Phase A edits are preserved only as transcript and
patch evidence for adjudication; they must not become implicit implementation
state for no-memory, seeded-lore, or lore-full-loop. Any cross-session
benefit must therefore flow through seeded context or Lore formation/wake-up.
Keep raw and adjudicated measurements separate. If a run fails because of
harness validation, infrastructure, or an over-narrow verifier, preserve the raw
artifact and classify the row before scoring. A row marked harness-error is
not evidence of real-world agent performance and must be rerun or excluded from
the headline measurement. If the transcript, diff, verifier output, and command
results objectively prove that an agent satisfied the intended scenario under a
repaired verifier, an adjudicated score may be reported without rerunning that
condition. Final reports should include raw, adjudicated, and exclusion-only
sensitivity views.
For lore-full-loop, formation and wake-up phase errors are product failures
unless they were caused by external infrastructure or harness setup. Expected
context matching is diagnostic, not an outcome override: missing expected memory
can explain a verifier failure, but it does not turn a verifier-passing task
into a failure. Longitudinal task artifacts write transcript and patch sidecars
next to the JSON result so reviewers can adjudicate verifier false negatives
after temporary workspaces are cleaned up.
npm run build
NOTION_ENV=dev \
NOTION_WORKSPACE_ID=415fc269-e68f-4da0-b3e3-b1273b741a7f \
LORE_EVAL_LONGITUDINAL_CONFIG_ROOT=/tmp/lore-eval-vaults/lore-dev-sandbox \
LORE_EVAL_TASK_REAL=1 \
LORE_EVAL_LONGITUDINAL_REAL=1 \
LORE_EVAL_LONGITUDINAL_SANDBOX_PROJECT="Eval Sandbox" \
node dist/cli.js eval run --runner task \
evals/task-suites/longitudinal-github-cli-pilot.yaml \
--out evals/results/longitudinal-github-cli-pilot.jsonFor a powered pool run, select a reproducible sample by difficulty bucket. The runner hashes the seed and scenario IDs, fails fast if a bucket is undersized, shuffles the selected scenarios into a deterministic cross-bucket run order so partial artifacts are not front-loaded with easy cases, and records the selected scenario IDs in the artifact:
node dist/cli.js eval run --runner task \
evals/task-suites/longitudinal-github-cli-powered-candidates.yaml \
--sample easy=25,medium=15,hard=10 \
--sample-seed 2026-05-27-nightly \
--parallel 16 \
--cost-kill-switch-usd 1500 \
--out evals/results/longitudinal-github-cli-powered-sample.jsonThe run writes Codex JSONL transcript sidecars next to the artifact, in
a directory derived from the output filename, such as
evals/results/longitudinal-github-cli-powered-sample-transcripts/. Use those
sidecars to inspect prompts, event streams, tool activity, and per-turn usage
evidence without bloating the main JSON artifact.
The GitHub CLI longitudinal suites set costKillSwitchUsd: 1500. This is an
overnight guard between condition runs: once observed priced cost reaches the
limit, the runner stops before launching another agent subprocess and persists
a partial artifact with a termination block. If agent usage is unavailable or
unpriced, the guard stops fail-closed instead of treating that work as free.
Operators can override the suite value for a run with
--cost-kill-switch-usd <n>.
Cost is split across two surfaces. Primary Phase A/Phase B agent spend is
recorded in the eval artifact under each phase's cost object using Codex
turn.completed usage when available. Lore-owned spend is recorded in the
local cost ledger: post-session mining rows use
eval.mining.background_model, and MCP/Notion activity appears as
mcp.invocation rows. Inspect the Lore-owned side from the sandbox config root:
LORE_REPO=$PWD
cd /tmp/lore-eval-vaults/lore-dev-sandbox
node "$LORE_REPO/dist/cli.js" costs summary --since 24h
node "$LORE_REPO/dist/cli.js" costs export --since 24h --format csv \
> /tmp/lore-eval-vaults/lore-dev-sandbox/eval-costs.csvAfter a pilot artifact exists, estimate the powered run size, budget fit, paired primary-agent token deltas, and paired runner phase elapsed deltas:
node dist/cli.js eval longitudinal plan \
evals/results/longitudinal-github-cli-pilot.json \
--mde 0.15 \
--power 0.8 \
--alpha 0.05 \
--budget-usd 1000The planner only projects cost from measured artifacts when every condition run
has complete primary-agent cost coverage. If any agent phase lacks priced usage,
the cost projection is reported as unavailable unless the operator supplies
--cost-per-condition-run-usd. The elapsed deltas are runner phase elapsed
time, not pure model latency: Phase A elapsed includes formation work, Phase B
elapsed includes verifier execution, and wake-up retrieval is outside the Phase
B timer.
Because the live sandbox vault state is not committed, runners against it are not deterministic the way fixture runs are; treat notion-mode CI as a coarser signal that catches retrieval-stack regressions across the rate limiter, search composition, and Notion-side ranking -- not the per-row precision the fixture suite measures. Single-task pass/fail flips between adjacent runs are expected noise from Notion's hybrid search; only aggregate trends and sustained per-task regressions matter.