8000 bench: add foreign runtime coverage for #776 P16 by adityamukho · Pull Request #832 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]
Skip to content

bench: add foreign runtime coverage for #776 P16 - #832

Merged
adityamukho merged 2 commits into
v3from
bench/776-p16-bindings
Nov 3, 2026
Merged

bench: add foreign runtime coverage for #776 P16#832
adityamukho merged 2 commits into
v3from
bench/776-p16-bindings

Conversation

@adityamukho
@adityamukho adityamukho commented Nov 3, 2026 •
edited
Loading
Copy link
Copy Markdown
Collaborator

Adds the core routes/reporting for #776 P16, G15. Real Python, browser, Node and C runtimes use independent Bencher testbeds; native Criterion history is not used as FFI evidence. The four binding PRs must land before these routes.

  • IDs: binding/{python,browser,node,c}/{prepared_execute/1k,cursor_batch_1/1k,cursor_batch_256/1k,set_roundtrip/2,predicate/1k,aggregate/1k}; browser also binding/browser/{indexeddb_write/1k,indexeddb_reopen/1k} (26 total).
  • Route: bench / binding_<runtime> in the bindings matrix, selecting the entire fixed shortlist; v2/main checkouts skip build/upload. Manual phase B can set binding-ref=bench/776-p16-bindings for all four binding PR branches.
  • Builds resolve the exact measured core checkout, retaining core/binding/workflow hashes and Cargo metadata. Required IDs, positive finite metrics, execution and complete uploads are checked; existing repeat-alert policy is preserved.
  • Dedicated testbeds: binding-<runtime>-ubuntu24-v1. Reserve 5–15 minutes per initial binding job plus build/setup, unverified. No timed local runs or workflow dispatches; phase B will collect Actions timing/upload evidence serially.
  • Java/JVM, Android, Swift, WASI and remaining conversion/persistence paths have explicit manual coverage/release-sign-off dispositions in docs/BINDING_BENCHMARKS.md.

Validation: 17 reporting tests, actionlint, shell syntax and formatting checks passed. The binding entry points passed untimed --test checks; exact results and review outcome will be posted in a follow-up comment. Binary-size gate, core behavior and release dependency pins are unchanged. Existing benchmark PRs did not update CHANGELOG, so this PR follows that precedent.

Refs #776.

Binding dependencies (merge first): Python #17, browser #21, Node #21, C #14.

Verified review fixes: untimed failures preserve their exit status; experimental binding refs use separate history and skip production regression issues. Binding PRs move checksum validation outside timing, force the Node local artifact, decode C sets and free Wasm handles explicitly. All 26 cases passed untimed after those fixes.

🤖 Generated with Codex, orchestrated by Claude Code

This was referenced Nov 3, 2026
Merged
Merged
Merged
Merged
@codecov
codecov Bot commented Nov 3, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@adityamukho
adityamukho commented Nov 3, 2026
Copy link
Copy Markdown
Collaborator Author

Phase-A untimed evidence after the independent reviews and fixes:

The four host entry points (no Criterion regex applies) each execute their whole shortlist once:

  • Python: target/bench-venv/bin/python bench/nightly.py --test
  • Browser/Node: node bench/nightly.mjs --test
  • C: target/p16/nightly --test
ok: binding/python/prepared_execute/1k
ok: binding/python/cursor_batch_1/1k
ok: binding/python/cursor_batch_256/1k
ok: binding/python/set_roundtrip/2
ok: binding/python/predicate/1k
ok: binding/python/aggregate/1k
ok: binding/browser/prepared_execute/1k
ok: binding/browser/cursor_batch_1/1k
ok: binding/browser/cursor_batch_256/1k
ok: binding/browser/set_roundtrip/2
ok: binding/browser/predicate/1k
ok: binding/browser/aggregate/1k
ok: binding/browser/indexeddb_write/1k
ok: binding/browser/indexeddb_reopen/1k
ok: binding/node/prepared_execute/1k
ok: binding/node/cursor_batch_1/1k
ok: binding/node/cursor_batch_256/1k
ok: binding/node/set_roundtrip/2
ok: binding/node/predicate/1k
ok: binding/node/aggregate/1k
ok: binding/c/prepared_execute/1k
ok: binding/c/cursor_batch_1/1k
ok: binding/c/cursor_batch_256/1k
ok: binding/c/set_roundtrip/2
ok: binding/c/predicate/1k
ok: binding/c/aggregate/1k

All 26 IDs passed using local release builds, also with a temporary patch resolving the exact P16 core source. The core shell build helper passed end to end for Node, including Cargo metadata source verification. No timed local run, SSH or benchmark dispatch was performed.

Validation: 17 reporting tests pass on both branches, actionlint passes, shell syntax/format checks pass, C compiles with -Wall -Wextra -Werror, and the two core workflow/reporting changes are byte-identical.

All six PRs received codex exec -m gpt-6-astra -s read-only ... < /dev/null reviews. Seven unique verified findings were fixed: checksum reductions moved outside timing; executable Python build instructions; forced local Node addon; actual C set decoding; prerequisite pipefail; separate experimental binding histories; deterministic Wasm teardown. No findings were rejected. Untimed checks and CI were rerun after fixes. Normal CI: green on the latest head (20 passing checks; gh pr checks --watch exited 0).

Routes: bindings matrix, bench / binding_{python,browser,node,c}, entire fixed shortlists; main/v2 checkouts skip before build/upload. Experimental binding refs use binding-experiment/<core-ref>/<binding-ref> histories and skip production issue reporting. Merge the four binding PRs before the core routes; phase B may select binding-ref=bench/776-p16-bindings in its serial dispatch. Dedicated testbeds and the 5–15 minute/job plus setup allowance remain unverified until phase B.

Java/JVM, Android, Swift, WASI and other deliberately unmeasured paths require release sign-off as documented in docs/BINDING_BENCHMARKS.md. This does not claim #776 can close.

adityamukho and others added 2 commits November 3, 2026 18:24
@adityamukho @codex
ci(bench): cover foreign runtime costs (Refs #776)
3ff4b9d 
Add dedicated Python, browser, Node and C jobs with exact core provenance, complete JSON metric checks and isolated Bencher testbeds.

Co-Authored-By: Codex <noreply@openai.com>
@adityamukho @codex
ci(bench): preserve verification and isolate binding experiments (Refs …
fef01c5 
…#776)

Apply verified review findings: preserve failed untimed checks and keep experimental binding refs out of nightly history and issue reporting.

Co-Authored-By: Codex <noreply@openai.com>
@adityamukho
adityamukho force-pushed the bench/776-p16-bindings branch from 5ffc7a3 to fef01c5 Compare November 3, 2026 12:54
@adityamukho
adityamukho merged commit 0394c90 into v3 Nov 3, 2026
20 checks passed
@adityamukho
adityamukho deleted the bench/776-p16-bindings branch November 3, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@adityamukho
0