8000 ci: schedule P16 binding benchmarks from main by adityamukho · Pull Request #833 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]
Skip to content

ci: schedule P16 binding benchmarks from main - #833

Merged
adityamukho merged 2 commits into
mainfrom
ci/776-p16-main
Nov 3, 2026
Merged

ci: schedule P16 binding benchmarks from main#833
adityamukho merged 2 commits into
mainfrom
ci/776-p16-main

Conversation

@adityamukho
@adityamukho adityamukho commented Nov 3, 2026 •
edited
Loading
Copy link
Copy Markdown
Collaborator

Identical main companion to the v3 P16 workflow/reporting PR. Main owns the nightly schedule; its 14:00 UTC checkout runs v3 binding cases, and its 02:00 main checkout skips them. Merge binding code first, then both core workflow PRs.

Adds the core routes/reporting for #776 P16, G15. Real Python, browser, Node and C runtimes use independent Bencher testbeds; native Criterion history is not used as FFI evidence. The four binding PRs must land before these routes.

  • IDs: binding/{python,browser,node,c}/{prepared_execute/1k,cursor_batch_1/1k,cursor_batch_256/1k,set_roundtrip/2,predicate/1k,aggregate/1k}; browser also binding/browser/{indexeddb_write/1k,indexeddb_reopen/1k} (26 total).
  • Route: bench / binding_<runtime> in the bindings matrix, selecting the entire fixed shortlist; v2/main checkouts skip build/upload. Manual phase B can set binding-ref=bench/776-p16-bindings for all four binding PR branches.
  • Builds resolve the exact measured core checkout, retaining core/binding/workflow hashes and Cargo metadata. Required IDs, positive finite metrics, execution and complete uploads are checked; existing repeat-alert policy is preserved.
  • Dedicated testbeds: binding-<runtime>-ubuntu24-v1. Reserve 5–15 minutes per initial binding job plus build/setup, unverified. No timed local runs or workflow dispatches; phase B will collect Actions timing/upload evidence serially.
  • Java/JVM, Android, Swift, WASI and remaining conversion/persistence paths have explicit manual coverage/release-sign-off dispositions in docs/BINDING_BENCHMARKS.md.

Validation: 17 reporting tests, actionlint, shell syntax and formatting checks passed. The binding entry points passed untimed --test checks; exact results and review outcome will be posted in a follow-up comment. Binary-size gate, core behavior and release dependency pins are unchanged. Existing benchmark PRs did not update CHANGELOG, so this PR follows that precedent.

Refs #776.

Binding dependencies (merge first): Python #17, browser #21, Node #21, C #14.

Verified review fixes: untimed failures preserve their exit status; experimental binding refs use separate history and skip production regression issues. Binding PRs move checksum validation outside timing, force the Node local artifact, decode C sets and free Wasm handles explicitly. All 26 cases passed untimed after those fixes.

🤖 Generated with Codex, orchestrated by Claude Code

This was referenced Nov 3, 2026
Merged
Merged
Merged
Merged
@codecov
codecov Bot commented Nov 3, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@adityamukho
adityamukho commented Nov 3, 2026
Copy link
Copy Markdown
Collaborator Author

Phase-A untimed evidence after the independent reviews and fixes:

The four host entry points (no Criterion regex applies) each execute their whole shortlist once:

  • Python: target/bench-venv/bin/python bench/nightly.py --test
  • Browser/Node: node bench/nightly.mjs --test
  • C: target/p16/nightly --test
ok: binding/python/prepared_execute/1k
ok: binding/python/cursor_batch_1/1k
ok: binding/python/cursor_batch_256/1k
ok: binding/python/set_roundtrip/2
ok: binding/python/predicate/1k
ok: binding/python/aggregate/1k
ok: binding/browser/prepared_execute/1k
ok: binding/browser/cursor_batch_1/1k
ok: binding/browser/cursor_batch_256/1k
ok: binding/browser/set_roundtrip/2
ok: binding/browser/predicate/1k
ok: binding/browser/aggregate/1k
ok: binding/browser/indexeddb_write/1k
ok: binding/browser/indexeddb_reopen/1k
ok: binding/node/prepared_execute/1k
ok: binding/node/cursor_batch_1/1k
ok: binding/node/cursor_batch_256/1k
ok: binding/node/set_roundtrip/2
ok: binding/node/predicate/1k
ok: binding/node/aggregate/1k
ok: binding/c/prepared_execute/1k
ok: binding/c/cursor_batch_1/1k
ok: binding/c/cursor_batch_256/1k
ok: binding/c/set_roundtrip/2
ok: binding/c/predicate/1k
ok: binding/c/aggregate/1k

All 26 IDs passed using local release builds, also with a temporary patch resolving the exact P16 core source. The core shell build helper passed end to end for Node, including Cargo metadata source verification. No timed local run, SSH or benchmark dispatch was performed.

Validation: 17 reporting tests pass on both branches, actionlint passes, shell syntax/format checks pass, C compiles with -Wall -Wextra -Werror, and the two core workflow/reporting changes are byte-identical.

All six PRs received codex exec -m gpt-6-astra -s read-only ... < /dev/null reviews. Seven unique verified findings were fixed: checksum reductions moved outside timing; executable Python build instructions; forced local Node addon; actual C set decoding; prerequisite pipefail; separate experimental binding histories; deterministic Wasm teardown. No findings were rejected. Untimed checks and CI were rerun after fixes. Normal CI: green on the latest head (17 passing checks; gh pr checks --watch exited 0).

Routes: bindings matrix, bench / binding_{python,browser,node,c}, entire fixed shortlists; main/v2 checkouts skip before build/upload. Experimental binding refs use binding-experiment/<core-ref>/<binding-ref> histories and skip production issue reporting. Merge the four binding PRs before the core routes; phase B may select binding-ref=bench/776-p16-bindings in its serial dispatch. Dedicated testbeds and the 5–15 minute/job plus setup allowance remain unverified until phase B.

Java/JVM, Android, Swift, WASI and other deliberately unmeasured paths require release sign-off as documented in docs/BINDING_BENCHMARKS.md. This does not claim #776 can close.

adityamukho and others added 2 commits November 3, 2026 18:24
@adityamukho @codex
ci(bench): schedule foreign runtime costs from main (Refs #776)
0b7750c 
Identical companion to the v3 P16 workflow and reporting changes.

Co-Authored-By: Codex <noreply@openai.com>
@adityamukho @codex
ci(bench): preserve verification and isolate binding experiments (Refs …
f7dbae2 
…#776)

Identical companion review fixes to the v3 P16 workflow changes.

Co-Authored-By: Codex <noreply@openai.com>
@adityamukho
adityamukho merged commit 4d65022 into main Nov 3, 2026
17 checks passed
@adityamukho
adityamukho deleted the ci/776-p16-main branch November 3, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@adityamukho
0