8000 INK-923: publish Jev vs Gemini benchmark and raw evidence by simon-inkie · Pull Request #4 · simon-inkie/inkie-auto-mode · GitHub
[go: up one dir, main page]

Skip to content

INK-923: publish Jev vs Gemini benchmark and raw evidence - #4

Merged
simon-inkie merged 11 commits into
masterfrom
doctor2/ink-923-jev-benchmark
Sep 20, 2026
Merged

simon-inkie merged 11 commits into
masterfrom
doctor2/ink-923-jev-benchmark

Conversation

@simon-inkie
Copy link
Copy Markdown
Owner

Summary

Adds the reproducible INK-923 Jev versus Gemini benchmark harness, provider-contract tests, thresholded-A candidates, and the public evidence bundle.

The bundle contains exactly five clean 20 September 2026 runs: 159 fixtures, four repeats each, zero provider errors, SHA-256 manifest, raw JSON, methodology, reproduction pins, fallback accounting, and pricing limitations. Invalid exploratory attempts remain separately documented and are not pooled into aggregate results.

Key evidence

  • Jev current: 566/636 pass, 27 expected-BLOCK misses, 318 ms mean latency
  • Gemini 3.8 Flash: 599/636, 11 misses, 7.34 s
  • Gemini 3.5 Flash Lite: 533/636, 14 misses, 1.15 s
  • Jev thresholded-A .86/.61/.09: 631/636, zero expected-BLOCK misses, 358 ms, with conservative false blocks as the explicit trade-off

Verification

  • Five raw files parse with jq
  • SHA-256 manifest verifies 5/5
  • git diff --check clean
  • Aggy PASS and Andor PASS on the amended public bundle
  • No provider calls, deployment, or production configuration changes in this PR

The public benchmark report links to the bundle under docs/benchmarks/ink-923-jev-2026-09-20/.

@simon-inkie
simon-inkie merged commit 9dbc7f9 into master Sep 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0