8000 Benchmark Jev against Gemini Flash in io-auto-mode OSS · Issue #8 · simon-inkie/inkie-auto-mode · GitHub
[go: up one dir, main page]

Skip to content

Benchmark Jev against Gemini Flash in io-auto-mode OSS #8

Open

Description

@simon-inkie

Original Linear description

Owner: platform maintainer.

Goal

Benchmark Jev against Gemini Flash as the classifier in io-auto-mode, on the public GitHub OSS master baseline, and publish reproducible receipts.

Scope

  1. Baseline: public GitHub OSS master (simon-inkie/inkie-auto-mode).
  2. Proposed public code-default change: use a pinned google/gemini-3.8-flash, subject to its own explicit build decision and current model verification. This does not change live configuration. See the historical preconditions below.
  3. Add a benchmark-only TypeSafe/Jev adapter. Benchmark path only.
  4. Run the existing 159-case fixture corpus against both providers.
  5. Publish reproducible receipts, containing:
    • public commit SHA
    • exact model IDs for both providers
    • prompt hash and fixture hash
    • category-level safety metrics
    • every expected-block miss, enumerated, not just a count
    • confidence, latency
    • API usage and cost where the provider exposes it

Explicitly out of scope

  • No live or private routing changes. Nothing that alters what the classifier does for real tool calls.
  • No credential changes.

Review gate before any provider recommendation

A repeat-run and disagreement review must pass before this benchmark is used to recommend a provider. Single-run results do not support a recommendation: the classifier is non-deterministic, so a single pass cannot distinguish a real capability difference from sampling noise. Re-run, and review every case where the two providers disagree, and where a provider disagrees with itself across runs.

Corpus, as it actually exists on disk

Verified at benchmarks/fixtures/: 12 JSONL files, one case per line, 159 cases total. The filenames are the safety categories, so category-level metrics fall out of the file split directly.

Category file Cases
credential-access.jsonl 24
prompt-injection.jsonl 20
code-from-external.jsonl 15
irreversible-destruction.jsonl 15
obfuscation.jsonl 15
agent-autonomous.jsonl 10
config-mutation.jsonl 10
destruction-safe-path.jsonl 10
dev-build.jsonl 10
io-own-scripts.jsonl 10
version-control-read.jsonl 10
version-control-write.jsonl 10
Total 159

Note the corpus is unbalanced by design. Any headline accuracy figure will be weighted toward credential-access and prompt-injection, so report per-category and treat a single aggregate number as misleading on its own.

Historical preconditions

The original record stated that provider access and model-availability prerequisites had been satisfied on 2026-09-19. That is historical input, not fresh verification of access, availability or the model's current status. Verify availability and the exact pinned model ID when the benchmark starts.

The baseline remains the public OSS code, not personal or private configuration. The proposed public code-default change requires its own explicit build decision; this ticket authorises no live configuration or routing change.

Origin

Migrated from Linear: INK-923. The original wording is retained except the disclosed privacy and rendering edits below.

Current scope and qualifications

The description above is the historical Linear account. Its historical verification and provider claims have not been independently revalidated by this transfer.
Current Linear state is Backlog. The historical report is retained; this transfer does not independently establish completion against the current public baseline.

Provider calls, recommendations, routing changes and live installation retain their own approval and evidence gates.

Publication transfers the task; it does not authorise live recovery, deployment, spending or configuration changes.

Transfer edits

  • Historical principal names are aliased in prose; artificial shell-filename web links are rendered as code where present.
  • Historical fleet identities are aliased; code identifiers and permitted source paths remain references.
  • Private access/provisioning and configuration details are omitted; historical prerequisites and the proposed public code-default decision are qualified.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmajor-featureA substantial feature or new capability

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0