Repository navigation
Description
Original Linear description
Owner: platform maintainer.
Goal
Benchmark Jev against Gemini Flash as the classifier in io-auto-mode, on the public GitHub OSS master baseline, and publish reproducible receipts.
Scope
- Baseline: public GitHub OSS master (
simon-inkie/inkie-auto-mode). - Proposed public code-default change: use a pinned
google/gemini-3.8-flash, subject to its own explicit build decision and current model verification. This does not change live configuration. See the historical preconditions below. - Add a benchmark-only TypeSafe/Jev adapter. Benchmark path only.
- Run the existing 159-case fixture corpus against both providers.
- Publish reproducible receipts, containing:
- public commit SHA
- exact model IDs for both providers
- prompt hash and fixture hash
- category-level safety metrics
- every expected-block miss, enumerated, not just a count
- confidence, latency
- API usage and cost where the provider exposes it
Explicitly out of scope
- No live or private routing changes. Nothing that alters what the classifier does for real tool calls.
- No credential changes.
Review gate before any provider recommendation
A repeat-run and disagreement review must pass before this benchmark is used to recommend a provider. Single-run results do not support a recommendation: the classifier is non-deterministic, so a single pass cannot distinguish a real capability difference from sampling noise. Re-run, and review every case where the two providers disagree, and where a provider disagrees with itself across runs.
Corpus, as it actually exists on disk
Verified at benchmarks/fixtures/: 12 JSONL files, one case per line, 159 cases total. The filenames are the safety categories, so category-level metrics fall out of the file split directly.
| Category file | Cases |
|---|---|
credential-access.jsonl |
24 |
prompt-injection.jsonl |
20 |
code-from-external.jsonl |
15 |
irreversible-destruction.jsonl |
15 |
obfuscation.jsonl |
15 |
agent-autonomous.jsonl |
10 |
config-mutation.jsonl |
10 |
destruction-safe-path.jsonl |
10 |
dev-build.jsonl |
10 |
io-own-scripts.jsonl |
10 |
version-control-read.jsonl |
10 |
version-control-write.jsonl |
10 |
| Total | 159 |
Note the corpus is unbalanced by design. Any headline accuracy figure will be weighted toward credential-access and prompt-injection, so report per-category and treat a single aggregate number as misleading on its own.
Historical preconditions
The original record stated that provider access and model-availability prerequisites had been satisfied on 2026-09-19. That is historical input, not fresh verification of access, availability or the model's current status. Verify availability and the exact pinned model ID when the benchmark starts.
The baseline remains the public OSS code, not personal or private configuration. The proposed public code-default change requires its own explicit build decision; this ticket authorises no live configuration or routing change.
Origin
Migrated from Linear: INK-923. The original wording is retained except the disclosed privacy and rendering edits below.
Current scope and qualifications
The description above is the historical Linear account. Its historical verification and provider claims have not been independently revalidated by this transfer.
Current Linear state is Backlog. The historical report is retained; this transfer does not independently establish completion against the current public baseline.
Provider calls, recommendations, routing changes and live installation retain their own approval and evidence gates.
Publication transfers the task; it does not authorise live recovery, deployment, spending or configuration changes.
Transfer edits
- Historical principal names are aliased in prose; artificial shell-filename web links are rendered as code where present.
- Historical fleet identities are aliased; code identifiers and permitted source paths remain references.
- Private access/provisioning and configuration details are omitted; historical prerequisites and the proposed public code-default decision are qualified.
Activity