8000 0.4.1: re-measured long sessions (5 per strategy, with lean); corrected claims by dominicrico · Pull Request #7 · dominicrico/jev-router · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "jev-router",
"version": "0.4.0",
"version": "0.4.1",
"description": "Asks TypeSafe Jev before each task which Claude model and effort fit best, then runs the turn on it",
"types": "./types/index.d.ts",
"userConfig": {
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
All notable changes to jev-router are documented here.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project uses [Semantic Versioning](https://semver.org/).

## [0.4.1] - 2026-10-10

### Fixed (documentation, corrected measurements)
- Re-measured the long-session benchmark with 5 sessions per strategy and the new `lean` default. The guard still cuts model switches (6 to 7 down to 1) and halves cache re-writes, but it does **not** lower cost: guard on and off both cost +42% against Opus. The earlier "+24% vs +38%" (3 sessions) was noise and is retracted. `lean` averaged the same cost as Opus in long warm sessions (range $0.26 to $0.63 per session); plain Sonnet was 43% cheaper. README, benchmarks, site and charts updated.

## [0.4.0] - 2026-10-10

### Changed (behaviour change)
Expand Down Expand Up @@ -59,6 +64,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
- Benchmarks: routing fit, cost, stability, overhead and cache stickiness over 30 tasks, plus real token usage and cost per mode against no plugin.
- Ponytail-style README with logo, hero art and benchmark charts.

[0.4.1]: https://github.com/dominicrico/jev-router/compare/v0.4.0...v0.4.1
[0.4.0]: https://github.com/dominicrico/jev-router/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/dominicrico/jev-router/compare/v0.2.0...v0.3.0
[0.2.0]: https://github.com/dominicrico/jev-router/compare/v0.1.0...v0.2.0
Expand Down
30 changes: 20 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ If your default is Opus, most of your prompts pay for more model than they need.

**Best for:** Opus-by-default users, mixed workloads full of small prompts, agent runs with subagents, and anyone who wants to *see and steer* which model runs (the band, `!opus`, `!full`, `/jev ceiling`).

**Not for:** long sessions on one big warm context. There, routing cost +24% against Opus even with the cache guard, because switching models re-writes the cache; the default `lean` preset already holds Jev to Sonnet, and plain Sonnet is the other safe choice (the +24% was measured with the older `balanced` settings). And if plain Sonnet already 8000 does all your work, it is cheaper than routing. Every figure above comes from the [benchmarks](#does-it-reduce-claude-code-costs), including the ones that went against it.
**Not for:** long sessions on one big warm context. There, `lean` averaged the same cost as Opus (some sessions as cheap as Sonnet, some as dear as Opus), and plain Sonnet was 43% cheaper than both; switching models re-writes the cache and the guard only halves that. And if plain Sonnet already does all your work, it is cheaper than routing. Every figure above comes from the [benchmarks](#does-it-reduce-claude-code-costs), including the ones that went against it.

### Install in 30 seconds

Expand Down Expand Up @@ -116,7 +116,7 @@ Three benchmarks. First, 30 single-prompt tasks: 270 calls to Jev for the picks,
| --- | --- | --- | --- |
| Trivial tasks | -97% | -97% | -97% |
| Standard tasks | -65% | -65% | -68% |
| Hard tasks, **cap high (default)** | +43% | +24% | -56% |
| Hard tasks, cap high (the `balanced` preset) | +43% | +24% | -56% |
| Hard tasks, uncapped | +242% | +207% | -39% |
| **All 30, cap high (default)** | **-1%** | **-13%** | **-64%** |
| All 30, uncapped | +123% | +101% | -53% |
Expand Down Expand Up @@ -164,16 +164,26 @@ The tasks (a concurrency bug, an interval merger with open and closed bounds, a

### Long sessions: the cache guard, measured

Switching models throws away the warm prompt cache, so the mod holds the current model while the cache is warm and large. I measured this with real long sessions: one Claude Code process per session, about 20k tokens of source pasted first, then 8 mixed tasks, 3 sessions per strategy.
Switching models throws away the warm prompt cache, so the mod holds the current model while the cache is warm and large. I measured this with real long sessions: one Claude Code process per session, about 20k tokens of source pasted first, then 8 mixed tasks, 5 sessions per strategy (re-measured in 0.4.1 with more sessions and the new `lean` default).

| Mean per session | cost | vs always opus | model switches | cache tokens re-written |
| --- | --- | --- | --- | --- |
| always opus | $0.51 | | 0 | 42k |
| always sonnet | $0.27 | -46% | 0 | 41k |
| jev-router, cache guard on (`auto`) | $0.63 | +24% | 1.0 | 78k |
| jev-router, cache guard off | $0.70 | +38% | 6.3 | 194k |
<p align="center">
<img src="assets/cache.png" width="880" alt="Measured long sessions, 5 per strategy: always opus 48 cents, always sonnet 27, jev-router balanced with the cache guard 67 and without 68, lean 48 with a range from 26 to 63 cents.">
</p>

| Mean per session (5 sessions each) | cost | range | vs always opus | switches | cache tokens re-written |
| --- | --- | --- | --- | --- | --- |
| always opus | $0.48 | $0.29-$0.57 | | 0 | 36k |
| always sonnet | $0.27 | $0.26-$0.28 | -43% | 0 | 42k |
| jev-router `balanced`, cache guard on (`auto`) | $0.67 | $0.60-$0.77 | +42% | 1.0 | 81k |
| jev-router `balanced`, cache guard off | $0.68 | $0.58-$0.79 | +42% | 6.4 | 168k |
| **jev-router `lean` (the default)** | **$0.48** | $0.26-$0.63 | **0%** | 0.8 | 53k |

What the 5-session numbers say, including what changed from my first 3-session run:

The guard works (6 switches become 1, cache re-writes fall by 60%, cost falls from +38% to +24% against opus), but **in a long warm session routing still cost more than just staying on opus**, and always-sonnet was half the price. (My earlier simulation said "3x more"; the measurement says +38%, so the simulation exaggerated.) In short sessions with small contexts (5 to 10k tokens, below the guard's threshold) the guard barely matters and the picture is the same: routing saved 14% against opus with the guard off and 4% with it on, always-sonnet saved 59%. Routing earns its keep when the cache is cold or the context is small, and when your habit is to run everything on opus. If you work in long warm sessions, set a ceiling (below) or just use sonnet.
- **The guard does what it says on switches and cache writes**: 6 to 7 model switches become 1, and cache re-writes halve (168k to 81k tokens). **It did not lower the bill**: guard on and off both cost +42% against Opus. My first run (3 sessions) showed +24% against +38% and I read that as a guard benefit; with more sessions it was noise. Corrected.
- **`lean` averaged the same as Opus (+0%), but the average hides two kinds of session.** In 3 of 5 it stayed on Sonnet and cost $0.26 to $0.45. In 2 of 5 Jev was at least 90% sure about a hard task, the ceiling let Opus through, the guard then held Opus for the rest of the session, and the session cost about $0.60. Always-Sonnet cost $0.27 every time.
- **So in a long warm session `lean` does not beat staying on one model.** If you work like this, plain Sonnet is the cheapest safe choice, or raise `ceilingBreak` to 1.0 so the ceiling is never broken (not measured).
- Short sessions with small contexts and the cold-cache case look different (see the table above and the benchmarks folder); routing's wins there come from easy tasks.

### Presets

Expand Down
Binary file modified assets/cache.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 4 additions & 4 deletions assets/src/make.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,14 @@ ${[['efficient', fE, uE], ['balanced', fB, uB], ['cheap', fC, uC]].map(([m, f, u
<div class=foot>Fit means the pick matches what an engineer would choose for the tier: trivial haiku, standard sonnet, hard opus or fable. Tier labels are the author's. Measures what Jev picks, not answer quality.</div></div>`

const sess = JSON.parse(readFileSync(res('session-long.json'), 'utf8')).filter(x => x.complete)
const SN = [['no plugin: always opus', 'always opus', '#475569'], ['no plugin: always sonnet', 'always sonnet', '#64748b'], ['jev-router: stickiness auto', 'jev-router, cache guard on (auto)', G], ['jev-router: stickiness off', 'jev-router, cache guard off', 'linear-gradient(90deg,#fbbf24,#fb7185)']]
const SN = [['no plugin: always opus', 'always opus', '#475569'], ['no plugin: always sonnet', 'always sonnet', '#64748b'], ['jev-router: stickiness auto', 'jev-router balanced, cache guard on (auto)', G], ['jev-router: stickiness off', 'jev-router, cache guard off', 'linear-gradient(90deg,#fbbf24,#fb7185)'], ['jev-router: lean (the default)', 'jev-router, lean (the default)', G]]
const sMean = (st, f) => { const r = sess.filter(x => x.strategy === st); return r.reduce((a, x) => a + f(x), 0) / r.length }
const sCost = st => sMean(st, x => x.totalUsd), sSw = st => sMean(st, x => x.switches), sCw = st => sMean(st, x => x.turns.reduce((a, t) => a + (t.cacheWrite ?? 0), 0))
const sMax = Math.max(...SN.map(([st]) => sCost(st)))
const cacheCard = `<div class=card><div class=eye>Cache guard · measured · ${sess.length / SN.length} long sessions of 8 tasks per strategy</div>
<h1>The guard cut cache re-writes by <span class=g>${Math.round(100 - (100 * sCw(SN[2][0])) / sCw(SN[3][0]))}%</span>. In a long warm session routing still cost more than opus.</h1>
<div style="display:grid;gap:15px">${SN.map(([st, label, fill]) => `<div><div style="font-size:14px;margin-bottom:6px">${label}</div><div style="display:flex;align-items:center;gap:12px"><div style="width:${(sCost(st) / sMax) * 480}px;height:14px;border-radius:7px;background:${fill}"></div><span class=mono style="font-size:13px;font-weight:700">$${sCost(st).toFixed(2)}</span><span class=mono style="font-size:12px;color:#94a3b8">${sSw(st).toFixed(1)} switches · ${Math.round(sCw(st) / 1000)}k cache writes</span></div></div>`).join('')}</div>
<div class=foot>Mean per session, real runs: one long Claude Code process per session, about 20k tokens of source pasted first so the cache is large, then 8 mixed tasks. A model switch re-writes the whole warm context. The guard helped (+${Math.round((100 * sCost(SN[2][0])) / sCost(SN[0][0]) - 100)}% against opus instead of +${Math.round((100 * sCost(SN[3][0])) / sCost(SN[0][0]) - 100)}%), but staying on one model was cheaper. Routing pays when the cache is cold or the context is small.</div></div>`
<h1>The guard halved cache re-writes. <span class=g>It did not lower the bill</span>, and lean averaged the same as opus.</h1>
<div style="display:grid;gap:15px">${SN.map(([st, label, fill]) => `<div><div style="font-size:14px;margin-bottom:6px">${label}</div><div style="display:flex;align-items:center;gap:12px"><div style="width:${(sCost(st) / sMax) * 440}px;height:14px;border-radius:7px;background:${fill}"></div><span class=mono style="font-size:13px;font-weight:700">$${sCost(st).toFixed(2)}</span><span class=mono style="font-size:12px;color:#94a3b8">${sSw(st).toFixed(1)} switches · ${Math.round(sCw(st) / 1000)}k cache writes</span></div></div>`).join('')}</div>
<div class=foot>Mean per session, ${sess.length / SN.length} real long sessions per strategy: one Claude Code process, about 20k tokens of source pasted first, then 8 mixed tasks. A model switch re-writes the whole warm context. Guard on and off cost the same (+${Math.round((100 * sCost(SN[2][0])) / sCost(SN[0][0]) - 100)}% and +${Math.round((100 * sCost(SN[3][0])) / sCost(SN[0][0]) - 100)}% against opus). Lean was bimodal: $${Math.min(...sess.filter(x => x.strategy === SN[4][0]).map(x => x.totalUsd)).toFixed(2)} to $${Math.max(...sess.filter(x => x.strategy === SN[4][0]).map(x => x.totalUsd)).toFixed(2)} per session. Plain sonnet was the cheapest every time.</div></div>`

const rows = [['1', 'A task arrives', 'your prompt'], ['2', 'Ask Jev: which model, which effort?', `~${q(0.5)} ms per call`], ['3', 'Is the prompt cache warm?', 'stay put unless Jev is sure'], ['4', 'Cap the effort at high', '!full lifts it for one prompt'], ['5', 'Re-ask before each step, route each subagent', 'routeSteps, routeSubagents'], ['6', 'Show it above the prompt', 'the band']]
const howCard = `<div class=card><div class=eye>How it works</div>
Expand Down
24 changes: 13 additions & 11 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Full tables in [results/RESULTS.md](results/RESULTS.md).
- **The modes are a real dial.** `balanced` sends some hard tasks (40%) to sonnet to save money; `cheap` sends all of them there, which is why a third of its picks are under-served. Use `cheap` for work where a miss is cheap.
- **It is stable.** The same task got the same model on 97% to 100% of repeats.
- **It is fast.** Asking Jev adds about 250 ms before a task (p95 314 ms, max 397 ms; the mod's timeout is 4000 ms).
- **Cache stickiness, simulated first, then measured.** A simulation said following every pick in a long warm session would cost 3x more than opus; the real long-session run below says +38%, so treat the simulation as an upper bound.
- **Cache stickiness, simulated first, then measured.** A simulation said following every pick in a long warm session would cost 3x more than opus; the real long-session run below says +42%, so treat the simulation as an upper bound.

## Measured tokens and cost

Expand Down Expand Up @@ -113,18 +113,20 @@ Finding the right way to measure it took two probes. `claude -p --resume` per tu

Short contexts first (5 to 10k tokens, `results/SESSION.md`): always opus $0.225 per session, always sonnet $0.093, guard on $0.216 (4.7 switches), guard off $0.193 (6.3 switches). The guard is below its 8,000-token threshold for much of these, switches cost little, and keeping the guard on actually cost slightly more because it held the session on opus.

Long contexts (`results/SESSION-LONG.md`, 3 sessions per strategy, 12 complete sessions, $6.32):
Long contexts (`results/SESSION-LONG.md`, 5 sessions per strategy, 25 complete sessions, about $13 in total across two rounds):

| Mean per session | cost | vs always opus | model switches | cache write tokens | cache read tokens |
| Mean per session (5 sessions each) | cost | range | vs always opus | switches | cache tokens re-written |
| --- | --- | --- | --- | --- | --- |
| always opus | $0.506 | | 0 | 41.7k | 285.6k |
| always sonnet | $0.273 | -46% | 0 | 41.0k | 283.0k |
| jev-router, guard on (`auto`) | $0.627 | +24% | 1.0 | 78.4k | 247.9k |
| jev-router, guard off | $0.699 | +38% | 6.3 | 193.8k | 131.6k |

- **The guard does what it says:** switches 6.3 to 1.0, cache re-writes 194k to 78k tokens, cost +38% to +24% against opus.
- **But routing lost to staying on one model** in a long warm session, in both strategies. The savings in the other benchmarks come from easy tasks and cold or small contexts.
- **Caveats:** tools off, 8 short tasks, 3 sessions per strategy, one fixture of context. A real session has tool results growing the context, which would make switches dearer still.
| always opus | $0.48 | $0.29-$0.57 | | 0 | 36k |
| always sonnet | $0.27 | $0.26-$0.28 | -43% | 0 | 42k |
| jev-router `balanced`, cache guard on (`auto`) | $0.67 | $0.60-$0.77 | +42% | 1.0 | 81k |
| jev-router `balanced`, cache guard off | $0.68 | $0.58-$0.79 | +42% | 6.4 | 168k |
| **jev-router `lean` (the default)** | **$0.48** | $0.26-$0.63 | **0%** | 0.8 | 53k |

- **The guard cuts switches and cache writes, not the bill.** Switches 6 to 7 become 1 and cache writes roughly halve, but guard on and off both cost +42% against Opus. The first 3-session round showed +24% against +38%; the 5-session re-measurement shows that difference was noise, so I retracted it.
- **`lean` is bimodal.** Mean +0% against Opus, per-session range $0.26 to $0.63 (sd $0.13 against $0.006 for Sonnet). In 3 sessions Jev stayed under 90% confidence and the session stayed on Sonnet; in 2 it was confident about a hard task, Opus was allowed, and the guard then kept Opus until the end.
- **Routing still lost to staying on one model** in long warm sessions: always-Sonnet cost -43% against Opus, every time.
- **Caveats:** tools off, 8 short tasks, one fixture of context, 5 sessions per strategy (the Opus runs alone ranged $0.29 to $0.57). A real session with tool results growing the context would make switches dearer.

## The lean preset (the default since 0.4.0)

Expand Down
Loading
Loading
0