Routing benchmark agent
Cheapest-first is
a warning.
routerbench scores every routing strategy — cheapest-first, quality-first, oracle — on the identical seeded workload, then proves Pareto dominance instead of trusting a vibe. The router that wins has to be cheaper AND at least as accurate. No exceptions.
routerbench · cost-vs-quality Pareto benchmark · [higgsfield: noir gold scatterplot, three glowing points, one strictly dominant] · measurement layer
The failure mode
The router nobody benchmarked
“We picked the cheap model to save money. Nobody checked what it actually got wrong until the invoice for the redo work came in.”Platform lead · multi-model inference stack
Cheapest-first routing looks great in a cost dashboard and terrible in a quality audit — and those two dashboards rarely sit next to each other. Teams ship a router because it's cheap, then discover months later that a third of its outputs never cleared the bar the task actually required. There's no shared workload, no dominance test, no floor. Just a cost line that went down and a quality problem nobody was watching for.
How it works
Three steps. No faith required.
01
Route
Three candidate strategies — cheapest_first, quality_first, oracle — run the identical 50-task seeded workload.
02
Compare
Pareto dominance decides the winner: strictly cheaper AND at least as accurate, never just "close enough."
03
Seal
Dominance and Pareto-front membership both have to hold against an independently-computed reference oracle before the packet seals.
Proof band
One packet. Every claim accounted for.
packet rig/routerbench · Pareto routing benchmark claim "oracle routing Pareto-dominates quality-first while cheapest-first fails the quality floor" workload 50 tasks · seed 42 · shared across all candidates oracle pass_rate 100% · cost priced after a short conversation quality_first pass_rate 100% · cost priced after a short conversation (8.5x costlier, same quality) cheapest_first PASS — correctly fails the floor: pass_rate 34.0% < 85% floor dominance PASS — oracle dominates quality_first, sits on the Pareto front themis SEALED — terminal verifier signed
Plant the router-collapse bug — the oracle's per-task logic falls back to one fixed model — and it drops off the Pareto front entirely: 96% accuracy priced after a short conversation dominated on both axes by the real reference oracle. Both dominance gates go RED, nothing seals. How ProofPackets work →
Before you install
The obvious objections
Q We already track $/token in a dashboard.
A dashboard shows spend after the fact. routerbench proves, before you ship a router, that your "smart" routing strategy actually Pareto-dominates a dumb one — cheaper AND at least as accurate — on a shared, seeded workload. Dashboards don't have a REFUTED state.
Q Isn't cheapest-first obviously bad? Why benchmark the obvious?
You'd think so — but teams ship it anyway because the demo looks fine on five easy prompts. On our 50-task seeded workload, cheapest-first passes 34% of tasks. That's the number the quality-floor gate is built to catch before it hits prod, not after.
Q Why wouldn't oracle routing just win by construction?
It doesn't get a free pass — it has to Pareto-dominate quality_first AND stay on the Pareto front against an independently-computed reference oracle. Plant a regression (router logic collapses to one fixed model) and both dominance gates flip RED immediately.
Pricing
Start free. Pay when it pays.
Generalist ships today. Domain Expert adds vertical memory + priors. Custom is built to your stack.
Free
Priced after a short conversation
25 routing evaluations a month. Public evidence, full gate output.
Team
Priced after a short conversation
Unlimited evaluations, custom router fleets, private workloads.
Pro
Priced after a short conversation
Dedicated verifier, CI-gated routing regression, refute-turnaround SLA.
