Fleet benchmarking agent
Measure the split.
Not the story.
bench runs EdgeShard's shard plan against an all-cloud baseline across a suite of workload sizes — using a real per-link latency matrix, not a flat constant — and seals the geomean ratio into a hash-chained packet before anyone calls the throughput claim true.
bench · EdgeShard throughput suite · [higgsfield: noir stage, gold twin gauge needles comparing edge-fleet vs cloud-serial latency, dual exposure] · measurement layer
The failure mode
The claim that doesn't survive prod
“Our shard plan hit 1.6x in the demo. In prod it's basically the same as running everything on the cloud node.”Infra lead · edge inference platform
A single lucky scenario isn't a benchmark. Teams ship a throughput number measured once, on one workload size, against a merge-overhead constant that ignores real per-link latency between nodes. The first time sharding gets disabled — a config flag, a bad deploy, a fallback path nobody audited — the whole system quietly degenerates to serial cloud execution, and nothing in the dashboard says so.
How it works
Three steps. No faith required.
01
Plan
A bisection solver computes an equal-finish-time shard allocation across the real K×K pairwise latency matrix — not a flat per-node constant.
02
Measure
The all-cloud baseline and the EdgeShard plan run the identical deterministic transform, checked byte-for-byte, across three workload sizes.
03
Seal
Geomean throughput, per-scenario accuracy, and full token coverage all have to clear the gate before a packet enters the chain.
Proof band
One packet. Every claim accounted for.
packet rig/bench · edgeshard throughput suite claim "equal-finish-time shard plan sustains ≥1.3x geomean vs all-cloud baseline" suite 3 workloads (2048 / 8192 / 32768 tok) · seed 42 · 5-node fleet ratio 2048tok 1.53x · 8192tok 1.58x · 32768tok 1.60x throughput PASS — geomean 1.5721x ≥ floor 1.30x, min-scenario ≥ 1.0x accuracy PASS — byte-for-byte match, 3/3 scenarios themis SEALED — terminal verifier signed
Force single-node-serial routing on the identical suite and every ratio collapses to exactly 1.0x — gate_throughput_suite goes RED, the run is REFUTED, nothing enters the chain. How ProofPackets work →
Before you install
The obvious objections
Q Isn't this just a benchmarking script I could write myself?
Sure — until your fleet's latency profile changes under load. bench recomputes the K×K pairwise matrix and the equal-finish-time shard plan on every run, then seals the geomean ratio into a hash-chained packet. A script you wrote once and trust forever is exactly the failure mode this replaces.
Q How do I know 1.57x isn't a cherry-picked lucky seed?
The suite runs three workload sizes and requires every individual scenario to clear 1.0x, not just the average. Flip sharding off and the ratio collapses to exactly 1.0x on every workload — that's the planted-failure regression baked into the gate, not a marketing claim.
Q What happens when it's wrong?
gate_throughput_suite goes RED, the harness marks the run REFUTED, and nothing gets appended to the proof chain. A refuted run still leaves a full evidence snapshot — it doesn't disappear.
Pricing
Start free. Pay when it pays.
Generalist ships today. Domain Expert adds vertical memory + priors. Custom is built to your stack.
Free
Priced after a short conversation
25 benchmark suite runs a month. Public evidence, full gate output.
Team
Priced after a short conversation
Unlimited suite runs, private fleets, custom workload sizes.
Pro
Priced after a short conversation
Dedicated verifier, CI-gated throughput regression, refute-turnaround SLA.
