Evaluation agent
Your agents, graded.
By a stricter agent.
judge scores every agent run against a rubric no one can sweet-talk — pass@k, pass^k, full tool-call audits. A regression fails your build before it fails your users.
judge · rubric execution engine · eval bench v4
Why teams install
What judge earns back
Catches regressions humans miss
1,400-run eval suites finish in under nine minutes, cold.
‹e71b0d2›Grades the grader
Every rubric item is itself adjudicated. Nothing self-certifies.
‹a04f6c8›Transcript-level evidence
A failing run links the exact tool call that broke the contract.
‹f33d917›How it works
Three steps. No faith required.
01
Define
Write the rubric once: deterministic checks, scored judgments, real-world probes.
02
Execute
judge replays N runs across seeds and models — cold, every time.
03
Gate
Below threshold the merge blocks. Above it, the verdict seals.
Proof band
One packet. Every claim accounted for.
packet rig/judge · eval-bench v4 suite agent-crm-outbound · 1,400 runs · 8 seeds result pass@8 0.87 · pass^8 0.71 · Δ +0.02 rubric 41 items · 0 self-certified anchor PASS — transcripts hash-matched ‹f33d917› themis SEALED — merge gate armed
Every judge run emits a packet like this — hash-chained, verifier-signed, and queryable from your own dashboard. How ProofPackets work →
Buyers
Who runs judge
Head of AI Platform
Series C devtool
Staff MLE
fintech risk org
Evals Lead
frontier lab
Before you install
The obvious objections
Q Why did Team pricing drop priced after a short conversation?
Same eval fleet, real usage-based pricing. Most teams weren't running anywhere near the volume the old priced after a short conversation tier assumed. Pro priced after a short conversation now covers the dedicated eval fleet and custom graders that heavy users actually need.
Q Who grades the grader? What stops judge from just rubber-stamping?
Every rubric item is itself adjudicated — a passing verdict on a self-certified check doesn't count. judge's own gate carries a planted-failure regression: break the rubric logic, confirm it goes RED, restore, and that becomes a permanent regression case.
Q 1,400 runs in under nine minutes sounds too fast to be thorough.
The suite runs cold, across 8 seeds, with pass@k and pass^k computed separately — pass^k is the stricter number because it requires every seed to pass, not just one lucky run. Speed comes from parallelizing seeds, not cutting rubric items.
Pricing
Start free. Pay when it pays.
Generalist ships today. Domain Expert adds vertical memory + priors. Custom is built to your stack.
Free
Priced after a short conversation
100 eval runs a month. Core rubric library included.
Team
Priced after a short conversation
Unlimited suites, CI gates, transcript forensics.
Pro
Priced after a short conversation
Dedicated eval fleet, custom graders, SLA.
Runs well with
