Rubric verification agent
Rubric grades that
survive refutation.
rig-grader scores every step of an agent trace PASS / SUBOPTIMAL / FAIL from observed tool output alone — never from a self-report — then independently tries to refute the run’s own “this succeeded” claim. One FAIL step is grounds enough.
rig-grader · rubric execution engine · step-level refutation[higgsfield: gold-lit courtroom gavel resting on an engraved rubric grid, brass numerals, deep noir vignette, single key light upper-left]
Why teams install
What rig-grader earns back
Five failure classes, zero misses
Fabricated success, ignored error, silent stub, hallucinated output, wrong target — 23/23 planted errors caught, cold.
‹1a43d6f›Never grades the self-report
grade_step ignores claimed_status entirely; the verdict comes from actual_status and output_matches_goal alone.
‹b3d0562›Refutes runs, not just steps
5 of 6 simulated runs independently refuted; the zero-error clean-control run is never wrongly refuted.
‹3830b3e›How it works
Three steps. No faith required.
01
Define
Write the rubric once: claimed vs. actual status, output-matches-goal, redundant-call checks.
02
Grade
rig-grader scores every step from observed evidence, blind to what the agent claims happened.
03
Refute
One FAIL step is sufficient grounds to refute the whole run’s success claim — sealed either way.
Proof band
One packet. Every claim accounted for.
packet rig/rig-grader · eval bench v4 suite agent-crm-outbound · 72 steps · 6 runs · seed 42 result detection 1.0000 (23/23) · accuracy 1.0000 (72/72) · refuted 5/6 rubric 5 failure classes · 0 self-certified anchor PASS — clean-control run correctly never refuted ‹1a43d6f› themis SEALED — merge gate armed
Every rig-grader run emits a packet like this — hash-chained, verifier-signed, and queryable from your own dashboard. How ProofPackets work →
Buyers
Who runs rig-grader
Head of AI Platform
Series C devtool
Staff MLE
fintech risk org
Evals Lead
frontier lab
Objections
Answered before you ask
Isn’t this just an LLM-as-judge wrapper?
No self-report is ever trusted — grade_step reads actual_status and output_matches_goal, never claimed_status. The clean-control run is never wrongly refuted; a sweet-talkable judge can’t guarantee that.
What stops the grader from grading itself favorably?
It doesn’t self-certify. Every verdict is itself gated: detection rate ≥90%, accuracy ≥95%, and at least one run must be independently refuted — or the packet doesn’t seal.
Why trust the numbers on this page?
Every stat is the literal output of python3 main.py --seed 42, hash-chained into proof_packets.jsonl. Reproduce it yourself — the seed is public.
Pricing
Start free. Pay when it pays.
Generalist ships today. Domain Expert adds vertical memory + priors. Custom is built to your stack.
Free
Priced after a short conversation
100 eval runs a month. Core rubric library included.
Team
Priced after a short conversation
Unlimited suites, CI gates, transcript forensics.
Pro
Priced after a short conversation
Dedicated eval fleet, custom graders, SLA.
First 100 installs lock Team pricing priced after a short conversation — standard rate is priced after a short conversation.
