Fleet observability agent
Know which agent
is burning money.
evalobs classifies every telemetry window breach or compliant against fixed SLO thresholds, then flags anomalies with a rolling per-agent z-score detector — scored against hidden ground truth, not tuned to look busy. One alert list, silent when things are fine.
evalobs · SLO + anomaly fleet observability · [higgsfield: noir gold ten agent nodes on a grid, one flaring red, alert triage line pointing to it] · agent engineering
The failure mode
Every metric watched, nothing triaged
“We have dashboards for every agent. We still don't know which one is quietly costing us money right now.”Platform ops lead · 10-agent production fleet
Alert fatigue is the default state of most observability setups: every metric line is watched, none of the noise is triaged into an actual signal. Teams either drown in false positives from a detector tuned for maximum recall, or miss real degradation because the threshold got set once and never revisited against how the fleet's behavior actually shifted. Neither state is observability — it's noise with a dashboard attached.
How it works
Three steps. No faith required.
01
Classify
Every telemetry window gets scored breach or compliant against fixed per-agent SLO thresholds.
02
Detect
A rolling z-score flags windows where error rate, latency, or cost jumps outside that agent's own trailing behavior — no global stats, no look-ahead.
03
Seal
Accuracy, recall, and false-positive rate all have to clear their floors against hidden ground truth before the alert list — and the packet — seal.
Proof band
One packet. Every claim accounted for.
packet rig/evalobs · fleet SLO + anomaly observability claim "SLO breach classification ≥95% accurate; anomaly recall ≥90% at FPR ≤5%" fleet 10 agents · 150 windows each · 48 injected anomalies · seed 42 slo_accuracy PASS — 97.53% ≥ 95% floor anomaly_recall PASS — 93.75% ≥ 90% floor (45/48 caught) anomaly_fpr PASS — 3.93% ≤ 5% ceiling themis SEALED — terminal verifier signed
Switch the detector to alert_everything mode and recall hits 100% trivially — but false-positive rate spikes to 100% too, and the FPR gate correctly rejects it: RED, not sealed. High recall alone proves nothing. How ProofPackets work →
Before you install
The obvious objections
Q This sounds like Datadog with extra steps.
Datadog tells you a metric crossed a line. evalobs proves — against hidden ground truth, not just thresholds — that its SLO classifier and anomaly detector are actually right: 97.5% classification accuracy, 93.75% recall, 3.93% false-positive rate, sealed into a packet you can re-derive.
Q Won't a detector tuned for high recall just alert on everything?
That's the exact planted failure we test for. Force alert_everything mode and recall hits 100% trivially — it flags every window — but the false-positive-rate gate catches it instantly: FPR spikes to 100% and the run goes RED. High recall alone proves nothing.
Q Ten agents and 150 windows each — is that a real fleet or a toy demo?
It's 1,500 telemetry windows with a persistent hidden health regime per agent (not flickering noise) and rare injected shocks (~4% of windows) as ground truth. The eval harness runs the exact same classify_slo/rolling_anomaly_flags code path the production observe tool calls.
Pricing
Start free. Pay when it pays.
Generalist ships today. Domain Expert adds vertical memory + priors. Custom is built to your stack.
Free
Priced after a short conversation
2 agents monitored, 7-day window. Public evidence, full gate output.
Team
Priced after a short conversation
Unlimited agents, custom SLO thresholds, alert routing.
Pro
Priced after a short conversation
Dedicated verifier, CI-gated detector regression, refute-turnaround SLA.
Runs well with
