HOOP BENCHMARKS
An accuracy figure means nothing without the corpus, the labels, and the method behind it. This is what we test, how we test it, and which numbers you should read carefully.
METHOD
The harness is shared across every track. It loads a labelled dataset, runs it through the engine we ship, compares the output against the expected result, and reports accuracy next to throughput from the same run.
Every scenario is labelled by what a reviewer would call dangerous, never by what our own ruleset returns. A ruleset scored against its own output reports perfect recall and measures nothing.
Each command is labelled three times independently. Gold is unanimous or two of three. Raw agreement is 94.0% and Cohen’s kappa is 0.91. The 69 contested rows stay marked so anyone can exclude them and re-score.
It is loaded with near-misses: a SELECT that only mentions DROP, a kubectl delete behind --dry-run, a log search quoting a destructive command. An easy corpus produces a flattering number.
A run that cannot evaluate every scenario exits with an error instead of reporting metrics on whatever survived. We caught this the honest way: a simulated mid-run outage moved a score by 16 points on the surviving subset.
WHAT WE TEST
Each track answers one question about the product in production. Building the harness once means a new dataset is a new track rather than a new project. Results publish as each track finishes, including the ones that take longer than planned.
Does Hoop stop what a reviewer would stop, and hide what a reviewer would hide? Guardrails, session analysis, and data masking, scored against labelled ground truth.
Read the resultsHow many concurrent sessions does one deployment carry, on what hardware, and where does the bottleneck actually sit? Measured by replaying recorded sessions until latency degrades.
What does inspection add to a query round trip under sustained load, at the median and at the tail?
What does AI session analysis cost per action in model tokens, and how does that scale with session volume?
RESULTS
Hoop evaluates an action in two stages: session analysis scores intent and risk, then the guardrail engine matches rules and enforces. The benchmark composes them in that order, so these describe the system rather than one component of it.
Each figure carries the deployment path it came from, plus the configuration and the run-to-run variance behind it. A recall number without its false-positive rate is marketing, not measurement, so the detail page reports both.
WHAT CHANGED
Hoop used to put a gateway between every client and every resource, with an agent inside each network to reach back. It worked, and it made the gateway the scaling unit, the outage radius, and a network hop between the rules and the data they protect.
The sidecar runs next to one resource and enforces in process. The control plane distributes policy and collects evidence, and never sees a session. One thing the benchmark does support directly: the same rules returned identical verdicts on all 1,144 commands whether they were evaluated in process or sent to a separate service, and in process was about a hundred times faster.
What these numbers do not cover yet
The results above are a baseline of the gateway path, measured before the rebuild. The sidecar’s masking engine is already in them. Its guardrail engine is not: that engine runs the same 1,144 commands next, scored by the same harness against the same labels, and published on this page beside the gateway figures.
Measuring the architecture we are leaving is the only way to say honestly whether the one we are moving to is better.
Attach one sidecar to something you already run and watch the controls fire on real traffic. No infrastructure project, no new credentials, nothing rerouted.