SECURITY BENCHMARK
We built a deliberately hard corpus, labelled it three times independently, and ran it through the gateway path we ship today. This is the method, the results, and the parts you should read slowly.
What this measures
The gateway connection path, as deployed today: the guardrail engine that sits on the wire, evaluating the exact rule payload the gateway sends, composed with the session analyzer that scores risk ahead of it.
Hoop is moving to a sidecar. The sidecar’s masking engine is in the data results below. Its guardrail engine is not in the command results: it runs the same 1,144 commands next, against the same labels, and those numbers publish on this page beside these. We baseline before we rebuild, because otherwise there is nothing to compare the rebuild against.
THE CORPUS
1,144 commands across seven surfaces. Every one is written for the benchmark rather than pulled from a customer session, so nothing in the corpus is anyone’s real traffic and nothing in it can leak.
Each command carries one of three labels, and those labels are independent judgments about danger. They are never derived from our own ruleset. A ruleset scored against its own output returns perfect recall and tells you nothing about whether it catches what a reviewer would catch.
Three independent passes labelled every command. Gold is unanimous or two of three: 1,075 unanimous, 69 decided by majority, none split three ways. The contested rows stay marked in the dataset so anyone can drop them and re-score.
The hard part is the near-misses, and they are in there on purpose: a SELECT that only mentions DROP, a kubectl delete sitting behind --dry-run, a log search that quotes a destructive command. Those are the cases where a naive matcher fails in both directions at once.
Protocol surface
| SQL | 416 |
| Shell | 312 |
| Kubernetes | 208 |
| Cloud CLI | 52 |
| HTTP (proxied) | 52 |
| MongoDB | 52 |
| Redis | 52 |
Label
| blockIrreversible destruction, mass data loss, exfiltration, privilege escalation | 450 |
| approvalLegitimate, but a large blast radius. A reviewer sees it first | 296 |
| allowRoutine, read-only, or narrowly scoped | 398 |
RESULT ONE
The gateway evaluates an action in two stages. Session analysis scores intent and risk and maps the result to an action, then the guardrail engine matches rules and enforces. The benchmark composes them in the same order, so this measures the system rather than a component of it.
Of the 450 commands a reviewer labelled dangerous, 418 were blocked outright and 30 were routed for human approval. Two ran. Approval is a real third outcome here, not a rounding of block: 175 of the 296 commands that warrant a second pair of eyes were sent for review rather than stopped or waved through.
Reviewer label vs system decision
| Labelled | Blocked | Approval | Allowed |
|---|---|---|---|
| block (450) | 418 | 30 | 2 |
| approval (296) | 97 | 175 | 24 |
| allow (398) | 98 | 44 | 256 |
Rows are what a reviewer labelled the command. Columns are what the system decided. Overall accuracy 0.74, macro-F1 0.73.
Precision and recall per outcome
| Outcome | Precision | Recall | F1 |
|---|---|---|---|
| block | 0.68 | 0.93 | 0.79 |
| approval | 0.70 | 0.59 | 0.64 |
| allow | 0.91 | 0.64 | 0.75 |
Catching 93% of dangerous commands costs precision elsewhere: at this setting, 24.6% of genuinely routine commands were also stopped. The corpus is built from near-misses, so treat that as a ceiling on friction rather than a field rate, and see the caveats below for the dial that moves it.
RESULT TWO
Masking is measured on the shapes Hoop actually sees on the wire: database row frames from Postgres, MySQL, MongoDB and Oracle; JSON bodies from proxied HTTP; and terminal dumps from SSH sessions. The same rows are generated into all three envelopes, so the cost of the wire format itself is measurable rather than assumed.
Every value is generated. Free-form fields use reserved ranges, so the domains, addresses and phone numbers in the corpus cannot belong to anyone. Structured identifiers carry valid checksums, because detection depends on them.
We score twice, because the two questions have different answers. Redaction asks whether the value was blanked at all, which is what a user experiences. Attribution asks whether it was blanked by the recognizer for that type, which is what we can honestly claim to detect. A credit card blanked by a phone-number rule protects the user and proves nothing, so we report both numbers rather than the flattering one.
No single detector covers everything. Pattern matching finds every checksummed identifier and cannot see a person’s name. Statistical models do the reverse. So the masking track scores both engines Hoop ships, the gateway’s and the sidecar’s, on the same corpus. This is the one place where the new architecture is already measured.
A second, smaller dataset repeats the whole exercise on database rows, JSON payloads, log lines and CSV, so a score earned on generic prose is never reported as a score on production traffic.
Wire format under test
| Row framesPostgres, MySQL, MongoDB, Oracle |
| JSON bodiesProxied HTTP and MCP traffic |
| Terminal outputSSH and WebSocket sessions |
Identifier coverage
| Contact data and names | ✓ |
| Payment cards and bank IDs | ✓ |
| Government identifiers | ✓ |
| Credentials and API keys | ✓ |
| Crypto addresses | ✓ |
| Network and device IDs | ✓ |
| Postal addresses | ✓ |
54 labelled types in total, including formatted and unformatted variants of the same identifier, because a recognizer that finds one shape often misses the other.
RESULT THREE
The gateway’s guardrail engine evaluates 26,762 commands per second on a single machine, serially, with no concurrency. A real deployment does better, because nothing here is parallelised.
That engine can run the rules in its own process or hand them to a separate service over HTTP. Both returned identical verdicts on all 1,144 scenarios, and in process was about a hundred times faster. Same rules, same answers, two orders of magnitude apart on where the work happens. That is an argument about placement, and it is the argument the sidecar is built on.
The original question was how any of this compares to a person. At one minute per action, working through 1,000 sessions by hand is about 17 hours. The sidecar’s masking engine clears the same 1,000 sessions in under half a minute, which is where the 2,500x figure comes from. That one-minute assumption came from the brief, not from a stopwatch, and human review covers ground no engine does. We print the assumption next to the multiplier so you can substitute your own.
WHAT CHANGED
Hoop used to put a gateway between every client and every resource, with an agent inside each network to reach back through the firewall. It worked, and it made the gateway the scaling unit, the outage radius, and a network hop between the rules and the data they protect.
Enforcement is moving to a sidecar that attaches to one resource and decides in process. The control plane keeps the work that does not belong on the request path: distributing policy, collecting evidence, and tracking fleet health. Session traffic never touches it.
We are not going to tell you the benchmark proves that change was right. It does not. It measures the gateway, and it supports one piece of the case directly: identical verdicts, a hundred times faster, when the rules run in process rather than over a call to a service. The rest of the case gets tested the same way this page was.
What runs next
The sidecar’s guardrail engine goes through the same 1,144 commands, the same three-class scoring, and the same harness that produced everything above. Those numbers land on this page next to the gateway figures, whether or not they are better. A benchmark you only publish when it flatters you is a brochure.
HOW TO READ THESE NUMBERS
A benchmark without its caveats is a brochure. These six change how the figures above should be read. The first is the one a reader cannot work out from the numbers, and the second is the one most often missing from a vendor datasheet.
The command results come off the gateway connection path. The sidecar’s masking engine is in the data results, but its guardrail engine has not been scored at this scale yet. Quote a command figure as a measurement of the current gateway deployment, because that is what it is.
The same model scored 0.93 block recall under one risk-tier configuration and 0.06 under another, with identical classifications underneath. Only the map from tier to action changed. Never read a detection number without the configuration that produced it.
Three identical analyzer runs disagreed on 5.5% of scenarios. Treat any gap under two points as noise. The guardrail stage has no such variance: it is deterministic and reproduces byte for byte.
A system that blocks everything scores perfect recall. Models here predicted block for 50 to 61% of scenarios against a true rate of 39.3%, so recall only means something read next to the prediction rate and the precision.
It is weighted toward near-misses and hard identifiers, so these numbers are a floor rather than a field rate. A corpus shaped like ordinary traffic would score higher and prove less.
Throughput is measured serially with no concurrency, so a real deployment beats it. The human baseline is an assumption carried over from the original brief, not a timed study of a reviewer, and a person reviewing a command does more than detect spans.
The interesting test is not ours. Send the operations your team actually fears and watch the controls decide, on a resource you already run.