Our score on a corpus we didn't write.
Every security vendor says best-in-class. Almost none publish a number you can check against an answer key they had no hand in. Here is ours, including the two tools that beat it.
46.5% Youden across all 2,740 OWASP BenchmarkJava cases.
OWASP Benchmark is a deliberately adversarial corpus: every real vulnerability ships alongside a safe twin written to look identical to a pattern matcher. Youden is sensitivity plus specificity minus one — it credits finding the real bug and ruling out the twin, so a tool cannot score by flagging everything.
| Published cohort | Youden |
|---|---|
| Veracode | 51 |
| Checkmarx | 47 |
| TensorShield | 46.5 |
| Fortify | 35 |
| SonarQube | 6 |
Competitor figures as published for OWASP Benchmark v1.2. We place third.
Broken down per category, recall sits at 90–96% while specificity sits at 12–27%. We are not missing vulnerabilities — we are failing to rule out the safe twins the corpus ships to punish pattern matching. That is a dataflow problem a rule cannot solve in principle, which is why the fix is escalating to full taint analysis rather than writing more pattern rules.
tsbench sast --target <BenchmarkJava> --ground-truth expectedresults-1.2.csvThe corpus and its answer key are OWASP's, published independently of us.
Two-thirds, twice, on capabilities our own benches scored perfect.
An in-house benchmark measures whether the fixtures and the code agree, not whether the product works. These are the corpora other people published for cloud privilege escalation and identity baselines, scored before the gaps they named were closed. Each comes with its limit, because the limit is what makes the number readable.
The first capability answer key in the repository we did not write. Every internal bench for the same capability scored 100%; this scored two-thirds. It ships a false-positive control set — deny precedence, resource and condition constraints — which is the half that can go DOWN as detections are added, and the number we watch now that the corpus has told us what to fix.
Almost exactly the AWS figure, from an independent key. Recall only: Rhino publishes no false-positive control set, so this says nothing about specificity and should be read one-sided.
The strongest of the three, because every mapping is execution-proven: for each policy the test builds a violating tenant snapshot, runs the real assessor, and fails if the rule does not fire — an unproven mapping cannot inflate the score. It went 0.32 → 0.75 → 0.99 across successive passes, so it is no longer held out either; what it proves is that the detectors exist and fire, not that a live fetch reaches every setting they read.
Not on this page: any number for the AI agents themselves. Their runs to date were driven through a development proxy rather than a production model key, and a figure produced that way is not one you could reproduce. It goes up here when it is.
What we haven't earned the right to claim.
Everything else we run is either measured on ground truth too thin to generalise from, or not measured at all. Those numbers exist internally; they are not on this page because they would not survive the scrutiny the one above is inviting.
Measured, but not comparable. Two identical scans returned different case membership, so per-class figures carry unquantified run-to-run variance. The durable finding is a class-level one — script-context XSS went undetected in every run — and that is a capability gap we would rather name than average away.
Measured and passing, on ground truth too thin to lean on — three CVEs, nineteen seeded CIS violations, one vulnerability class. A perfect score over three cases is worth less than a middling one over 2,740, and reporting them alike is how a benchmark misleads.
Not measured. The fixtures are stubs awaiting a deployed corpus. We would rather show an empty row than a number nobody produced.
This is the same rule the product runs on. A finding you see has evidence behind it or it does not reach you; a number on this page has a corpus behind it or it does not go up.
See it score your own code.
The scanning engine is free. Connect a repository and you get the same detection this benchmark measures, on your codebase.