FlawDetector-LLM V1.0: 1,000 repeated trials, and what the variance told us

A single average turns a benchmark into marketing

Security scanner benchmarks usually reduce to two numbers: detection rate and false positive rate. Both typically come from one run, or from the mean of several. The catch is that the pain teams actually feel comes from variance, not from the mean.

If a finding is critical on Monday and gone on Friday, it does not matter whether the tool's detection rate is 94% or 97% — the team stops trusting the output. Experiencing a verdict change without a code change once is enough. So the first thing we fixed in the V1.0 trial design was not "how do we raise the average" but "how do we measure whether identical input yields an identical answer."

Trial design: what exactly was repeated 1,000 times

The unit of repetition is one verdict on one sample, not one pass over the benchmark. We froze the corpus, froze the sampling parameters, replayed identical inputs and measured how often the verdict diverged. A verdict here is a tuple — finding present or not, CWE class, severity — and any one of the three differing counts as a mismatch.

Repetition ran on a stratified sample of 1,200 cases, each replayed 1,000+ times — over 1.2 million verdicts in total. The scale was necessary for a simple reason: once a mismatch rate drops below 1%, 100 repetitions cannot separate it from noise inside any useful confidence interval.

FlawDetector-LLM V1.0 headline results

The 3.2% is not a raw number. The engine's first-pass output had a 5.8% false positive rate; deduplication and reachability-based ranking brought it to 3.2%. We publish both because a post-processed figure on its own makes comparison impossible.

Where the variance lived

The 99.1% aggregate hid a wide spread across finding families. The interesting part of the table below is that detection rate and consistency move together: the families we detect worst are the families whose verdicts wobble most.

The pattern was unambiguous: whether the evidence fits in one chunk explained almost everything. Injection findings assemble the query string inside one function, so the model sees the same evidence every time. Authorisation findings scatter router, middleware and handler across files, and what the model sees depends on where the chunk boundary happened to fall.

Why it wavered

We sampled roughly 10,000 mismatches and classified the cause. Three explanations covered most of them.

1. 01Chunk boundaries (54%) — validation and use split into different chunks. The model judged on whichever half it saw, and which half it saw shifted slightly between runs.
2. 02Context truncation (29%) — in large files, imports and configuration at the top were cut, so the model could not tell which library already provided a defence. This is where "the ORM parameterises this automatically" turned into a raw-SQL false positive.
3. 03Order sensitivity (13%) — the same set of chunks presented in a different order flipped the verdict, most often as a one-step move in the severity label.

What we changed

Three changes, applied in sequence, with the repeated-trial protocol re-run after each one.

1. 01

   Anchored chunking — align boundaries to code structure

   Instead of slicing at a fixed token length, we anchor on function, class and route definitions and force the definitions of referenced symbols into the same chunk. Chunk size becomes variable, but the evidence stops being cut in half. Consistency 96.4% → 98.0%.

2. 02

   Symbol-graph prepass — build the map before reading

   Before any LLM call, a static parser produces a call graph and data-flow summary that is attached as a header to each chunk. The model always knows that a function has three callers and that two of them are external entry points. Consistency 98.0% → 98.7%.

3. 03

   Best-of-three voting — absorb the residual wobble

   For high-severity candidates only, the chunk presentation order is permuted and the verdict is taken three times, then decided by majority. Because it is scoped to candidates, total latency rose by 9%. Consistency 98.7% → 99.1%.

False positives fell as a side effect. Once the prepass made framework defences visible, false positives against code using ORMs, template engines and validation middleware dropped noticeably: 5.8% → 4.1% on first-pass output, 3.2% after post-processing.

The remaining bottleneck, and V1.1

Consistency was bought with latency. Adding the prepass changed the shape of a scan.

The prepass output cannot change unless the file changes — it is a cacheable computation that V1.0 redoes on every run. Incremental caching is the most certain win in V1.1, and the payoff scales with how often you rescan, which means CI benefits most.

Five questions to ask of any security benchmark

Applied to our numbers as much as anyone else's, these five questions filter out most overstatement.

• Are repeated trials reported? With a single run, the reproducibility of the number is simply unknown.
• Is the false positive rate pre- or post-processing? Publishing only the post-ranking figure is common practice, but comparison needs the raw one.
• How large is the clean-code control group? Vulnerable samples alone cannot measure a false positive rate.
• Is a synthetic mutation set included? Public corpora alone cannot separate memorisation from detection.
• Is there a per-family breakdown? One aggregate average hides which finding families are weak.

In summary

The most useful output of the V1.0 trials was diagnosis, not performance. Where the variance clusters is where the next version has work to do, and finding that location requires decomposed numbers rather than an average. We spent 1.2 million verdicts not to obtain a good number but to learn where to aim.

Every item is maintained as an in-house test specification, and the full report — corpus composition and repetition protocol included — is available on request.