Evidence before verdict.

RepoAuditor combines deterministic scanners with model-assisted mapping, detection and falsification. Its validation work focused on whether each stage exposes enough provenance to tell a completed scan, an empty result, a missed mechanism and an unresolved candidate apart.

The records support the execution and review controls of the v0.1.0 workflow. They also show material coverage limits in public rule sets and in narrow learned ranking results. No project-wide real-world precision or recall claim is supported.

Pinned repositories and manufactured controls supplied bounded answer keys. Scanner outputs retained tool version, configuration, target count, finding count, output validity and run status. Candidates outside the reviewed target set remained unadjudicated instead of being counted as false positives.

Model-backed findings required source-resolving citations and entered a bounded falsification loop. Human decisions remained append-only and unresolved requests blocked final analysis. Evaluation-only commands were prevented from changing production findings or verdicts.

An initial Semgrep round scanned twenty-six fixture targets under 290 applicable rules. It recovered two of three reviewed static-code positives, SQL injection and SSRF, missed the reviewed IDOR mechanism, and raised both reviewed kill controls. Raw rule hits were not treated as deduplicated findings.

On a pinned OWASP Juice Shop snapshot, the sole checked-in reviewed SQL-injection target was recovered among fifty-one raw candidates. The remaining fifty candidates stayed unadjudicated because the answer key was not exhaustive.

A separate acquisition round tested four nonstandard historical CVE mechanisms. Semgrep Community recovered none of the four designated targets even though it emitted seventy-seven raw candidates on both pre-fix and post-fix snapshots. The result identified rule-coverage limits rather than proving that every candidate was invalid.

Deployment canaries later verified that zero-result runs could be classified as complete, empty, not applicable, partial, unavailable or failed with supporting execution evidence. A successful process exit alone was no longer accepted as a clean scan.

One prospective ranking check reported perfect classification on its narrow selected set, but all seven positives belonged to mutable GitHub Actions references. The record explicitly limits that result to rule-family separation and rejects cross-mechanism generalization.

The falsification convergence harness varies retrieval breadth and decomposition depth while holding the subject and model configuration fixed. It reports verdict flips, citation overlap and confidence spread as method-sensitivity evidence; it does not automatically confirm or kill a finding.

The reviewed answer keys are intentionally partial, model calls are not seed-reproducible on the tested transports, and several checks use manufactured controls. Aggregate candidate counts cannot establish finding identity, exploitability or recall.

The records validate operational distinctions and bounded mechanisms. Wider claims require larger held-out, family-diverse, human-adjudicated datasets with prospective protocols and repeatability measurement.

The strongest finding is procedural: security-review evidence is safer when execution coverage, semantic recovery, falsification and adjudication remain separate fields. This prevents a tool from converting an empty or ambiguous result into assurance by omission.

RepoAuditor’s current record supports a review workflow and its fail-closed boundaries. It does not support autonomous security verdicts or population-level performance claims.