Detection gaps, measured in the diff.

Chainwatch analyzes the source change between two package releases, then combines six model-scored risk dimensions with external threat feeds. The study asked which layer detected each known incident, how delayed feed coverage affected the result, and whether suspicious-looking benign changes crossed the severity threshold.

All nine malicious-labelled pairs scored above LOW and all nine benign-labelled pairs remained LOW on the final recorded corpus. Those figures describe this eighteen-report corpus only. Several fixes were developed after observing misses inside the same dataset, so the headline result is not a general precision or recall estimate.

The corpus covers six historical npm and PyPI incidents. Where malicious releases had been unpublished or deleted, the study used explicitly documented reconstructions from verified git history, independent archives or recovered payload fragments. Each included pair retains sourcing notes, a findings narrative and a machine-readable pipeline report.

The analysis decomposed every score into model dimensions and feed modifiers. Historical pre-fix runs and diagnostic experiments were retained but excluded from the final eighteen-report denominator, preventing superseded measurements from silently changing the result.

Complete attack artifacts scored materially higher than diluted registry remnants. A complete node-ipc reconstruction reached HIGH while the surviving dependency-mediated registry pair landed at MEDIUM. The ua-parser-js reconstruction reached the corpus maximum after a source-extension gap that hid shell and batch payloads was corrected.

The ctx reconstruction supplied a positive shaped differently from the incidents that drove earlier fixes and cleared the threshold using the original model dimensions with no feed contribution. A recovered Lucide Proxy payload supplied independent support for the resource-exhaustion dimension on an asynchronous polling loop rather than the synchronous loop that motivated it.

The corpus exposed a fetching-completeness problem, a source-file enumeration gap, a missing denial-of-service dimension and an aggregation rule that could bury a near-certain single-dimension signal. Each became a documented code or methodology change with a regression check.

This is evidence that the corpus functioned as an engineering instrument. It is also the central validity limit: changes derived from dataset failures cannot be counted as independent confirmation on that same dataset.

A controlled follow-up found that descriptive attack prose embedded in a diff could materially influence the model score. Replacing narrated placeholders with empty stubs reduced two experimental scores by 20 and 28.5 points, moving both from HIGH to MEDIUM.

The silent stubs still carried structural risk through unexplained install hooks, and famous-incident prior knowledge could not be eliminated. The experiment therefore establishes sensitivity to narrative content in this setup, not a universal estimate of model contamination.

The corpus is small, historically selected and partly reconstructed. Two mechanisms were initially tuned in response to corpus failures; only the resource-exhaustion dimension later received an out-of-corpus mechanism check, while the high-confidence dimension-floor rule remained without an independent firing case.

Threat feeds also change over time, and several final scores depended partly on feed state. The study supports traceable mechanism findings and regression claims, not a population-level security guarantee for npm, PyPI or unseen attacks.

The durable result is methodological: release-diff analysis can expose malicious behavior before or apart from mature signatures, but its validity depends on artifact completeness, transparent score decomposition and explicit controls for model sensitivity.

Future evaluation should freeze a prospective corpus before changing the detector, expand independent benign and malicious examples, and isolate code signal from narrative signal before reporting general performance.