X.js benchmark finds Node.js vulnerabilities are migrating into larger packages
A research paper posted to arXiv introduces X.js, a live benchmark for Node.js taint-style vulnerabilities, and reports that the median vulnerable package grew from roughly 250 lines of code in 2021–2022 to 6,100 lines of code in 2026. The paper, authored by researchers at INESC-ID/IST in Lisbon and Carnegie Mellon University, presents X.js as a response to the need for up-to-date collections of real vulnerabilities, noting that the most commonly used datasets stop recording vulnerabilities in 2022.
X.js was built from 1,738 advisories collected through mid-2026, primarily from the GitHub Advisory Database. For each vulnerability, the benchmark records precise source and sink locations, a validated executable exploit with its execution trace, and a patched version of the package when available. The current release contains 1,159 validated exploits covering 965 packages. To ensure long-term reproducibility, dependencies are mirrored in a self-hosted registry, protecting against package removal or version disappearance from the public npm registry. Dedicated infrastructure generates and validates exploits for newly reported vulnerabilities as they arrive.
The most striking finding concerns where vulnerabilities now appear: they are increasingly reported in large packages, with the median vulnerable package growing from roughly 250 LoC in 2021–2022 to 6,100 LoC in 2026.
The X.js pipeline has four main components: an advisory collector that ingests from the GitHub Advisory Database; a PoC creator that generates candidate exploits; a validation layer that checks exploit correctness through source-sink analysis, effect checking, capability analysis, and additional constraints; and a metadata gatherer that records package versions and patch states. Data flows from public advisory feeds into this pipeline, which outputs validated exploit entries with execution traces.
The design decision enabling this benchmark's existence is the self-hosted registry mirror. Public package ecosystems like npm are designed for availability to installers, not for historical preservation. Packages can be unpublished by authors or removed for policy violations, and version resolution depends on the current state of the registry. X.js's designers chose to freeze dependencies locally, trading the convenience of always-fresh upstream packages for the guarantee that an exploit validated today will still execute identically in future evaluations. This perenniality mechanism costs storage and maintenance labor, paid by the benchmark maintainers, in exchange for scientific reproducibility.
The concentration of vulnerabilities in larger packages follows from how npm's network effects reward package growth. Small utility packages accumulate dependents and merge functionality, or get absorbed into larger collections. The trade-off is between coverage and scalability: static analysis tools often sacrifice soundness for speed on large codebases, while dynamic analysis tools face combinatorial path explosion. Tool builders are rewarded for publications showing detection on benchmark datasets, but if the benchmark itself evolves toward larger packages without tool designers noticing, the evaluation gap widens silently.
The open question is whether vulnerability detection research can adapt its methods faster than the packages grow. If the median target continues expanding at this rate, tools optimized for the 2022 scale will face a different problem domain without necessarily knowing it, because the outdated datasets they were trained and tested on gave no signal of the shift.