Loupe Project: Update #1
643 findings, 70+ fixes, and seven projects asking for more. Loupe's trial run, by the numbers.
Back in May, we announced Project Loupe, an AI-powered vulnerability scanner for bitcoin open-source projects. As part of the trial phase, we onboarded a cohort of seven projects (Bitcoin Core, rust-bitcoin, LDK, BDK, CDK, SRI, and bitcoinj) and started scanning 24 repositories for security-relevant issues and vulnerabilities. Now, after two months, we’re wrapping up the trial run and can share some results.
To help us improve the tool for future cohorts, we collected feedback from all participating projects regarding the utility of the reports, their signal-to-noise ratio, and the number of actionable security issues identified.
In total, we reported 643 findings, of which 354 (55%) were deemed directly actionable. However, only 51 (8%) of findings were considered security-relevant, and 10 (2%) were deemed high severity or above. More than 70 reported issues have already been fixed as of today, and projects indicate they still plan to address more than 100 of the remaining issues in a timely fashion.
Overall, the feedback from the seven participating projects indicates that Project Loupe has been highly valuable, resulting in an average usefulness score of 4.71 out of 5. All participants expressed a strong desire to continue the scanning program. While the report quality was generally praised for identifying actionable security issues, participants noted several triage challenges. Specifically, the signal-to-noise ratio was occasionally affected by findings in test-only code or unsupported modules. While security-specific findings were a smaller portion of the total reports, participating projects highlighted that they derived significant utility from the reports, finding value in actionable insights even when those findings that weren’t immediately categorized as security-relevant.
Looking ahead, we are focused on refining our approach to improve the signal-to-noise ratio. We already utilize de-duplication for findings, ensuring that follow-up runs are cleaner and less redundant. We aim to tune our prompts further to sharpen our focus on security-critical issues. Additionally, we intend to experiment with providing our agents access to more granular, project-specific context, which we expect will enhance both the accuracy and the diagnostic depth of our future reports.
—
Stay up-to-date with Project Loupe. Follow us on X.