To help us improve the tool for future cohorts, we collected feedback from all participating projects regarding the utility of the reports, their signal-to-noise ratio, and the number of actionable security issues identified.

In total, we reported 643 findings, of which 354 (55%) were deemed directly actionable. However, only 51 (8%) of findings were considered security-relevant, and 10 (2%) were deemed high severity or above. More than 70 reported issues have already been fixed as of today, and projects indicate they still plan to address more than 100 of the remaining issues in a timely fashion.

Overall, the feedback from the seven participating projects indicates that Project Loupe has been highly valuable, resulting in an average usefulness score of 4.71 out of 5. All participants expressed a strong desire to continue the scanning program. While the report quality was generally praised for identifying actionable security issues, participants noted several triage challenges. Specifically, the signal-to-noise ratio was occasionally affected by findings in test-only code or unsupported modules. While security-specific findings were a smaller portion of the total reports, participating projects highlighted that they derived significant utility from the reports, finding value in actionable insights even when those findings that weren’t immediately categorized as security-relevant.

Looking ahead, we are focused on refining our approach to improve the signal-to-noise ratio. We already utilize de-duplication for findings, ensuring that follow-up runs are cleaner and less redundant. We aim to tune our prompts further to sharpen our focus on security-critical issues. Additionally, we intend to experiment with providing our agents access to more granular, project-specific context, which we expect will enhance both the accuracy and the diagnostic depth of our future reports.

—

Stay up-to-date with Project Loupe. Follow us on X.

You’ve heard this a hundred times, making this the one-hundred-and first time, but AI is changing the way we do almost everything. This includes one of its lesser-known, or less acknowledged, applications: finding security vulnerabilities in open-source code. But the kind of vulnerability detector that we at Spiral and Block wanted didn’t really exist, so we built it ourselves: a free “scanning-as-a-service” tool for FOSS bitcoin projects.

It’s called Loupe, an on-the-nose reference to the tool that detects imperfections in otherwise perfect gems.

Subscribe now

Like with writing, most of us are seldom objective about our work, which is why editors exist: to catch what we didn’t and then offer second opinions. But you might not always have access to someone with the time to audit one or more repositories, especially when they contain millions of lines that may impact billions or more in value. For FOSS bitcoin developers, second opinions have never mattered more, regardless of whether they come from a person or an affable robot.

With these types of scanners becoming increasingly relevant, being able to point a machine at your work to find vulnerabilities is something that’s on every bitcoin developer’s mind. Because of how much effort and money are riding on bitcoin’s success, we consider a tool that does this across the ecosystem’s many repositories to be mission-critical.

For us, bitcoin and open-source AI tooling are important public goods. As AI capabilities continue to improve, it’s important to apply them in ways that strengthen the security of the open-source software ecosystem rather than only well-resourced organizations or motivated attackers.

To most, frontier AI models that can detect software vulnerabilities seem like a universal good, one that doesn’t need to be defended, or wouldn’t if not for an obvious asymmetry: attackers can also use these tools to identify weaknesses, while many open-source bitcoin maintainers do not have the time or tools to do the same.

Block and Spiral have each been using AI-based security tools internally to identify vulnerabilities across several open-source bitcoin repositories. When a vulnerability is detected, we report it and any others we’ve found to the project’s maintainers and then work with them to remediate the issue. This will give them exposure to how Loupe works, setting maintainers up to use it themselves when we hand over the reins. Of course, not every project will start off motivated to routinely scan their projects, but by doing it for them, we’ll demonstrate Loupe’s value and necessity.

Loupe, which is designed to scan open-source projects for vulnerabilities, aims to correct this imbalance by staying ahead of adversaries and maintaining the highest security standards throughout bitcoin. Several well-known projects have already committed to being part of our initial tests, including Bitcoin Core, BDK, LDK, rust-bitcoin, Cashu, Blockstre am Jade, bitcoinj, and SRI. We expect to learn a lot from their participation.

There’s a concern in the open-source community that tools like Loupe will flood FOSS Bitcoin projects and maintainers—many of whom are already burdened by AI slop—with low-quality vulnerability reports. We’re aware of this, which is why Loupe will only report vulnerabilities backed by a demonstrable test case, cutting the slop while delivering high-quality reports.

Anyone is free to use Loupe to conduct security scans, but they’ll need to bring their own model access and tokens. The security scans we run ourselves, we’ll fund. The eventual goal is to pass control of Loupe to each project’s maintainers so they can audit independently. To get there, our plan is as follows:

1. Block and Spiral will identify projects interested in having Loupe scan their repositories. Upon completion, we will responsibly disclose any discovered vulnerabilities to the project’s maintainers and seek feedback on how we can improve our outreach process. Then we’ll iterate.

2. Once we’re confident that a robust communication process is in place and we are delivering high-signal, low-noise feedback, we’ll expand security scans to more open-source bitcoin repositories.

3. As projects realize the value of Loupe, we will nudge them to perform the security scans themselves, cutting Block and Spiral out of the loop.

Other best practices for Loupe include how to perform security scanning, how to report results, and what projects can do to optimize scanning (such as providing key information in markdown files). Some projects have multiple repositories that need to be inspected one at a time, which Loupe can accommodate.

In addition, it’s our guess that FOSS bitcoin projects would rather not depend on a specific vendor’s tooling (what happens when Google, OpenAI, Nvidia, SpaceX, and Qwen trade places every month for whose model is best at a given task?), so Loupe contributors will stay current on the LLM landscape, including which models work best for various use cases. Different projects will favor different LLMs, so our plan is to add a tailored Loupe software layer that optimizes for FOSS bitcoin projects on top of these LLMs. This layer will evolve over time in ways that projects like Claude Security might not accommodate.

While this reads like a lot, it’s really not. Loupe’s ultimate goal is to find vulnerabilities, report them, and see them eliminated, making bitcoin safer and bitcoin holders more secure. If you’re interested in having Loupe run a security scan on your project, reach out.

Join our Discord

Contribute on GitHub

Subscribe now