D-Squared

Subscribe

The best auditing firm does not only find issues but communicates them in a way that the Web3 community can understand. 

Two weeks ago I decided to read my first smart contract audit report, which snowballed into reading an additional 10 reports from different firms. From my experience, I realized that not all audits are created equal and the ability to communicate is critical. How do I define quality “communication”? In this context, it’s being able to walk the fine line between too much and too little, both quantity of text and jargon. The goal of an audit report is to be understood by the majority of readers, so jargon-free writing is the way to go.

Here are the firms and reports I decided to read… 

From these readings, I’ve come up with the ideal structure for an audit report (in my eyes). Before sharing my ideal structure, let’s first cover one major caveat, which is the “perspective” of a reader.  

Perspective – There are two general categories of readers when it comes to smart contract audits, investors and builders. 

  1. Investors: Most investors are not interested, nor capable of diving deep into each issue a firm raises within their report. Their intent when reading is to figure out if this project is one they’re willing to risk their money with. Investors will want to focus their attention on a subset of each report, instead of the entire thing (e.g. exec summary, code base maturity, testing, and documentation). 
  2. Builders: On the other hand builders are interested in either learning or mitigating the issues shared. They intend to understand as much as possible without burning out during the reading process. With that being said, my intent as a “builder” is to uncover common patterns in security flaws, understand how an auditor came to their conclusion, and how I can recreate interesting findings.

My ideal structure is tailored towards “builders”, but I’ll highlight which sections are relevant for “investors” as well. Remember, no matter how solid the structure is, the quality of the writing is the most important.

Audit Report Structure

The smart contract auditing firms in parenthesis () are those that did the best for each section. 

  1. Quality Writing (Paladin and Trail of Bits)
    • Before any structure, we accept that high-quality simplified writing comes before all else. As Malcolm Gladwell would say “Writing should be simple enough that it does not defeat the reader.” 
  2. Exec Summary (Trail of Bits)
    • The Exec summary should tell the reader why this audit report is happening and the most interesting findings. I really appreciated that Chain Security left a thank you note below each of their exec summaries. 
  3. Limitations (Chain Security)
    • The reader should know that a security audit doesn’t automatically make a contract safe, this is for education and to legally cover the auditing firm.
  4. Contract Flow (Consensys Diligence and Chain Security)
    • Diagram: Consensys was the only auditing firm to include a high-level flow for a contract, which I found extremely useful and wish other firms would adopt. Looking at the 10,000-foot overview of a contract is an effective way to quickly improve the reader’s understanding. 
    • Purpose: Chain Security did well at explaining the overall purpose of each contract. The simple question that was answered – what is it used for? 
  5. Automated Testing (Trail of Bits)
    • Trail of Bits is the only audit firm that emphasizes the type and results of its automated testing. Additionally, most audit reports include Echidna test scripts the developer can build on top of with an emphasis on incorporating these into their CI/CD pipelines. 
  6. Code Base Maturity (Trail of Bits)
    • A subjective, but important data point for investors to understand from an expert auditor on how mature this code base is relative to their peers.
  7. Finding Categories (Chain Security and Trail of Bits)
    • Each auditing firm has a similar approach to labeling the severity of a finding, which is “likelihood + impact = severity”. Both Trail of Bits and Chain Security went a step further to bucket their findings into categories, which is a beautiful addition to their structure.
  8. Exploit Scenarios (Trail of Bits)
    • The only auditing firm to explain the exploit scenario for each finding was Trail of Bits, which is a critical piece of information to truly understand how an exploit manifests. As a builder, it’s more convincing to implement a recommendation, when I know exactly how this exploit occurs in my smart contract. 
  9. Recommendation (Trail of Bits and Paladin)
    • Crafting a concise, but specific recommendation is a difficult task, especially when you include short-term and long-term recommendations, but Trail of Bits pulled it off. Paladin was another standout firm able to get straight to the point, with minimal jargon.

 Note: Check the above video for my ideal Frankensteined audit report.  

My Inspiration

While reading broadly about auditing firms and their reports, I came across three quality resources that inspired this wandering through multiple audit reports.