Holistic Security: Managing Complexity and Trust

In my role as a technical partner at Ethereal Ventures, I've been spending a day each week acting as a security advisor to our portfolio companies.

Previously, as a security auditor at Consensys Diligence, I'd spend weeks at a time auditing clients' code. These audits would take place during a brief but crucial moment in a product's lifecycle - usually, just before a big launch. As a result, my goal was usually to catch the biggest, show-stoppiest, TVL-riskingest bugs. When you're a week away from launch, focusing on anything else would be a distraction.

Now, I'm faced with a different challenge: fitting security into something I can do in a single day.

My "clients" are different, too. Rather than auditing a completed product just before it launches, many of the portfolio companies I work with are in the early stages of development. This often means their code and documentation are rapidly changing, instead of the frozen codebase an auditor expects.

Initially, this combination of "one day per week" and "rapidly changing code" felt daunting.

One of the first portfolio companies I worked with, "Airplane," was in these early stages of development. Product launch was at least 6 months out, and nothing was set in stone. Airplane was in the middle of heavy iteration - there was lots of code being written and daily discussions around product direction.

With so much in flux, focusing on code review wasn't yielding many results. Bugs I found would become irrelevant a week later as features were added, removed, and changed. So after a few advisory sessions with Airplane, I developed a routine: spend the morning reading through small portions of new code and documentation, and spend the afternoon chatting with their engineers.

Although I wasn't doing as much code review, I was getting to know Airplane (and its engineers) through these regular conversations.

Airplane was a rare breed. Unlike many early-stage projects, Airplane's development felt more like cutting-edge research. They'd put together a team of skilled engineers and had an ambitious multi-year roadmap. My questions about product direction would spark impassioned discussion, with Airplane's engineers chomping at the bit to push all the limits of their project.

Their smart contracts were modular, upgradable, decentralized, permissionless... and highly complex.

High complexity isn't inherently wrong, but it needs to be managed properly. This is a challenge many teams face: one that's common for teams in Airplane's position.

A complex smart contract is a lot like a fighter jet: tons of interdependent parts that must function together perfectly, or risk catastrophic consequences. Deploying a protocol with several complex smart contracts is akin to putting on an air show, where your jets are flying together in unison with only a hair's breadth separating them. Also, the DPRK is throwing rocks at your planes.

As a general principle, I recommend taking your first plane out for a test flight before putting on a full air show. Here are the three main things I wish more projects would do as they launch:

1. Roll out features in stages

Your V1 launch should be a simpler, pared-down version of your final product. Additionally, capping your project's TVL can help minimize risk and allow you to identify issues and other pain points early, before they become show-stopping catastrophes.

It's much easier to develop and deploy new features once you have a solid foundation to build on top of.

2. Upgrade early and often

The road to functional DAO governance is paved with multisigs.

Upgrading early and often ensures your signers are experienced and that they understand their role in the upgrade process. Over time, you'll naturally accumulate safer upgrade procedures and testing methodologies.

And hopefully, because of this practice, you'll have a better idea of how to contact your signers in an emergency. Because above all else:

3. Prepare for the worst case scenario

It's important to acknowledge that you can't fix everything, your auditors can't catch every bug, and you'll never have the time to do all the testing you need to do. Things will go wrong, and when they do, every second counts.

If you're following the list so far, you have a multisig in place that can perform upgrades. Given how security-critical upgrades are, that multisig should require several signatures to execute a transaction. Finally, each of your signers should keep their signing key on a hardware wallet in a secure physical location.

This is all well and good, but if there's an active incident in your smart contracts, relying on this setup alone means you need 3/5/7+ people in different timezones to be at their computer at a moment's notice.

Frankly, this isn't realistic. And it's why I highly recommend implementing an "emergency pause."

The emergency pause role is held by a separate multisig. It requires only a single signature to execute a transaction, and the only ability it has is to immediately halt all activity in your contracts. Your upgrade multisig must be used to unpause the contracts.

The goal of the emergency pause is to give yourself time to calmly and correctly react and respond to an incident. Because the scope of the role is highly limited, you can allow more of your team members to hold the role, ensuring you're never caught waiting for signers when you need them most.

I worked with Airplane on these three major goals over several advisory sessions.

Although I was still getting used to working with projects for a day per week, Airplane's willingness to engage with security at an early stage of development made all the difference. It meant they were more flexible and therefore more capable of handling the type of broad feedback I'd always wanted to deliver as an auditor.

Since a code review minimized approach worked so well with Airplane, I was keen to explore even more options for working with portfolio companies. So, when "Banana" approached me with an odd request, I jumped at the opportunity.

Banana was nearing a beta release. They hadn't been audited yet, but were getting quotes from multiple firms and generally gearing up for a future push to production. Banana planned to use a whitelisted beta to onboard users onto their platform and work out any kinks in the system before going public. They were still working on a handful of small projects, but the bulk of their product had been built and wouldn't be changing too much.

Banana was unique because they were prepared with a specific request. They wanted me to interview their project leads, then prepare a report on their software development practices, vulnerability escalation procedures, and key person risk.

This was exciting: I'd never had an opportunity to advise on security solely based on interviews I'd be conducting. I spent the morning piecing together an interview template with ChatGPT, and then an hour each with Banana's project leads.

Through the lens of interviews, I formed a mental model of Banana's product and team that I've never had during an audit. By asking similar questions to multiple people, it became clear where Banana's teams differed from each other and how these differences tied into the big picture.

For example, users were relying on Banana's frontend to deploy contracts that would ultimately handle a large amount of funds. In other words, each user would get their own personal escrow: a "Banana address" to which they would send funds. This wasn't like a typical DeFi project, where users could check that they were interacting with a specific, well-known, widely-used address. Instead, they had to trust that the frontend-generated Banana address was valid.

Wanting to interrogate this trust, I asked each team member to provide details on their GitHub repository's access controls. Many repositories required multiple approvals to push updates. However, updating Banana's website required the approval of only a single developer.

Ultimately, I concluded that there were several potential attack vectors. Lenient access controls on the frontend repository meant there were multiple potential targets for a social engineeering attack. On top of this, there was always a risk that an underlying software dependency could be compromised to similar effect, otherwise known as a supply chain attack.

If an attacker was able to use these vectors to tamper with the frontend and give users a malicious Banana address, there would be no way for the average user to know. In this scenario, the attacker could wait until as many users as possible had deposited funds into malicious contracts, then make off with the funds. A "Banana split," if you will.

This risk is fairly common to projects whose frontends deploy contracts on behalf of their users. And lately, the rise in supply chain attacks, social engineering and phishing attacks, and private key compromise make it all the more important to take risks like these seriously.

For projects in this position, it can be difficult to completely mitigate the risk of a compromised frontend. There are so many variables to consider: software dependencies, access controls, DNS, and malicious links sent to users.

These variables can be addressed somewhat with stricter access policies, intense vetting of dependencies, and more.

However, I feel one of the more effective mitigations is to minimize the role of trust in the product website by educating users to perform their own verification based on chain data. The goal is to reduce the need to rely on your website to tell your users "send funds here." Here are two ideas:

1. Deploy contracts through a known on-chain entry point

When deploying a new contract for each user, the resulting address is, to the user, random data. It's not possible to determine whether the deployment was "correct" from the address itself.

To combat this, contracts should be deployed through a known entry point - a factory contract located at some existing address (preferably an ENS). This way, the user is able to go to etherscan, enter a simple ENS (e.g. deployer.banana.eth), and validate that their deployment details appear in the "Read Contract" tab.

2. Document the process for users

Of course, this wouldn't be complete without documentation. The steps above should be written down and explained in a way that even nontechnical users can follow along, e.g:

• Did I send a transaction to deployer.banana.eth?

• Can I use the ENS name to locate and query the contract on etherscan?

• If I input my address into the "Read Contract" methods, does etherscan confirm the configuration reported by Banana's website?

Bonus points if this documentation has screenshots, is supported by video, and is referenced repeatedly by your documentation, community managers, and anywhere else users might go to learn about your project.

Managing complexity, trust, and user education can't be done well at the last minute. As different as my experiences with Airplane and Banana were, they shared a common thread of identifying and mitigating these risks before they become emergencies.

Rolling out features in stages, planning versions and contract upgrades, preparing for the worst case scenario, creating comprehensive documentation - each of these requires careful consideration and foresight.

As an auditor, I've often wished I could have worked with my clients sooner, as no amount of finding bugs can change the underlying issues in development practices. The efforts Airplane and Banana put in to involve security at an early stage will surely serve them well, and I hope to see other projects follow their lead.

The views expressed here are those of the individual Ethereal Ventures Ltd. (“EV”) personnel quoted and are not the views of EV or its affiliates. Certain information contained in here may haves been obtained from third-party sources, including from portfolio companies of funds managed by EV. While taken from sources believed to be reliable, EV has not independently verified such information and makes no representations about the accuracy of the information or its appropriateness for a given situation.

This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by EV. (An offering to invest in an EV fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by EV, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Ethereal Ventures (excluding investments for which the issuer has not provided permission for EV to disclose publicly as well as unannounced investments in publicly traded digital assets) is available at https://etherealventures.com/portfolio/.

Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://etherealventures.com/disclosures for additional important information.