Quilibrium Blog
Cover photo

Quil Builder's Guide

Cassandra Heart

Cassandra Heart

post image

The explosion in popularity around Quilibrium is truly one of the most awe inspiring moments to me – I've been working on the technology and iterating on it for several years at this point, so it's amazing now that so many share the same excitement I've had this whole time. One particular question I've gotten from people is usually followed by another: "What can I build on it?" and then "How can I build on it?" I intend this guide to answer both questions, but it starts with the same answer I've given since I started work on the latest iteration of the MPC runtime over two years ago: just write go.

What can I build on it?

Philosophically, Quilibrium's mission of securing every bit of traffic on the Internet is the kind of mission that is an infectious one, ergo, the end goal is that the answer to that question is: anything and everything. But pragmatically, no library, service, protocol or application develops traction without having reasonable initial use cases its better suited for. A great example of this is Postgres – traditionally a standard SQL-based RDBMS, but is Swiss army knife-d to the point of supporting not only many other practical forms of data stores, but also entirely different domains: message queues, scheduled tasks/workers, and with the right plugins, even as unexpected things as web servers.

Architecturally, it should be no surprise then that Quilibrium's most intuitive base use cases revolve around the manner in which data lives on the network: fetches and stores of structured data. How does this naturally fit into the most "obvious" (in this case, to someone who's been building it for seven years) emergent patterns?

Global Object Storage (S3)

Storing raw data in buckets is something that almost every application needs:

  • Single Page Applications use it for hosting

  • Content store for social media (long-form posts, images, etc.)

  • Package repositories store many discrete, immutable bundles

  • Backups of a client-facing application's data

This list would get impressively exhaustive if fully enumerated, so it's easier to describe in parallels – what people use S3 for, would work well very simply on Q.

Key-Value Stores (Elasticache/Redis)

Even for applications not needing to handle storage of large objects typically do need some state stored, and the storage approach equally scales down into the discrete key-value level. The network is maintaining a hypergraph as the storage medium, and most computer science students learn and prove for themselves in their studies how easily translated graphs and KV stores are between one another.

Data isn't the only driver for utility, though, and there are very compelling use cases to be had as a fabric for compute itself. The hypergraph is an intrinsic: a term used in Q development that somewhat parallels a "precompile" in Ethereum parlance, or as directly borrowed from compiler theory the notion of functional intrinsics – hardware-optimized opcode substitution, like substituting an entire matrix multiplication routine with a single opcode that handles it for far fewer CPU cycles. Intrinsics are a property of the execution engine of Quilibrium, for the purpose of doing things more efficiently with MPC than "the hard way" by explicitly compiling a routine into a garbled circuit. There are many realizations of this model in the wild – entire families of threshold signature algorithms exist for ECDSA because there are mathematical "shortcuts" that do the arithmetic faster with the same level of security without having to build out the full ECDSA in garbled circuits. Being able to compose intrinsics together with garbled circuits in intuitive ways is the core strength of Quilibrium's execution engine, and by doing that, it enables other categories of applications implicitly:

Discrete Functions (Lambda)

Code is data, and deploying code to be executed on Quilibrium is a matter of simply storing it following the structured format expected for code. From a tooling perspective this is largely invisible, it is simply a deploy command. And the structure of the deployment object is the compiler's output – essentially like an intermediary language suited for garbled circuits. The code itself? For now, with other languages to be added later, is basically Go – things like syscalls, nondeterministic functions, certain types of I/O are not supported, but a large subset of the language itself is, making it very easy to port over large amounts of application code with minimal change.

Key Management Service

With a straightforward medium for MPC applications, lending itself towards "obvious" conclusions, MPC-based signing is an easy demonstration of the network's unique value propositions. It is indeed so easy that it will be the first example in this document.

So, that being said:

How can I build on it?

When I've been saying "just write go", there's obviously the small caveat aforementioned, but we have a cohesive example of a token application on our respective Labs page that shows more broadly a use of weaving in both the intrinsics of the hypergraph and additional intrinsics like the augmented EdDSA. But to illustrate the simplicity of saying "just write go", let's sketch a very simple MPC signer:

package main

import (
	"crypto/ed448"
)

type PartyInput struct {
	message   [64]byte
	keyShare  [57]byte
}

func main(a, b PartyInput) []byte {
	equal := true
	for i := 0; i < len(a.message); i++ {
		// verifies both parties agreed to sign the message
		equal &= a.message[i] == b.message[i]
	}

	var signature [114]byte

	if !equal {
		// returns an empty signature
		return signature
	}

	var key [57]
	for i := 0; i < len(a.keyShare); i++ {
		key[i] = a.keyShare[i] ^ b.keyShare[i]
    }

	return ed448.Sign(key, a.message, "")
}

There are a few small differences from regular go in this small example that are worth calling out:

  • The main function declares the parties for MPC (instead of bastardizing os.Args)

  • crypto/ed448 isn't an official library – but it is added to the built in libraries for this language given its heavy presence on Q

  • The arguments to the main function must be explicitly sized

Explicit Sizing

That last one is probably the most important observation when building applications for Q – you can use structs for your input variables between parties, but you must size them for the garbler. There are a few minor edge cases, but this is the most important one. This variant of Go also enables something that Go 2.0 might do (given Rob Pike's proposal): arbitrarily sized integers. If you need a four bit int: use int4, if you need it to be 2048 bits: use int2048, if you need a three bit uint: use uint3.

Important Caveats

The most crucial thing for building an MPC application is to not produce behaviors that can leak inputs: early termination (note how we run through all bytes of the message which both parties must agree to and know before we return the failure condition), conditional termination (note how we don't take a public key value and verify the private key corresponds to it – that could leak bits of the private key)

Packages

Most applications do not consist of a single main entry point function and nothing else – this is no different. You can deploy code as entrypoint functions or as packages which you can include in your code, just like regular Go.

Storing/Retrieving Data

Interacting with the hypergraph itself is also extremely straightforward, using a small collection of types that the provided intrinsics can wire together into parallel fetch/store calls for you. To illustrate, let's take the more advanced example from the Token lab for the CreateTransaction function:

func main(request CreateTransactionRequest, relay hypergraph.Network) (hypergraph.UpdateExtrinsic, hypergraph.UpdateExtrinsic, hypergraph.CreateExtrinsic, bool) {
  coinBytes, ok := hypergraph.RetrieveExtrinsic(request.OfCoin)
  if !ok {
    return hypergraph.UpdateExtrinsic{}, hypergraph.UpdateExtrinsic{}, hypergraph.CreateExtrinsic{}, false
  }
  coin := UnmarshalCoin(coinBytes[:1088])

  acctBytes, ok := hypergraph.RetrieveExtrinsic(coin.OwnerAccount)
  if !ok {
    return hypergraph.UpdateExtrinsic{}, hypergraph.UpdateExtrinsic{}, hypergraph.CreateExtrinsic{}, false
  }
  acct := UnmarshalAccount(acctBytes[:89])

  msg, err := hypergraph.HashExtrinsics([3]hypergraph.Extrinsic{request.OfCoin, request.RefundAccount, request.ToAccount})
  if err != "" {
    return hypergraph.UpdateExtrinsic{}, hypergraph.UpdateExtrinsic{}, hypergraph.CreateExtrinsic{}, false
  }

  verify := sign.Verify(acct.PublicKey, msg, request.Signature, []byte{})
  if !verify {
    return hypergraph.UpdateExtrinsic{}, hypergraph.UpdateExtrinsic{}, hypergraph.CreateExtrinsic{}, false
  }

  acctExt := coin.OwnerAccount
  updatedAccount := Account{
    TotalBalance: acct.TotalBalance - coin.CoinBalance,
    PublicKey: acct.PublicKey,
  }
  updatedAcctBytes := MarshalAccount(updatedAccount)
  updatedAcctExt := hypergraph.Update("account:Account", acctExt.Ref, updatedAcctBytes)

  coinExt := request.OfCoin
  updatedCoin := Coin{
    CoinBalance: coin.CoinBalance,
    OwnerAccount: hypergraph.Empty(),
    Lineage: bloom.Apply(coin.Lineage, acctExt.Ref),
  }
  updatedCoinBytes := MarshalCoin(updatedCoin)
  updatedCoinExt := hypergraph.Update("coin:Coin", coinExt.Ref, updatedCoinBytes)

  pendingTransaction := PendingTransaction{
    ToAccount: request.ToAccount,
    RefundAccount: request.RefundAccount,
    OfCoin: coinExt,
  }
  pendingTransactionBytes := MarshalPendingTransaction(pendingTransaction)
  pendingTransactionExt := hypergraph.Create("pending:PendingTransaction", pendingTransactionBytes)

  return updatedAcctExt, updatedCoinExt, pendingTransactionExt, true
}

There's a lot going on here, so let's break it down into pieces. First, the function declaration:

func main(request CreateTransactionRequest, relay hypergraph.Network) (hypergraph.UpdateExtrinsic, hypergraph.UpdateExtrinsic, hypergraph.CreateExtrinsic, bool) {

When you are writing a function that any node on the network can process (i.e. there is no secret material expected from another party), the convention is to set the second argument to relay hypergraph.Network. The CreateTransactionRequest object is a struct with some additional annotation courtesy of the schema repository functionality of the network:

type CreateTransactionRequest struct {
  ToAccount hypergraph.Extrinsic `rdf:"create:ToAccount,extrinsic=account:Account"`
  RefundAccount hypergraph.Extrinsic `rdf:"create:RefundAccount,extrinsic=account:Account"`
  OfCoin hypergraph.Extrinsic `rdf:"create:OfCoin,extrinsic=coin:Coin"`
  Signature [114]byte `rdf:"create:Signature"`
}

Let's ignore the RDF annotation for this article as it's more relevant when getting deeper into the schema related functionality of the network, and observe three fields of type hypergraph.Extrinsic. An extrinsic on the hypergraph is something like a pointer – it is a referential address to data on the network. You can see the immediate relevance in producing fetch and store calls in the method that uses it:

  coinBytes, ok := hypergraph.RetrieveExtrinsic(request.OfCoin)
  if !ok {
    return hypergraph.UpdateExtrinsic{}, hypergraph.UpdateExtrinsic{}, hypergraph.CreateExtrinsic{}, false
  }
  coin := UnmarshalCoin(coinBytes[:1088])

Handling fetch failures – either produced by data that does not exist or data the requestor has no key to access results in the simple value, ok pattern common in Go like the map type. The return value of the function has two important characteristics, and it shows in the failure case here. When a function fails, the output structs (encoded as actions on the hypergraph: Create/Update/Delete) are empty, and the final value of the return tuple is the boolean false. Functions on Quilibrium essentially take two forms:

  • Direct MPC (arguments are all active parties)

  • Open 2PC (the second argument is the relay)

In the first case, the final boolean argument is not necessary. In the second case, it is.

The Unmarshal* and Marshal* functions are automatically generated convenience functions from the schema repository, but you can think of them as simple byte mappers of the stored extrinsic to the struct you wish to interact with.

  acctExt := coin.OwnerAccount
  updatedAccount := Account{
    TotalBalance: acct.TotalBalance - coin.CoinBalance,
    PublicKey: acct.PublicKey,
  }
  updatedAcctBytes := MarshalAccount(updatedAccount)
  updatedAcctExt := hypergraph.Update("account:Account", acctExt.Ref, updatedAcctBytes)

To update the Account extrinsic, the underlying Account struct can be seen being synthesized, byte packed with the MarshalAccount function, and finally the hypergraph.Update function is called, serializing the storage into a hypergraph.UpdateExtrinsic type provided as one of the return values. Note that while fetches appear to happen at any time in the code, they all happen before the circuit is actually evaluated. Similarly, while it appears as if the value is being updated in this hypergraph.Update call, the actual settlement occurs at the end. In this respect, the common safety pattern of Checks-Effects-Interactions that gets violated in building smart contracts, leading to many of the vulnerabilities within, is completely eliminated, because checks become the basis of the executed function, effects and interactions happen after the function's conclusion and order is always well-formed. There are downstream consequences with respect to reentrancy and concurrency, but this is a deeper topic for another article.

  return updatedAcctExt, updatedCoinExt, pendingTransactionExt, true

The function's conclusion is merely returning the update and create extrinsic actions to be processed, along with a boolean status indicating the function was successfully evaluated.

Takeaways

There are some very powerful technologies behind Quilibrium, and it can make the idea of developing for such a network feel very daunting for newcomers. The simplicity of the developer experience utilizes all of this heavy lifting being done at the protocol level in a way that results in an application development process that should feel very comfortable for many non-crypto-oriented developers, and by consequence, this introduction hopefully makes it clear that to build for Quilibrium, the only thing you really need to do is

just write go

Collect this post as an NFT.

Quilibrium Blog

Subscribe to Quilibrium Blog to receive new posts directly to your inbox.

Over 2.2k subscribers

treethoughtFarcaster
treethought
Commented 10 months ago

excited to share tofui, the Terminally On Farcaster User Interface! Right now it requires a @neynar application, but also working on hosting it as an ssh app https://github.com/treethought/tofui/raw/main/media/demo.gif https://github.com/treethought/tofui

Michael PfisterFarcaster
Michael Pfister
Commented 10 months ago

love this

depatchedmodeFarcaster
depatchedmode
Commented 10 months ago

Whoa 1000 $degen

jp 🎩Farcaster
jp 🎩
Commented 10 months ago

im so happy to see the reception this had, amazing work. congratulations. are you familiar with quilibrium? i think of you and come back to this article. there may be something brewing here 1618 $degen https://paragraph.xyz/@quilibrium.com/quil-builders-guide

treethoughtFarcaster
treethought
Commented 10 months ago

thanks man and really appreciate your interest in it! and i've been meaning to dig into quilibrium for a few months now, but haven't gotten around to it, especially after seeing that blog post (haven't read it yet). But it seems super cool, i've been fascinated with MPC since the ico mania days when i first learned about it. will definitely be doing some research this weekend!

GramajoFarcaster
Gramajo
Commented 10 months ago

Fucking ay this is dope AF 1000 $DEGEN

Shashank Farcaster
Shashank
Commented 10 months ago

this is very cool. thanks for open sourcing this

treethoughtFarcaster
treethought
Commented 10 months ago

Thanks! Castsense looks awesome by the way

Shashank Farcaster
Shashank
Commented 10 months ago

2958 $DEGEN

SteveFarcaster
Steve
Commented 10 months ago

This was on my never ending list of things to build and I’m glad I didn’t have to lol 😂 this is awesome!!

mike perhatsFarcaster
mike perhats
Commented 10 months ago

Now that’s my kind of FC client.

TonyFarcaster
Tony
Commented 10 months ago

will it run on a local hub? 1000 $degen nice work

treethoughtFarcaster
treethought
Commented 10 months ago

thanks! it currently requires a neynar application and won't work with a local hub. but definitely want to add support for local hubs once i get the basics into a good state.

HarrisFarcaster
Harris
Commented 10 months ago

what APIs did you need from neynar to get the relevant data?

MetaEnd🎩Farcaster
MetaEnd🎩
Commented 10 months ago

What exactly do i nees from neynar? An api key?

treethoughtFarcaster
treethought
Commented 10 months ago

Need an api key and client id. Running tofui init will show urls to get them

MetaEnd🎩Farcaster
MetaEnd🎩
Commented 10 months ago

ah, I guess that's not free. any plans to move to public hubs/apis? 200 $DEGEN 🎭 ⚡

Samuel ツFarcaster
Samuel ツ
Commented 10 months ago

wanna develop on Quilibrium? get prepared! https://paragraph.xyz/@quilibrium.com/quil-builders-guide

Cassie HeartFarcaster
Cassie Heart
Commented 10 months ago

And now for something different than the ongoing GitHub nonsense: https://paragraph.xyz/@quilibrium.com/quil-builders-guide

RoyalFarcaster
Royal
Commented 10 months ago

If I want to play around and do some simple examples where do I go? My guess would be the ceremony repo but would love some simple directions on where and how to get started, either here or in the blog post 😁

Cassie HeartFarcaster
Cassie Heart
Commented 10 months ago

We have a full build tool repo, although GitHub is an ongoing issue we need to have a conclusion on before making any more updates there, so I’m going to release a local simulator tool so you can experiment all you'd like in a Q like environment next week, will land on https://source.quilibrium.com

HectorFarcaster
Hector
Commented 10 months ago

I really love Go :) !

dycpFarcaster
dycp
Commented 10 months ago

@kimono

killjoy.ethFarcaster
killjoy.eth
Commented 10 months ago

Been waiting for this!!

demipoet 🎩Farcaster
demipoet 🎩
Commented 10 months ago

⚡ What everyone has been waiting for, for a long time - The Quilibrium Builder's Guide https://paragraph.xyz/@quilibrium.com/quil-builders-guide

demipoet 🎩Farcaster
demipoet 🎩
Commented 10 months ago

Thank you @cassie for sharing this so quick. It's not yet even a month since 1.14.17's been released, and made Quilibrium a little bit more known via its wQUIL token in the Eth mainnet. Above and beyond what we can expect

Quil Builder's Guide