Shared Blob Compression

As appchain adoption grows, there will be an increasing need for chains to opt-into having their blobs aggregated and compressed with blobs from other appchains. Aggregation carries the following benefits:

• appchains can submit transactions to their chosen DA layer more frequently. They no longer need to wait to have enough transaction data to fill a blob before submitting, or pay the cost of submitting partially filled blobs.

• appchains can use blobspace more efficiently, particularly chains with lower tx volume.

• appchains no longer have to worry about scenarios where an L1 block runs out of available blob slots, as their blobs will be aggregated with other appchains.

In addition to aggregating blobs, this would allow appchains to specify additional parameters such as which DA layers they support, which data compression schemes they use, which L1 block they are targeting, and how they choose to update their state. In this post, we explore some possibilities for what a specification would look like.

We don’t intend for this to be a single implementation that all appchains must use. The goal is to create a neutral specification to enable general access to shared blob compression and aggregation. As Spire heavily supports the growth of appchains, we may provide an implementation of this specification to appchains running on Spire.

Appchains wishing to opt into this design can define a list of DA layers, compression schemes, and other fields that they support via json. This format is flexible, and the json can be converted to a binary or uri representation as needed.

For example:

{
		"chain_id": "123", 
    "capabilities": {
        "da_layers": [
            "beacon", "altda1", "altda2"
        ],
        "compression_schemes": [
            "gzip"
        ],
        "shared_state": false, // indicates whether or not the appchain will run its own nodes and derivation pipeline.
        "uniform_blob": false, // indicates whether or not the appchain wants to be in its own blob or a shared blob
        "target_block": 1234567, // The L1 slot in which the appchain’s blob data must be published. This might need to be specified elsewhere
        "tip": 5 gwei // A way for appchains to prioritize inclusion of their blobs in times of high aggregation requests.
    }
}

TBD. This is the information required to retrieve a unit of blob data. Fields may include:

• target DA

• compression scheme

• data length

• data offset

• chain_id/namespace

The layout of an aggregated blob could be a namespaced merkle tree (NMT), with chunks of individual blob data at the leaves, along with a rollup identifier as a namespace. NMTs provide some guarantees that the contents of the blobs will not have been forged.

An alternative approach could use a flat structure with lengths and offsets.

For context, here is a brief overview of how blobs currently work in rollups like Optimism.

Blobs are fetched from the consensus layer in the derivation step of each rollup using the GetBlobSidecars method of the L1 beacon client.

What follows is a hypothetical design of how appchains/rollups could opt in to using a blob aggregator service. After initially registering with the service by sending the above JSON data and receiving a rollup id, blob data could be sent to the aggregator:

In this design, a rollup would:

1. Send their blob data to the off-chain aggregator service, along with commitments to each blob, their rollup/chain-id, supported compression schemes, target DA layer, target L1 block, and potentially some other fields.

2. The aggregator service would then sort incoming blobs by size, target DA, compression scheme, etc. Implementation details can vary.

3. The aggregator service concatenates blobs targeting the same DA into a “super blob”, and generates and stores a NMT for allowing rollups to quickly identify their blobs within the super blob via their rollup-id.

4. The super blobs are sent to their respective DA layers via sidecars. L1 transactions targeting the Blob Aggregator’s inbox contract w/commitments to the super blobs are sent to the L1. Note that it may be necessary for the aggregator to obtain some sort of preconf at this stage, if rollups have specified a target L1 block.

5. Rollups update their state (e.g. derivation pipelines, etc) to recognize batch transactions or proofs coming from the aggregator service address. Blob data relevant to the rollup is fetched from the super blob, either by the rollup itself, or using some endpoint provided by the aggregator service

6. Rollups are responsible for providing a mechanism for confirming that the retrieved blob(s) match the ones they originally sent. For example, providing a commitment to a blob that can be verified with a public key. Additional confirmations may be required. For example, based rollups may need to confirm that the original L2 data in the blob was sequenced by the correct election winner for the L1 block, etc.

7. Rollups then use the unpacked data to update their state and advance the canonical derivation.

This specification places no restrictions on which types of compression are supported. Appchains are free to choose between file and streaming compression schemes.

We assume one compression scheme per root DA specification address.

TBD. One approach could be to simply always select the largest blob that will fit in the remaining space for the current blob.

Given an aggregated blob, appchains are able to efficiently verify and retrieve the sections of the aggregated blob relevant to them using namespaced merkle trees. Espresso uses a similar approach to allow individual rollups to easily derive their blocks from a shared stream of HotShot blocks [ref]. Every appchain will have a unique appchain-id which serves as a namespace identifier.

This specification defaults to using Ethereum’s beacon layer for blob DA. However, rollups are free to choose an alternative DA layer if they need to support faster and cheaper DA not bound to the L1’s fixed block time.

TBD. Fees could be distributed proportionally over rollups based on the percentage of a given blob that they use.

This specification is designed to be extensible. A future extension could involve sending encrypted blobs to help with fair-exchange problems with preconfers.

• Sample implementation of blob aggregation from EthGlobal Istanbull [ref]

• HackMD paper on blob sharing [ref]

• Dankrad’s paper on shard blob header format [ref]

• References to how Espresso enables easy namespace access to rollups [ref] [ref]

Get the latest from Spire:

• X(Twitter): https://x.com/Spire_Labs

• Website: https://www.spire.dev/

• Docs: https://docs.spire.dev/