Quilibrium Blog
Cover photo

A Taste of the Future

Cassandra Heart

Cassandra Heart

Farcaster Frames was released to much fanfare yesterday, and on the surface the deceptively simple idea has already yielded incredible creativity, with everything from MUDs, to polls, to NFT mints. And the number of folks ideating over the potential was even larger. Then along came this post:

post image

Challenge accepted.

post image

So naturally of course, people learned two things very quickly:

  1. Frames can be animated

  2. Q has a metavm project

And boy did that generate some questions. So what I'm setting out to do in this article is circle back on my post from three days ago, where I spoke about showing what the future of computing and crypto will look like very soon. For those who got to try the demo out before the VPS host piping the command data over decided to nerf the vCPU and not respond to tickets (Big shout out to Vultr, you're a garbage service), congrats – you just got your first taste of that future I was describing.

So in this article, I'm going to describe how it works, and how I was able to build it in only two hours.

Farcaster Frames use meta tags in a style very similar to opengraph, enhancing it with custom button texts and submission urls that can be signed by users on Farcaster, presenting a basic loop:

post image

That frame data includes an image url to display, which is expectedly rendered in an <img> tag. But browsers support a lot of image formats, and so I remembered an old trick used by webcams back in the day: MJPEG. With MJPEG, you can hang onto a request indefinitely until the user closes it, and send back raw JPEG images one by one. This meant I didn't need special video support, I could just use an endpoint that returned MJPEG as a response for the frame image url.

Now I just needed to test it, so I built a simple Next.js API handler that kept an open connection and streamed to all active connections the same frame as it was rendered. I started simple, and just made a setInterval loop that loaded six images from local fs and sent them every 500ms:

import type { NextApiRequest, NextApiResponse } from 'next';
import { join } from 'path';
import { createCanvas } from "canvas";

let frame = 0;
const clients: any[] = [];

export const images = [
  fs.readFileSync(join(process.cwd(), 'images/image1.jpg')),
  fs.readFileSync(join(process.cwd(), 'images/image2.jpg')),
  fs.readFileSync(join(process.cwd(), 'images/image3.jpg')),
  fs.readFileSync(join(process.cwd(), 'images/image4.jpg')),
  fs.readFileSync(join(process.cwd(), 'images/image5.jpg')),
  fs.readFileSync(join(process.cwd(), 'images/image6.jpg')),
];

export default async function handler(this: any, req: NextApiRequest, res: NextApiResponse) {
  var headers: any = {};
  var multipart = '--mjpeg';

  headers['Cache-Control'] = 'private, no-cache, no-store, max-age=0';
  headers['Content-Type'] = 'multipart/x-mixed-replace; boundary="' + multipart + '"';
  headers.Connection = 'close';
  headers.Pragma = 'no-cache';

  res.writeHead(200, headers);

  const ref = {
    mjpegwrite: (buffer: any) => {
      res.write('--' + multipart + '\r\n', 'ascii');
      res.write('Content-Type: image/jpeg\r\n');
      res.write('Content-Length: ' + buffer.length + '\r\n');
      res.write('\r\n', 'ascii');
      res.write(buffer, 'binary');
      res.write('\r\n', 'ascii');
    },
    mjpegend: () => {
      res.end();
    },
  };

  var close = function() {
    var index = clients.indexOf(ref);
    if (index !== -1) {
      clients[index] = null;
      clients.splice(index, 1);
    }
  };

  res.on('finish', close);
  res.on('close', close);
  res.on('error', close);

  clients.push(ref);
}

export const mjpegsend = (buffer: any) => {
  for (var client of clients)
      client.mjpegwrite(buffer);
};

setInterval(() => {
  mjpegsend(frames[frame]);
  frame++;
  frame %= 6;
}, 100);

Loading the API handler url in the browser worked swimmingly. Ok, great, so then the next challenge: make it run Doom.

Luckily for me, I already solved this problem in the research around Quilibrium. One of the advanced features that will be launched later this year is the metaVM, which translates instruction set architectures into an executable format usable by the network, along with many other important components to support a fully functioning VM. The metaVM supports a basic framebuffer device which is IO mapped to RAM at a specific location. The VM translates to a choice of execution calls: durable – on the hypergraph, and therefore somewhat slower, or ephemeral – does not store execution state, and is merely piped over. Supporting keyboard and mouse inputs work similarly with hardware interrupts. Finally, the file system itself is fulfilled with a virtio-9p compatible application, which translates R/W requests for inodes into hypergraph calls. Together, you get a fully distributed virtual machine with optional durability at multiple levels. This, despite sounding rather complicated, is quite simple to implement on Q, and looks a lot like a traditional emulator when you dive into the code.

So then the remaining tasks became only the following remaining items:

  • Handle inputs from the buttons and send them back as key down/key up events

  • Build the framebuffer worker and kick it off on start of the Frame server

  • Convert the framebuffer data into JPEGs

  • Deploy a filesystem map compatible with the metaVM virtio-9p driver with Linux with Doom to the hypergraph

Execution state updates with the RPC client can be streamed directly from metaVM, so we carved it out to the section of RAM containing the framebuffer, and directly invoked the interrupts for keyboard inputs. That's the first two down.

Node doesn't have a clearcut way to quickly convert buffers to JPEGs, and I wanted to hack this together quickly, so I used node-canvas to serve as the render target for the raw image data, then used canvas.toBuffer('image/jpeg') to create the image. Publishing the buffer data over the worker, the message handler on the API side then only needs to directly call the mjpegsend(buffer) method defined above. Next one down.

For the last one, I had a bit of a cheat here, in that I already built this filesystem map a while ago to demo metaVM (hi friends on Unlonely!) and the QConsole. That's all the work needed done.

Architecturally, the frame integration then looks like this:

post image

And there you have it: Doom on Frames.

Collect this post as an NFT.

Quilibrium Blog

Subscribe to Quilibrium Blog to receive new posts directly to your inbox.

Over 2.2k subscribers

Ryan J. ShawFarcaster
Ryan J. Shaw
Commented 9 months ago

IMPORTANT ANNOUNCEMENT: We can play @cassie's DOOM again! It seems MJPEG streams started working again - maybe when WC switched to CF for image proxying? What's next? ScummVM port? Remote controlled robots/webcam streams? Tx @samuellhuber.eth for making me doublecheck this

Ryan J. ShawFarcaster
Ryan J. Shaw
Commented 9 months ago
Ryan J. ShawFarcaster
Ryan J. Shaw
Commented 9 months ago

Turns out it only works on web, not on mobile 😭

tricilFarcaster
tricil
Commented 9 months ago

hell yeah

CARDELUCCI🎩Farcaster
CARDELUCCI🎩
Commented 9 months ago

👀

Andre Messina Farcaster
Andre Messina
Commented 9 months ago

For quite a while, we were able to upload gifs up to 15mb. It seems within the last week this has changed back to 10mb. I’ve tested this with multiple gifs previously worked.

Cassie HeartFarcaster
Cassie Heart
Commented 9 months ago

ScummVM already works, it's just really limited because you only get four buttons

Ryan J. ShawFarcaster
Ryan J. Shaw
Commented 9 months ago

Yeah I figure we can only do the early games, right - 4x cursor keys and text entry (any text input + button == enter). I think that was the "AGI" engine.

Cassie HeartFarcaster
Cassie Heart
Commented 9 months ago

got a game in mind?

memes4airdropFarcaster
memes4airdrop
Commented 9 months ago

ScummVM 👌

alecFarcaster
alec
Commented 9 months ago

@remindbot 3 days

BlockheimFarcaster
Blockheim
Commented 9 months ago

new with frames, what are some of the best ones I should try out?

jp 🎩Farcaster
jp 🎩
Commented 9 months ago

building this onboarding one plz 88 $degen https://warpcast.com/jpfraneto/0x60320267

LeoFarcaster
Leo
Commented 9 months ago
Brock The BreadcatFarcaster
Brock The Breadcat
Commented 9 months ago

save real orcas from this frame: (tips work too but don't tip my frame here, tip the @wavewarriors one!). https://donate.framesframes.xyz/api?frameId=1

Brock The BreadcatFarcaster
Brock The Breadcat
Commented 9 months ago

$BORED made a love letter to Oregon Trail with Boregon Trail! https://boredbored.framesframes.xyz/api

jackjack.base.eth 🛰️Farcaster
jackjack.base.eth 🛰️
Commented 9 months ago

Yo @blockheim, try out Base Name Service Frame App. https://frame.basename.app/api

SidFarcaster
Sid
Commented 9 months ago

If in the mood for degeneracy https://warpcast.com/tybb/0x9ee1b16a

Corbin PageFarcaster
Corbin Page
Commented 9 months ago

We categorize more all popular Frames here: https://www.degen.game/frames/featured

BlockheimFarcaster
Blockheim
Commented 9 months ago

oh wow, this is a great starting point. Thank you!

BstractFarcaster
Bstract
Commented 9 months ago

Love this 1 $degen

ComplexlityFarcaster
Complexlity
Commented 9 months ago

You could swap tokens directly on arbitrum https://warpcast.com/complexlity/0x06011240

Ferran 🐒Farcaster
Ferran 🐒
Commented 9 months ago

Create an e2e verifiable Frame poll with Farcaster.vote!

Cassie HeartFarcaster
Cassie Heart
Commented 1 year ago
Cassie HeartFarcaster
Cassie Heart
Commented 1 year ago

whoops, forgot to update the endpoint update matching, one sec

Cassie HeartFarcaster
Cassie Heart
Commented 1 year ago

the improved framerate is revealing that we need an action button, @v pls let us have more buttons 🥺

Cassie HeartFarcaster
Cassie Heart
Commented 1 year ago

Reminder for folks who missed it the first time around: this does not work on mobile, visit it on web

Phil CockfieldFarcaster
Phil Cockfield
Commented 1 year ago

💪⚡️

elleFarcaster
elle
Commented 1 year ago

yes! i finally got to see it in action (o.o) amazing!

Darryl Yeo 🛠️Farcaster
Darryl Yeo 🛠️
Commented 1 year ago

ruh-roh

Cassie HeartFarcaster
Cassie Heart
Commented 1 year ago

yeah sorry about that, fixed now

Darryl Yeo 🛠️Farcaster
Darryl Yeo 🛠️
Commented 1 year ago

👌

Darryl Yeo 🛠️Farcaster
Darryl Yeo 🛠️
Commented 1 year ago

It's wild to me that you can stream a live video feed to an <img> tag by using the MJPEG file format.

BenFarcaster
Ben
Commented 1 year ago

my favorite frame so far tbh

AdamFarcaster
Adam
Commented 1 year ago

Of all the frame experiments that I've seen, this is the one I'm still thinking about days later. Would love to hear/read a full run down of how you felt this experience went, as well as the edge cases you came across in the process. Its a huge learning experience for many & your insight has a lot of value Cassie

Cassie HeartFarcaster
Cassie Heart
Commented 1 year ago
AdamFarcaster
Adam
Commented 1 year ago

insightful as always🙏

Victor🎩🚴↑Farcaster
Victor🎩🚴↑
Commented 1 year ago

@launch Gaming onframe

LaunchcasterFarcaster
Launchcaster
Commented 1 year ago
horsefacts 🚂Farcaster
horsefacts 🚂
Commented 1 year ago

me: wow, I made the picture show a little flag! @cassie: https://paragraph.xyz/@quilibrium.com/doom-on-frames

horsefacts 🚂Farcaster
horsefacts 🚂
Commented 1 year ago

me: guess the only way to solve this one is to pay $900 to Vercel @cassie:

Syed Shah🏴‍☠️🌊Farcaster
Syed Shah🏴‍☠️🌊
Commented 1 year ago

Darryl Yeo 🛠️Farcaster
Darryl Yeo 🛠️
Commented 1 year ago

Built different

Guapolocal.eth🎩Farcaster
Guapolocal.eth🎩
Commented 1 year ago

NGL this workflow chart is way underrated alpha. 69 $degen

Victor🎩🚴↑Farcaster
Victor🎩🚴↑
Commented 1 year ago

333 $DEGEN

Cassie HeartFarcaster
Cassie Heart
Commented 1 year ago

How did I put Doom on Farcaster Frames with only two hours of work? I told you I'd show you what the future of crypto looks like very soon, this was just the teaser of what abandoning the blockchain looks like. https://paragraph.xyz/@quilibrium.com/doom-on-frames

Michael PfisterFarcaster
Michael Pfister
Commented 1 year ago

This is incredible

Darryl Yeo 🛠️Farcaster
Darryl Yeo 🛠️
Commented 1 year ago

I was today years old when I learned about the MJPEG image format. Pretty cool! https://en.wikipedia.org/wiki/Motion_JPEG

@Timshel  (mee.fun)Farcaster
@Timshel (mee.fun)
Commented 1 year ago

tip 69 $degen to @cassie heck ya

Lord Dalresin🐝Farcaster
Lord Dalresin🐝
Commented 1 year ago

This is amazing! I just discussed with with my teammate and I think we can implement something like this. Any help with getting into Qulibrium or source code would be amazing. I am going to take a look at the docs. 10 $degen!

floar.eth ツ🎩🔵⚪️ Farcaster
floar.eth ツ🎩🔵⚪️
Commented 11 months ago

Gray read 500 $degen

0xbhaisaabFarcaster
0xbhaisaab
Commented 11 months ago

amazing, what are the limitations here, though? how many image frames can I send? or is there a timeout?

dhadrien.ethFarcaster
dhadrien.eth
Commented 1 year ago

hey loved what you did there, thanks for sharing! dumb quenstion, was playing with your code sample, what the client side looks like within nextjs? using the api route as img source within nextjs make it interpret it as html and 404

dhadrien.ethFarcaster
dhadrien.eth
Commented 1 year ago

nevermind fixed it, had nothing to do with client side, had to update route code

PhoenixFarcaster
Phoenix
Commented 1 year ago

Amazing.

kliulessFarcaster
kliuless
Commented 1 year ago

Oh so you were the dev who did this. Awesome ty for this write up!

MennoFarcaster
Menno
Commented 1 year ago

TIP unlimited $DEGEN to Cassie

A Taste of the Future