Create Farcaster Postgres mirror

I needed a copy of Farcaster raw data for a new project building on top of the network. There were some hiccups for me to get started, so I've compiled a script.

I replicated the Farcaster data into a new DB in AWS and am running this script in an EC2 instance. It can stream data from any Linux server to any Postgres DB (fly.io, Supabase, etc).

This is not a script to run a hub to support the Farcaster decentralization, because it downloads data and does not sync anything back into the network.

the script

# install basic tools
sudo yum install git tmux htop

# install nvm 
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

# activate nvm
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"

# install node 
nvm install 20
npm install --global yarn

# get farcaster repo
git clone https://github.com/farcasterxyz/hub-monorepo.git

# configure script to use DB
export POSTGRES_URL=postgres://USER:PASSWORD@URL/DATABASE

# go to directory
cd hub-monorepo/packages/hub-nodejs/examples/replicate-data-postgres

# install
yarn install 

# run
yarn start

If you want to restart the sync after stopping or rebooting:

## go here after new session or restart 

# configure script to use DB
export POSTGRES_URL=postgres://USER:PASSWORD@URL/DATABASE

# go to directory
cd hub-monorepo/packages/hub-nodejs/examples/replicate-data-postgres

# get changes to repo
git pull

# run
yarn start

Notes

• the Supabase free tier has not enough storage to get a full data copy

• AWS details: EC2 instance is t2.small and smallest possible RDS postgres instance