Random Mastodon Setup/Migration Thoughts

Like seemingly many people, I recently started experimenting with hosting a Mastodon instance (mastodon.hockey, in this particular case).

When I started, because I was considering it “just” an experiment, I decided to take the path of (nearly) least resistance and fired up a droplet over at DigitalOcean using their 1-Click app.  That gave me a good opportunity to play with the software a bit and decide that I wanted to do more than just experiment.

While I was experimenting, an upgrade to the Mastodon software was released.  I attempted to upgrade and hit some roadblocks based on me not knowing exactly what was installed where on that droplet.  I decided not to upgrade at that time, since I was still experimenting.

Eventually I determined that this wasn’t going to be a throwaway thing.  If I was going to continue maintaining the instance, I wanted to know exactly what was installed where, so I decided to rebuild the server from scratch.

And…  As long as I was doing that, I’d move it from DigitalOcean to AWS.  All my other stuff was at AWS so it just made sense that if I wasn’t going to be taking advantage of the 1-Click anymore, there was no need to be off at DigitalOcean (not that I had any problems with DigitalOcean’s service, to be clear).

One thing I would not have to migrate was uploaded media.  As I said, I took the path of (nearly) least resistance when I first set up mastodon.hockey, the “nearly” accounting for having dumped media off to an S3 bucket deployed via CloudFront.  As such, the following setup and migration notes wouldn’t have to account for that.

Before doing this I did see a pretty awesome “how-to” on getting Mastodon set up in the AWS ecosystem.  That one, however, assumes that you’re going all-in, with load balancing and RDS and ElastiCache.  Maybe that will be my next step.  For this, however, I decided to do a more one-to-one migration – one droplet to one EC2 instance.

I should probably note that this project led me on a whole other side-quest of reorganizing my AWS properties.  Because, as seen by the aforementioned path of (nearly) least resistance, sometimes I can’t stop myself from adding complications.

After that side quest, I made a couple attempts at this that failed miserably.  I won’t document exactly what went wrong there but the great thing about all these cloud-based resources is that when something goes wrong, it’s easy to just trash it and start over.

EC2 Instance

I started by creating the EC2 instance.  I used Ubuntu 22.04 with a 64-bit arm processor.  Since I’ve got hardly anyone using this (as of writing, there are 42 users, five of which are accounts that I own for various projects), I started small with a t4g.micro instance.

I also added a couple security groups that I already had in place; one that had permission for my home IP address to connect via SSH and another that allowed the world to connect via HTTP/S.

Elastic IP Address

I hopped over to Elastic IPs, allocated one, and assigned it to the newly-created EC2 instance.

I’m honestly not sure what would happen without a static IP address.  Restarting the instance would get me a new IP but I think I could alias the DNS records to point at the EC2 instance directly, so the change wouldn’t matter.  Maybe SSL certs would be a problem?

I’ll admit that my AWS-foo is weak.  It may no longer be strictly necessary but I expect a web server to have a dedicated IP so I gave it one.

General Server Setup

After SSHing into the server, I ran sudo apt update, just to get that out of the way.  I also added 2GB of swap using the following commands:

sudo fallocate -l 2G /swapfile

sudo chmod 600 /swapfile

sudo mkswap /swapfile

sudo swapon /swapfile

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

sudo apt install systemd-zram-generator

I edited /etc/systemd/zram-generator.conf to set zram-fraction = 1 and compression-algorithm = zstd.

This was necessary to solve some issues with compiling Mastodon assets.

I’d thought about editing /etc/apt/apt.conf.d/50unattended-upgrades to enable unattended upgrades (Unattended-Upgrade::Automatic-Reboot “true”) but decided that, since this instance does have users other than me, I shouldn’t do that.

I gave it a restart, then I moved on to the actual requirements.

Install Node.js

I chose to install Node.js via a NodeSource  PPA because I needed v16 and NPM comes pre-packaged this way.

curl -sL https://deb.nodesource.com/setup_16.x -o /tmp/nodesource_setup.sh

sudo bash /tmp/nodesource_setup.sh

sudo apt install nodejs

Install Yarn

Yarn is a package manager for Node.js.  Supposedly it comes pre-packaged with modern versions of Node but I wasn’t seeing it so I just installed it myself.

curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null

echo "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

sudo apt-get update && sudo apt-get install yarn

Install PostgreSQL

Postgres is the database behind Mastodon.  There’s some more setup that happens later but getting the initial install done is pretty simple.

sudo apt install postgresql postgresql-contrib

Install Nginx

Nginx is the web server used by Mastodon.  There’s a ton of setup after getting Mastodon itself installed but getting Nginx installed is another single-line command.

sudo apt install nginx

Add Mastodon User

The mastodon software expects to run under a “mastodon” user.

sudo adduser mastodon

sudo usermod -aG sudo mastodon

The first command prompts for a password and additional user details.

Add PostgreSQL User

Mastodon also needs a “mastodon” user in Postgres

sudo -u postgres createuser --interactive

It should have the name “mastodon” and be assigned superuser access when prompted.

Install Mastodon Dependencies

There are a handful of packages that Mastodon depends on that need to be installed before we get into working with Mastodon itself.

sudo apt install imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev libprotobuf-dev protobuf-compiler pkg-config redis-server redis-tools certbot python3-certbot-nginx libidn11-dev libicu-dev libjemalloc-dev

Switch to Mastodon User

From here on out, we want to be operating as the newly-created “mastodon” user.

sudo su - mastodon

This puts us in the /home/mastodon/ directory.

Clone Mastodon Code from Git

Mastodon’s code lives in a Git repository.  This pulls that code down and gets us working with the correct version.

git clone https://github.com/mastodon/mastodon.git live

cd live

git checkout v4.0.2

Install Ruby

Mastodon is currently requiring v3.0.4 of Ruby so we explicitly get that version.

sudo apt install git curl libssl-dev libreadline-dev zlib1g-dev autoconf bison build-essential libyaml-dev libreadline-dev libncurses5-dev libffi-dev libgdbm-dev

curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash

echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc

echo 'eval "$(rbenv init -)"' >> ~/.bashrc

echo 'export NODE_OPTIONS="--max-old-space-size=1024"' >> ~/.bashrc

source ~/.bashrc

rbenv install 3.0.4

rbenv global 3.0.4

Installing Ruby takes a minute.

We also need the “bundler” Ruby gem.

echo "gem: --no-document" > ~/.gemrc

gem install bundler

bundle config deployment 'true'

bundle config without 'development test'

bundle install

Install Javascript Dependencies

Mastodon requires that Yarn be set in “classic” mode.

sudo corepack enable

yarn set version classic

yarn install --pure-lockfile

Mastodon Setup

At long last, I was finally ready to actually set up the Mastodon instance.  I started by allowing SMTP connections, so that I could send a test message during the setup process.

sudo ufw allow 587

After that, I ran the Mastodon setup process.

RAILS_ENV=production bundle exec rake mastodon:setup

Since I was migrating from an existing Mastodon instance, I partially used dummy data here.  Specifically, I used a different S3 bucket than I was using in production so that it wouldn’t overwrite any live data.

I said “yes” to preparing the database and to compiling assets.

Then I created an admin user.  The only real reason to do that was to give something to test with after finishing the setup and before transferring the existing instance data over.

Configure Nginx

With Mastodon configured, it was time to configure Nginx to actually serve up those files via the web.  I started by opening the server up to HTTP and HTTPS traffic.

sudo ufw allow 'Nginx Full'

I’d been thinking that I could get away with Nginx HTTPS for that setting but wasn’t accounting for how Certbot requires HTTP access, which I would have run into a few steps later had I not caught it here.  I started having a bad feeling about my plan at this point, which came to a head a little bit later.

Next up was adding Mastodon to the Nginx configuration, which was easy thanks to a file that just needed to be copied over from the Mastodon install and a symbolic link that needed to be set up.

sudo cp /home/mastodon/live/dist/nginx.conf /etc/nginx/sites-available/mastodon

sudo ln -s /etc/nginx/sites-available/mastodon /etc/nginx/sites-enabled/mastodon

That file doesn’t have all of the correct configuration, though.  I opened it up in a text editor and updated the server_name values to “mastodon.hockey” (I would have also needed www.mastodon.hockey but I don’t have DNS set up for that [“www” on a gTLD just looks weird to me]).   I also needed to comment out the “listen 443” lines because I didn’t have an SSL cert yet.

At this point I realized what that bad feeling was.  There was no way I could get the SSL cert for mastodon.hockey without moving the domain over to this box.  Expecting it to fail, I ran the cert request command anyway.

sudo certbot certonly --nginx -d mastodon.hockey

Had I been using the “www” domain, I would have also needed to request that here.  It wouldn’t have mattered, though, because after entering my email address and agreeing to terms and conditions, the request failed, as expected.

If I were using Elastic Load Balancer, the certificate would be handled on the AWS side of things and I wouldn’t need Certbot.  I could have updated my DNS to answer the challenge regardless of where the domain was actually pointed.  But I wasn’t going that route.  I decided to move forward and add a step for requesting the certificate later.

I restarted Nginx to at least put the changes I had made into use.

sudo systemctl reload nginx

Set Up Mastodon Services

There are three services to enable to get Mastodon working.

sudo cp /home/mastodon/live/dist/mastodon-*.service /etc/systemd/system/

sudo systemctl daemon-reload

sudo systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streaming

I also added a couple weekly cleanup  tasks to the mastodon user’s crontab

47 8 * * 2 RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove
37 8 * * 3 RAILS_ENV=production /home/mastodon/live/bin/tootctl preview_cards remove

These could run at any time.  They don’t even have to be weekly.

Migrate Existing Data

If this had just been an initial setup, I’d be about ready to go.  Because I was migrating, though, I needed to take care of a handful of other things.

First, I updated my Mastodon config so that the secrets matched those on my old instance.  This made it so that admin account I set up earlier wouldn’t work but I wasn’t worried about that anymore.

On my original instance, I disabled the Mastodon services and ran a database backup

sudo systemctl stop mastodon-{web,sidekiq,streaming}

pg_dump -Fc mastodon_production -f backup.dump

I copied that backup file over to the new instance, stopped the services there, deleted and recreated the database (since it was junk at this point anyway), restored the backup, and (because the backup was from an older version of Mastodon) ran the database migration script.

sudo systemctl stop mastodon-{web,sidekiq,streaming}

dropdb mastodon_production

createdb -T template0 mastodon_production

pg_restore -Fc -U mastodon -n public --no-owner --role=mastodon -d mastodon_production /home/mastodon/backup.dump

RAILS_ENV=production bundle exec rails db:migrate

I updated the DNS to point to the new instance, then ran the Certbot commands from above to get an SSL certificate.

sudo certbot certonly --nginx -d mastodon.hockey

Then I went back into the Nginx config to un-comment the “listen 443” lines and update the location of the certificate files.  I also went back to the Mastodon config to update it to point at the correct S3 bucket for file storage.

With that done, I confirmed that the Nginx config was valid, restarted Nginx, brought the Mastodon services back online, rebuilt the user home feeds, and deleted the outdated database backup.

sudo nginx -t

sudo systemctl reload nginx

sudo systemctl start mastodon-{web,sidekiq,streaming}

RAILS_ENV=production ./bin/tootctl feeds build

rm /home/mastodon/backup.dump

Then I noticed that the site still wasn’t being served up properly due to permissions issues, which was an easy fix.

chmod o+x /home/mastodon

Wrap Up

At this point, mastodon.hockey was up and running on the EC2 instance but I still had some cleanup to do.

I left the Digital Ocean droplet up and running for a bit longer, just in case I needed something from it.

I also had some custom tools I’d built that needed to migrate over to the EC2 instance.  I manually copied them over and left myself with a future project of updating the deployment process for them to account for the move.

I’m not certain I did all of this the “right” or “best” way.  I was learning as I went, though, and learning is important.