ric/mastodon-setup-guide
ric/mastodon-setup-guide
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

initial commit

Browse source
This commit is contained in:
Ric Harvey 2024-10-25 17:28:07 +01:00
commit 4684a5c726
Signed by: ric
GPG key ID: 7FDEB4C0C7D5CB61
2 changed files with 669 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

669
README.md Normal file
View file

@ -0,0 +1,669 @@
# Building my Mastodon Setup with Docker
So many admins of [mastodon](https://joinmastodon.org) instances prefer to use a server and install the OS and mastodon straight onto it. I on the other hand prefer to containerize my setup with [docker](https://www.docker.com). I find it neater and easier to upgrade, plus if I want to run other services on the same server I'm free to do so without messing up dependancies or versions. Now there are some ok(ish) guides on how to do this out there but I wanted pull together a few guides in order to produce a complete setup, a [traefik](https://traefik.io) reverse proxy in front of mastodon with full text search and translation enabled using [libretranslate](https://libretranslate.com), I also wanted to use object storage for my assets, in my case I'm running on [AWS](https://aws.amazon.com) so I'm using S3. I also set myself the challenge of removing software with dubious licenses such as Redis and ElasticSearch and opting for the more FOSS friendly [Valkey](https://valkey.io) and [OpenSearch](https://opensearch.org) componants. So here is my setup.
## The Setup
A picture speaks a thousand words so let me show you the main companants of this setup. Each bit of software is running in it's own container:
```mermaid
flowchart TD
Users --> |HTTPS port: 80/443| A
A[Traefik] -->|Web Interface port: 3000| B(mastodon)
A -->|Streaming port: 4000| C(mastodon-streaming)
B --> D(sidekiq)
C --> D
B --> E[postgres]
C --> E
B --> F[valkey]
C --> F
B --> G[opensearch]
B --> H[S3]
B --> I[libretranslate]
Users --> | assets | H
```
Now the easiest way to run all these companants and have them interconnected is to set up the entire system with [docker compose](https://docs.docker.com/compose/) but before we get going we are going to have to set up the basics. Now I built this on a [debian 12](https://debian.org) system but most of the compose setup will be identical, you'll just need to tweak the install commands for docker and compose.
### Install Docker and Compose
1. First lets make sure theres no unoffical packages in place that will conflict by removing them.
``` bash
for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done
```
2. Now lets set up dockers official apt repository:
``` bash
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
3. Now install the tool chain:
``` bash
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin vim
```
4. Verify that the installation is successful by running the hello-world image:
``` bash
sudo docker run hello-world
```
> 👉 Note
>
> If you want to run docker as a none root user you'll need to add that user to the docker group by running the following command:
> ```sudo usermod -aG docker $USER```
### Setup storage directories on the host
I like to try and keep the persistant data neat and tidy on my system so I make sure group my configs and data directories together. For this I store them in */opt/containers/\<application_name>* in this case **mastodon**
- Let's start by creating that folder structure, for ease we are going to use sudo to become root for these commands(you could just prefix all instructions with sudo if you really wanted):
``` bash
sudo su -p
mkdir -p /opt/docker/traefik
mkdir -p /opt/docker/mastodon
cd /opt/docker/mastodon
```
- Now we are in the main directory lets create the directories where the conatiners are going to write their persistant data:
``` bash
cd /opt/docker/traefik
mkdir data
mkdir logs
cd /opt/docker/mastodon
mkdir -p public/system
mkdir postgres
mkdir -p opensearch/data
mkdir valkey
mkdir -p lt/data/{key,local}
mkdir lt/api_keys
```
- Now lets set the permissions on those directories so the containers can write their data to them without having to tweak the containers:
``` bash
chown -R 991:991 public
chown -R 70:root postgres
chown -R 1000:root opensearch
chown -R 999:root valkey
chown -R 1032:1032 lt
```
### Setting up traefik
To access your site and ensure you have a valid SSL certificate we are going to run traefik as a reverse proxy, this means we can also force all traffic to be HTTPS which is great for security. Traefik will automtically take care of getting a cert from [LetsEncrypt](https://letsencrypt.org) and rotating it when its due to expire. All you have to do is provide an email address for registration.
Ensure your in the */opt/docker/traefik* directory and create a new file call *compose.yml*
```bash
vi compose.yml
```
now add the follwoing to the file:
> 💡 Tip
>
> To enter **insert mode** in vim press i
```yaml
services:
traefik:
image: "traefik:latest"
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.routers.traefik.entrypoints=web"
- "traefik.http.routers.traefik.rule=Host(`${HOSTNAME}`)"
- "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_PASSWORD}"
- "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
- "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
- "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
- "traefik.http.routers.traefik-secure.entrypoints=websecure"
- "traefik.http.routers.traefik-secure.rule=Host(`${HOSTNAME}`)"
- "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
- "traefik.http.routers.traefik-secure.tls=true"
- "traefik.http.routers.traefik-secure.tls.certresolver=myresolver"
- "traefik.http.routers.traefik-secure.service=api@internal"
# Define the port inside of the Docker service to use
- "traefik.http.services.traefik-secure.loadbalancer.server.port=8080"
env_file: .env
ports:
- "80:80"
- "443:443"
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /opt/docker/traefik/logs:/logs:rw
- /opt/docker/traefik/data/acme.json:/acme.json:rw
- /opt/docker/traefik/data/traefik.yml:/traefik.yml:rw
- /opt/docker/traefik/data/config.yml:/config.yml:rw
networks:
- proxy
networks:
proxy:
external: false
```
Save and exit the file.
> 💡 Tip
>
> To get out of vim just type ```:wq!```
You're also going to need to set up some environment variables for this to work so once you've saved the file above you'll need to create a new file called *.env*
```bash
vi .env
```
Populate it with these details, and remember to update the domain:
```text
HOSTNAME=traefik.example.com
TRAEFIK_PASSWORD=admin:<GENERATED_PASSWORD>
```
You'll need to generate a password for traefik basic_auth to be able to login to the dashboard API. you can do that with the following command:
```bash
echo $(htpasswd -nb user password) | sed -e s/\\$/\\$\\$/g
```
Paste the output user:password keypair into your *.env* file and save and exit.
Theres a couple of other files (4 actually) we now need to prep. 3 are going to be empty files that the container will need and one will be the initial config startup for traefik.
```bash
cd /opt/docker/traefik/data
touch dynamic-config.yml
touch acme.json
cd /opt/docker/traefik/logs
touch traefik.log
```
Now for the important file to pull all this together and get traefik working.
```bash
vi traefik.yml
```
Enter the following information and update your email address:
```yaml
api:
dashboard: true
debug: true
entryPoints:
web:
address: ":80"
websecure:
address: ":443"
log:
level: ERROR
filePath: "/logs/traefik.log"
format: common
serversTransport:
insecureSkipVerify: true
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedbydefault: false
file:
filename: dynamic-config.yml
certificatesResolvers:
myresolver:
acme:
email: <YOUR_EMAIL_ADDRESS>
storage: acme.json
httpChallenge:
# used during the challenge
entryPoint: web
```
Now traefik is ready to run and accept HTTP and HTTPS connections. To get it up and running is pretty simple with docker compose, run the following command:
```bash
cd /opt/docker/traefik
docker compose up -d
```
#### Test Traefik
To test traefik you can browse direction to [https://traefik.example.com](https://traefik.example.com) where you will be prompted to enter your basic_auth details in your browser.
![ScreenSHot of Basic Auth Prompt](img/basic_auth.png)
### Setup object storage
##### Create a S3 Bucket
##### Generate an IAM key
##### Setup Cloudfront
### Setting up email with SES
##### Setup + Verify Domain
##### Generate SMTP credentials
### Create configuration files
This first file is the all important docker compose file, so lets go ahead and create *compose.yml* and paste in the following content.
``` bash
cd /opt/docker/mastodon
touch .env.production
touch .env.translate
vi compose.yml
```
Now lets paste in the following content:
``` docker
services:
db:
restart: unless-stopped
image: postgres:14-alpine
shm_size: 256mb
networks:
- mastodon
healthcheck:
test:
- CMD
- pg_isready
- -U
- postgres
volumes:
- /opt/docker/mastodon/postgres:/var/lib/postgresql/data
environment:
- POSTGRES_HOST_AUTH_METHOD=trust
valkey:
restart: unless-stopped
image: valkey/valkey:7-alpine
networks:
- mastodon
healthcheck:
test:
- CMD
- valkey-cli
- ping
volumes:
- /opt/docker/mastodon/valkey:/data
web:
image: richarvey/mastodon-bird-ui:latest
restart: unless-stopped
env_file: .env
#command: bash -c "rm -f /mastodon/tmp/pids/server.pid; bundle exec rails s -p 3000"
command: bundle exec puma -C config/puma.rb
networks:
- traefik_proxy
- mastodon
healthcheck:
test:
- CMD-SHELL
- wget -q --spider --proxy=off localhost:3000/health || exit 1
depends_on:
- db
- valkey
- os
volumes:
- /opt/docker/mastodon/public/system:/mastodon/public/system
labels:
- traefik.enable=true
- traefik.http.routers.mastodon.entrypoints=web
- traefik.http.routers.mastodon.rule=Host(`mastodon.squarecows.com`)
- traefik.http.middlewares.mastodon-https-redirect.redirectscheme.scheme=https
- traefik.http.routers.mastodon.middlewares=mastodon-https-redirect
- traefik.http.routers.mastodon-secure.entrypoints=websecure
- traefik.http.routers.mastodon-secure.rule=Host(`mastodon.squarecows.com`)
- traefik.http.routers.mastodon-secure.tls=true
- traefik.http.routers.mastodon-secure.tls.certresolver=myresolver
- traefik.http.routers.mastodon-secure.service=mastodon
- traefik.http.services.mastodon.loadbalancer.server.port=3000
- traefik.docker.network=traefik_proxy
streaming:
image: ghcr.io/mastodon/mastodon-streaming:v4.3.1
restart: unless-stopped
env_file: .env
command: node ./streaming/index.js
networks:
- traefik_proxy
- mastodon
healthcheck:
test:
- CMD-SHELL
- wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health ||
exit 1
depends_on:
- db
- valkey
labels:
- traefik.enable=true
- traefik.http.routers.mastodon-api.entrypoints=web
- traefik.http.routers.mastodon-api.rule=(Host(`mastodon.squarecows.com`)
&& PathPrefix(`/api/v1/streaming`))
- traefik.http.middlewares.mastodon-api-https-redirect.redirectscheme.scheme=https
- traefik.http.routers.mastodon-api.middlewares=mastodon-api-https-redirect
- traefik.http.routers.mastodon-api-secure.entrypoints=websecure
- traefik.http.routers.mastodon-api-secure.rule=(Host(`mastodon.squarecows.com`)
&& PathPrefix(`/api/v1/streaming`))
- traefik.http.routers.mastodon-api-secure.tls=true
- traefik.http.routers.mastodon-api-secure.tls.certresolver=myresolver
- traefik.http.routers.mastodon-api-secure.service=mastodon-api
- traefik.http.services.mastodon-api.loadbalancer.server.port=4000
- traefik.docker.network=traefik_proxy
sidekiq:
#image: richarvey/mastodon-bird-ui:latest
image: ghcr.io/mastodon/mastodon:v4.3.1
restart: always
env_file: .env
command: bundle exec sidekiq
depends_on:
- db
- valkey
- os
networks:
- mastodon
volumes:
- /opt/docker/mastodon/public/system:/mastodon/public/system
healthcheck:
test:
- CMD-SHELL
- ps aux | grep '[s]idekiq 6' || false
os:
restart: always
image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- node.name=opensearch-node1 # Name the node that will run in this container
- discovery.type=single-node
- bootstrap.memory_lock=true # Disable JVM heap memory swapping
- DISABLE_SECURITY_PLUGIN=true
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m # Set min and max JVM heap sizes to at least 50% of system RAM
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=<OPENSEARCH_PASSWORD>
networks:
- traefik_proxy
- mastodon
#healthcheck:
#test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
volumes:
- /opt/docker/mastodon/opensearch/data:/usr/share/opensearch/data
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
ports:
- 9200:9200
- 9600:9600
libretranslate:
image: libretranslate/libretranslate:v1.6.1
restart: unless-stopped
healthcheck:
test:
- CMD-SHELL
- ./venv/bin/python scripts/healthcheck.py
env_file:
- .env
volumes:
- /opt/docker/mastodon/lt/data:/home/libretranslate/.local
- /opt/docker/mastodon/lt/api_keys:/app/db
networks:
- traefik_proxy
- mastodon
ports:
- 5000:5000
networks:
mastodon:
external: false
traefik_proxy:
external: true
```
> 👉 Note
>
> There are a few variables you'll need to setup in the above script to match your environment and secure it, make sure you change these values.
>
> - \<POSTGRES_PASSWD> \*1
> - \<URL_MASTODON_URL> \*4
> - \<OPENSEARCH_PASSWORD> \*1
Once you've changed these variables you'll need to save and exit vim.
### Setting up the environment variables
Now to make this easy I've included a skeleton *.env.production* file for you to copy and populate with your own variables. But first we need to generate some keys and setup the DB.
#### Generate keys for mastodon
First we'll need to generate some keys you'll need to secure mastodon and you'll need to run the following command **twice** once for the SECRET_BASE_KEY and once for OTP_SECRET.
> 👉 Note
>
> make a note of them you'll need them for the config file
```bash
docker compose run --rm -- web bundle exec rails secret
```
Now we need to generate the VAPID_PRIVATE_KEY.
> 👉 Note
>
> yet again make a note of this output
```bash
docker compose run --rm -- web bundle exec rails mastodon:webpush:generate_vapid_key
```
#### Initialize the database
Now we need to start setting up the database ready for mastodon to use. Run the following command which will create the DB and populate the initial tables:
```bash
docker compose run --rm -- web bundle exec RAILS_ENV=production rails db:setup
```
#### Generate DB AR Encryption Values
Since mastodon 4.3.0 you'll also need to set up Active Record (AR) encryption keys. The following command will generate you the required keys.
```bash
docker compose run --rm -- web bundle exec RAILS_ENV=production rails db:encryption:init
```
> 👉 Note
>
> Make a note of these values as we'll need them for the configuration file.
#### Setup Libre Translate
Mastodon allows you to translate posts into your native language by using libretranslate. You can run this service locally in your stack with very little setup. Let's look at the steps
##### Create the environments file
```bash
vi libretranslate.env
```
and enter the following details, you don't need to edit anything in this file:
``` text
LT_DEBUG=true
LT_UPDATE_MODELS=true
LT_SSL=true
LT_SUGGESTIONS=false
LT_METRICS=true
LT_API_KEYS=true
LT_THREADS=12
LT_FRONTEND_TIMEOUT=2000
#LT_REQ_LIMIT=400
#LT_CHAR_LIMIT=1200
LT_API_KEYS_DB_PATH=/app/db/api_keys.db
```
##### Run the container
Now lets start the container ready to set up the required files and generate an API key.
```bash
docker compose exec -it lt bash
```
##### Install all the language packs
We now need to download the language packs for libretranslate to use.
```bash
for i in `/app/venv/bin/argospm list`;do /app/venv/bin/argospm install $i;done
```
> 👉 Note
>
> Note: this will take a fair bit of time so please do be patient
##### Generate an API key
Mastodon will need an API key to access libretranslate. Still within the container lets create an API key that can make 120 requests per min:
```bash
/app/venv/bin/ltmanage keys add 120
```
> 👉 Note
>
> Make a note of the output as this is the API key
### Create the mastodon environment variables
We now need to gather those credentials we've just created and edit our *.env.production* file. Open that file and copy the below contents into that file and change the values in the **< >**.
``` bash
vim .env.production
```
Contents:
``` text
LOCAL_DOMAIN=<URL_MASTODON_URL>
SINGLE_USER_MODE=false
SECRET_KEY_BASE=<GENERATED_SECRET_KEY_BASE>
OTP_SECRET=<GENERATED_OTP_SECRET>
VAPID_PRIVATE_KEY=<GENERATED_VAPID_PRIVATE_KEY>
VAPID_PUBLIC_KEY=<GENERATED_VAPID_PUBLIC_KEY>
DB_HOST=db
DB_PORT=5432
DB_NAME=postgres
DB_USER=postgres
DB_PASS=<DB_PASSWORD>
REDIS_HOST=valkey
REDIS_PORT=6379
# This is blank on purpose
REDIS_PASSWORD=
S3_ENABLED=true
S3_PROTOCOL=https
S3_BUCKET=<AWS_S3_BUCKET>
S3_REGION=<AWS_REGION>
S3_HOSTNAME=<s3.AWS-REGION-1.amazonaws.com>
AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY>
AWS_SECRET_ACCESS_KEY=<AWS_SECRET_KEY>
# Only needed if you use cloudfront
S3_ALIAS_HOST=<CLOUDFRONT_URL>
SMTP_SERVER=<EMAIL_SERVER>
SMTP_PORT=587
SMTP_LOGIN=<EMAIL_USERNAME>
SMTP_PASSWORD=<EMAIL_PASSWORD>
SMTP_AUTH_METHOD=plain
SMTP_OPENSSL_VERIFY_MODE=none
SMTP_ENABLE_STARTTLS=auto
SMTP_FROM_ADDRESS=<EMAIL_ADDRESS>
ES_ENABLED=true
ES_HOST=http://os
ES_PORT=9200
ES_PRESET=single_node_cluster
ES_USER=admin
ES_PASS=<OPENSEARCH_PASSWORD>
ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=<GENERATED_AR_KEY>
ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=<GENERATED_AR_SALT>
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=<GENERATED_AR_PRIMARY_KEY>
LIBRE_TRANSLATE_ENDPOINT=http://libretranslate:5000
LIBRE_TRANSLATE_API_KEY=<LT_API_KEY>
```
## enable admin
```bash
docker compose exec web /bin/bash
```
```bash
tootctl accounts deploy -approve <ADMIN_NAME>
```
## enable opensearch
```bash
docker compose exec web /bin/bash
```
```bash
tootctl search deploy
```
## get rid of yellow cluster warning
```bash
docker compose exec os /bin/bash
```
```bash
curl -X PUT -u admin:<OPENSEARCH_PASSWORD> "http://os:9200/_settings" -H 'Content-Type: application/json' -d'{
"index" : {
"number_of_replicas" : 0
}
}'
```
### libretranslate
https://blog.gcn.sh/howtos/integrating-mastodon-and-libretranslate

BIN
img/basic_auth.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 39 KiB