Skip to content

Instantly share code, notes, and snippets.

@mheffner

mheffner/fly.toml

Last active September 24, 2023 14:50
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mheffner/a367e4f8424d937c511949d2d42c7943 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mheffner/a367e4f8424d937c511949d2d42c7943 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. mheffner revised this gist Sep 24, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions fly.toml
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,6 @@

    app = "mastiff"
    primary_region = "iad"
    kill_signal = "SIGINT"
    kill_timeout = 5
    processes = []
  2. mheffner revised this gist Sep 24, 2023. 1 changed file with 20 additions and 29 deletions.
    49 changes: 20 additions & 29 deletions fly.toml
    View file
    Original file line number Diff line number Diff line change
    @@ -35,35 +35,32 @@ processes = []

    REDIS_HOST = "fly-mastodon.upstash.io"

    [processes]
    web = "bundle exec rails s -p 3000"
    streaming = "node ./streaming"
    sidekiq = "bundle exec sidekiq"

    [experimental]
    allowed_public_ports = []
    auto_rollback = false

    [[services]]
    http_checks = []
    internal_port = 3000
    [http_service]
    processes = ["web"]
    protocol = "tcp"
    script_checks = []
    [services.concurrency]
    hard_limit = 25
    internal_port = 3000
    force_https = true
    auto_stop_machines = false
    auto_start_machines = false
    min_machines_running = 1
    [http_service.concurrency]
    type = "requests"
    soft_limit = 20
    type = "connections"

    [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

    [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

    [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
    hard_limit = 25
    [[http_service.checks]]
    grace_period = "5s"
    interval = "30s"
    method = "GET"
    timeout = "5s"
    path = "/health"

    [[services]]
    http_checks = []
    @@ -85,9 +82,3 @@ processes = []
    interval = "15s"
    restart_limit = 0
    timeout = "2s"


    [processes]
    web = "bundle exec rails s -p 3000"
    streaming = "node ./streaming"
    sidekiq = "bundle exec sidekiq"
  3. mheffner revised this gist Feb 20, 2023. 1 changed file with 1 addition and 2 deletions.
    3 changes: 1 addition & 2 deletions guide.md
    View file
    Original file line number Diff line number Diff line change
    @@ -245,12 +245,11 @@ are specified in the `fly.toml`.
    AWS_SECRET_ACCESS_KEY The secret user key for the R2 bucket
    DATABASE_URL Set automatically when the Postgres DB is attached
    OTP_SECRET Generated below
    REDIS_PASSWORD From your Redis.com API credentials
    REDIS_PASSWORD Redis password, see Redis section above.
    SECRET_KEY_BASE Generated below
    SMTP_PASSWORD From Sendgrid's API auth creds
    VAPID_PRIVATE_KEY Generated below
    VAPID_PUBLIC_KEY Generated below
    REDIS_PASSWORD Redis password, see Redis section above.
    ```

    ### Rails setup
  4. mheffner revised this gist Feb 20, 2023. 2 changed files with 28 additions and 35 deletions.
    3 changes: 1 addition & 2 deletions fly.toml
    View file
    Original file line number Diff line number Diff line change
    @@ -33,8 +33,7 @@ processes = []

    STREAMING_API_BASE_URL = "https://mastiff.party:4000"

    REDIS_HOST = "redis-XXXXX.us-east-1-3.ec2.cloud.redislabs.com"
    REDIS_PORT = 13255
    REDIS_HOST = "fly-mastodon.upstash.io"

    [experimental]
    allowed_public_ports = []
    60 changes: 27 additions & 33 deletions guide.md
    View file
    Original file line number Diff line number Diff line change
    @@ -35,6 +35,9 @@ future.
    [setting](https://github.com/mastodon/mastodon/pull/20510#issuecomment-1312601413)
    an additional environment variable. See the revisions of this doc
    for instructions on how to use AWS S3.

    **UPDATE: Feb, 2023**: Upstash redis on Fly.io now supports HyperLogLog,
    so this article was updated to use the free Fly.io Redis instead.

    ## Setup

    @@ -43,17 +46,14 @@ any skin in the game on any of these, so it should be relatively
    unbiased.

    - Google Domains: domain registrar
    - Fly.io: VMs and Postgres
    - Redis.com: Redis instance
    - Fly.io: VMs, Redis, and Postgres
    - ~AWS: S3~
    - Cloudflare: R2, DNS, and cached hosting of Mastodon Assets and
    user-uploaded content
    - Sendgrid: email delivery

    The current service costs I'm incurring at the moment are:

    - Redis.com: $5-7/month for the >30 concurrent connection tier
    - Fly - unsure on costs here yet
    As to costs, the current usage of Fly.io comes in under their minimum monthly charge of $5, so the costs are
    waived.

    This document walks through the approximate order I would use to set
    up these services. However, I built this with a lot of trial and
    @@ -97,33 +97,6 @@ Lastly, setup an API key in your settings and remember the key
    value. You'll use this when setting up the configuration for your
    site.

    ## Redis

    Mastodon relies on Redis as part of its architecture, for example
    to run Sidekiq jobs. For a small Mastodon instance the load is very
    small from what I can tell.

    I initially tried the public beta Redis support
    [offered](https://fly.io/docs/reference/redis/) by Fly.io
    since I was already using Fly. Fly uses a
    Redis-compatible implementation based on the work by
    [Upstash](https://docs.upstash.com/redis). Unfortunately, Mastodon
    relies on the HyperLogLog support of Redis and this is not
    [supported](https://docs.upstash.com/redis/overall/rediscompatibility)
    yet by Upstash, so Mastodon fails when trying to run a `PFCOUNT`. Once
    this is available I would be curious to try again.

    I ended up using the Redis support from
    [Redis.com](https://redis.com/) since it was the
    easiest one I could find. I originally boot up the free tier service
    instance and it appeared to work fine for my case, however I quickly
    found that I was exceeding the 30 connection count limitation. It was
    often exceeded when deploying since I was running twice the number of
    instances. I had to upgrade to the [next
    tier](https://redis.com/redis-enterprise-cloud/pricing/) at
    $5-7/month. While I don't require the additional capacity at the
    moment, I did need >30 connections.

    ## Cloudflare

    My setup uses Cloudflare for a few aspects:
    @@ -197,6 +170,7 @@ An example `fly.toml` is attached to this gist with the relevant
    values that should match the rest of the config in this doc. I'm going
    to use the Fly app name of `mastiff` for this example.


    ### VMs

    The app is split between three VMs: web, sidekiq and streaming. This
    @@ -222,6 +196,25 @@ should be fine. Provision the DB with `flyctl` and once it is running
    attach it to your Mastodon application. This will set the
    `DATABASE_URL` in the app allowing Rails to connect.

    ### Redis

    Mastodon relies on Redis as part of its architecture. The Redis provided
    on Fly.io is based on the [Upstash](https://docs.upstash.com/redis) distribution.
    For a small site, the free Redis tier of 100MB was fine to get started with.

    Just start with the following:

    ```shell
    $ flyctl redis create
    ```

    Follow the steps to provision a 100MB instance in the same region as your
    VMs.

    Record the password exported by `fly redis status <instance>` in the "Private URL"
    and set that as the secret `REDIS_PASSWORD`. You'll notice the `REDIS_HOST` is part
    of the `fly.toml` and should match the Private URL.

    ### Custom domain

    You'll want to follow the steps
    @@ -257,6 +250,7 @@ SECRET_KEY_BASE Generated below
    SMTP_PASSWORD From Sendgrid's API auth creds
    VAPID_PRIVATE_KEY Generated below
    VAPID_PUBLIC_KEY Generated below
    REDIS_PASSWORD Redis password, see Redis section above.
    ```

    ### Rails setup
  5. mheffner revised this gist Nov 24, 2022. 1 changed file with 0 additions and 3 deletions.
    3 changes: 0 additions & 3 deletions guide.md
    View file
    Original file line number Diff line number Diff line change
    @@ -163,9 +163,6 @@ In your DNS settings you will want to setup the following records:
    - Do not configure Proxying
    - CNAME record: `assets` -> `mastiff.party`
    - Configure Proxying (caching)
    - CNAME record: `files` -> `files.mastiff.party.s3.us-east-2.amazonaws.com`
    - Configure Proxying
    - (Use the correct AWS region name where you created your bucket)

    Under the *SSL/TLS* settings for your domain, enable **Full (strict)**
    encryption mode. Otherwise Cloudflare will attempt an HTTP connection
  6. mheffner revised this gist Nov 24, 2022. 3 changed files with 39 additions and 72 deletions.
    9 changes: 6 additions & 3 deletions fly.toml
    View file
    Original file line number Diff line number Diff line change
    @@ -21,13 +21,16 @@ processes = []
    CDN_HOST = "https://assets.mastiff.party"

    S3_ENABLED = "true"
    S3_BUCKET = "files.mastiff.party"
    S3_REGION = "us-east-2"
    S3_BUCKET = "files-mastiff-party"
    S3_REGION = "auto"
    S3_PROTOCOL = "https"
    S3_HOSTNAME = "s3.us-east-2.amazonaws.com"
    S3_HOSTNAME = "<acct num>.r2.cloudflarestorage.com"
    S3_ENDPOINT = "https://<acct num>.r2.cloudflarestorage.com"
    AWS_ACCESS_KEY_ID = "AKIAXXXXXXX"
    S3_ALIAS_HOST = "files.mastiff.party"

    S3_PERMISSION = "private"

    STREAMING_API_BASE_URL = "https://mastiff.party:4000"

    REDIS_HOST = "redis-XXXXX.us-east-1-3.ec2.cloud.redislabs.com"
    71 changes: 33 additions & 38 deletions guide.md
    View file
    Original file line number Diff line number Diff line change
    @@ -30,6 +30,12 @@ Caveat, this is for a small setup at the moment. It is unclear how
    this will scale or which components will be bottlenecks in the
    future.

    **UPDATE: Nov 24, 2022**: This was updated to use Cloudflare R2 in
    place of AWS S3. This required
    [setting](https://github.com/mastodon/mastodon/pull/20510#issuecomment-1312601413)
    an additional environment variable. See the revisions of this doc
    for instructions on how to use AWS S3.

    ## Setup

    For this setup I'm using the following active services. I don't have
    @@ -39,16 +45,14 @@ unbiased.
    - Google Domains: domain registrar
    - Fly.io: VMs and Postgres
    - Redis.com: Redis instance
    - AWS: S3
    - Cloudflare: DNS and cached hosting of Mastodon Assets and
    - ~AWS: S3~
    - Cloudflare: R2, DNS, and cached hosting of Mastodon Assets and
    user-uploaded content
    - Sendgrid: email delivery

    The current service costs I'm incurring at the moment are:

    - Redis.com: $5-7/month for the >30 concurrent connection tier
    - AWS S3: ~$0.10/month - not enough data to calculate full cost, but
    minimal
    - Fly - unsure on costs here yet

    This document walks through the approximate order I would use to set
    @@ -93,28 +97,6 @@ Lastly, setup an API key in your settings and remember the key
    value. You'll use this when setting up the configuration for your
    site.

    ## AWS S3

    We're going to run stateless VMs for Mastodon, so we need somewhere
    to dump user uploaded content (images, custom emojis). Mastodon
    supports S3 compatible stores to keep this content. I ended up going
    back to AWS after finding that other providers, like Cloudflare's R2,
    were not fully compatible with the ACL's Mastodon uses.

    For a server name of `mastiff.party`, you will want to create an S3
    bucket of `files.mastiff.party`. This will allow the files to load
    when accessed by CNAME from Cloudflare. This
    [guide](https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md#setting-up-aws-and-an-s3-bucket)
    has good instructions for how to setup the S3 bucket for
    Mastodon including the permissions to enable on the bucket creation
    page. However, ignore the concerns about the bucket name syntax, just
    use `files.mastiff.party`.

    You'll want to create an IAM user that can access this bucket. See the
    attached file `s3policy.json` in this gist for an example policy for
    this user. This policy appears to cover all that's required, but it
    likely contains a few more that aren't strictly required.

    ## Redis

    Mastodon relies on Redis as part of its architecture, for example
    @@ -146,20 +128,33 @@ moment, I did need >30 connections.

    My setup uses Cloudflare for a few aspects:

    - HTTPS CNAME hosting of the AWS S3 bucket, `files.mastiff.party`
    - R2 object storage, similar to AWS S3
    - Exposing the R2 bucket as public under the domain name `files.mastiff.party`
    - Caching of the Mastodon static assets, as `assets.mastiff.party`
    - General DNS

    Instead of Cloudflare, I image the same could be handled with AWS
    Route53 and Cloudfront. I
    originally looked at Cloudflare because I was interested in kicking
    the tires of their R2 service. R2 provides object storage at
    competitive pricing and has an S3 compatible API. Unfortunately, the
    Mastodon ACL `public-read` use is not compatible with R2's
    implementation and failed to upload anything.
    Create a new account with Cloudflare or start a new site. During the
    setup you will be instructed on how to move your DNS to
    Cloudflare. This is not an optional step.

    Once setup, first create a R2 bucket in Cloudflare, for this guide I'll name it
    `files-mastiff-party`. Once created go to the "Manage R2 API Tokens"
    to grab the key and secret key. These will be set as the `AWS_*`
    variables below.

    Under the new R2 bucket, grab the endpoint URL under the bucket name
    at the top of the page. Second, go to Bucket Settings and create a
    domain under the section "domain access". Enter the name
    `files.mastiff.party`, it will then create a DNS entry for `files`
    under your site that points to the bucket. This will expose the files
    in the bucket as public even though they are by default private.

    In the environment variables, shown later in the *fly.toml*, you will
    need to set `S3_PERMISSION=private`. This prevents Mastodon from
    trying to use a `public-read` object ACL that is not supported on
    Cloudflare. Instead, the custom domain exposes the objects as public.

    Once you get setup with Cloudflare and migrate your DNS hosting to
    them, you will want to setup the following records:
    In your DNS settings you will want to setup the following records:

    - A record: `mastiff.party`(`@` root) -> Fly.io IPv4 address
    (configured next)
    @@ -257,7 +252,7 @@ limit this to only values that are actually secret. All other values
    are specified in the `fly.toml`.

    ```
    AWS_SECRET_ACCESS_KEY The secret IAM user key for the S3 bucket
    AWS_SECRET_ACCESS_KEY The secret user key for the R2 bucket
    DATABASE_URL Set automatically when the Postgres DB is attached
    OTP_SECRET Generated below
    REDIS_PASSWORD From your Redis.com API credentials
    @@ -298,7 +293,7 @@ mentions:
    assets, make sure to include the scheme (`https://`) here or else
    you'll have problems
    - S3_ALIAS_HOST: This is the Cloudflare proxy rule that will expose
    the S3 assets under your custom domain name in case you want to move
    the R2 assets under your custom domain name in case you want to move
    them in the future
    - STREAMING_API_BASE_URL: This allows websocket streaming to work by
    requesting the streaming on a different port. In theory you would
    31 changes: 0 additions & 31 deletions s3policy.json
    View file
    Original file line number Diff line number Diff line change
    @@ -1,31 +0,0 @@
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": [
    "s3:GetBucketPolicyStatus",
    "s3:GetBucketPublicAccessBlock",
    "s3:ListBucketMultipartUploads",
    "s3:GetBucketTagging",
    "s3:GetObjectAttributes",
    "s3:ListBucket",
    "s3:GetBucketVersioning",
    "s3:GetBucketAcl",
    "s3:GetBucketPolicy",
    "s3:ListMultipartUploadParts",
    "s3:PutObject",
    "s3:GetObjectAcl",
    "s3:GetObject",
    "s3:PutObjectRetention",
    "s3:DeleteObject",
    "s3:PutObjectAcl"
    ],
    "Resource": [
    "arn:aws:s3:::files.mastiff.party",
    "arn:aws:s3:::files.mastiff.party/*"
    ]
    }
    ]
    }
  7. mheffner revised this gist Nov 21, 2022. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions guide.md
    View file
    Original file line number Diff line number Diff line change
    @@ -49,6 +49,7 @@ The current service costs I'm incurring at the moment are:
    - Redis.com: $5-7/month for the >30 concurrent connection tier
    - AWS S3: ~$0.10/month - not enough data to calculate full cost, but
    minimal
    - Fly - unsure on costs here yet

    This document walks through the approximate order I would use to set
    up these services. However, I built this with a lot of trial and
  8. mheffner revised this gist Nov 21, 2022. 1 changed file with 7 additions and 2 deletions.
    9 changes: 7 additions & 2 deletions guide.md
    View file
    Original file line number Diff line number Diff line change
    @@ -26,6 +26,10 @@ want a quick solution the Digital Ocean
    instances are
    likely easier to get started with.

    Caveat, this is for a small setup at the moment. It is unclear how
    this will scale or which components will be bottlenecks in the
    future.

    ## Setup

    For this setup I'm using the following active services. I don't have
    @@ -81,7 +85,7 @@ address.

    Ensure that link branding is turned off, even if the domain
    verification says that it should work. SSL problems mean that the
    links won't work. Just use the direct sendgrid links for now (see
    links won't work. Use the direct Sendgrid links for now (see
    Hiccup below).

    Lastly, setup an API key in your settings and remember the key
    @@ -145,7 +149,8 @@ My setup uses Cloudflare for a few aspects:
    - Caching of the Mastodon static assets, as `assets.mastiff.party`
    - General DNS

    These could easily be handled by AWS Route53 and Cloudfront. I
    Instead of Cloudflare, I image the same could be handled with AWS
    Route53 and Cloudfront. I
    originally looked at Cloudflare because I was interested in kicking
    the tires of their R2 service. R2 provides object storage at
    competitive pricing and has an S3 compatible API. Unfortunately, the
  9. mheffner revised this gist Nov 21, 2022. 2 changed files with 122 additions and 0 deletions.
    91 changes: 91 additions & 0 deletions fly.toml
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,91 @@

    app = "mastiff"
    kill_signal = "SIGINT"
    kill_timeout = 5
    processes = []

    [build]
    image = "tootsuite/mastodon:v3.5.5"

    [env]
    LOCAL_DOMAIN = "mastiff.party"

    IP_RETENTION_PERIOD = 31556952
    SESSION_RETENTION_PERIOD = 31556952
    RAILS_ENV = "production"
    ALTERNATE_DOMAINS = "mastiff.fly.dev,assets.mastiff.party,files.mastiff.party"
    SMTP_SERVER = "smtp.sendgrid.net"
    SMTP_PORT = 587
    SMTP_LOGIN = "apikey"
    SMTP_FROM_ADDRESS = "[email protected]"
    CDN_HOST = "https://assets.mastiff.party"

    S3_ENABLED = "true"
    S3_BUCKET = "files.mastiff.party"
    S3_REGION = "us-east-2"
    S3_PROTOCOL = "https"
    S3_HOSTNAME = "s3.us-east-2.amazonaws.com"
    AWS_ACCESS_KEY_ID = "AKIAXXXXXXX"
    S3_ALIAS_HOST = "files.mastiff.party"

    STREAMING_API_BASE_URL = "https://mastiff.party:4000"

    REDIS_HOST = "redis-XXXXX.us-east-1-3.ec2.cloud.redislabs.com"
    REDIS_PORT = 13255

    [experimental]
    allowed_public_ports = []
    auto_rollback = false

    [[services]]
    http_checks = []
    internal_port = 3000
    processes = ["web"]
    protocol = "tcp"
    script_checks = []
    [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

    [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

    [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

    [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

    [[services]]
    http_checks = []
    internal_port = 4000
    processes = ["streaming"]
    protocol = "tcp"
    script_checks = []
    [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

    [[services.ports]]
    handlers = ["tls", "http"]
    port = 4000

    [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"


    [processes]
    web = "bundle exec rails s -p 3000"
    streaming = "node ./streaming"
    sidekiq = "bundle exec sidekiq"
    31 changes: 31 additions & 0 deletions s3policy.json
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,31 @@
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": [
    "s3:GetBucketPolicyStatus",
    "s3:GetBucketPublicAccessBlock",
    "s3:ListBucketMultipartUploads",
    "s3:GetBucketTagging",
    "s3:GetObjectAttributes",
    "s3:ListBucket",
    "s3:GetBucketVersioning",
    "s3:GetBucketAcl",
    "s3:GetBucketPolicy",
    "s3:ListMultipartUploadParts",
    "s3:PutObject",
    "s3:GetObjectAcl",
    "s3:GetObject",
    "s3:PutObjectRetention",
    "s3:DeleteObject",
    "s3:PutObjectAcl"
    ],
    "Resource": [
    "arn:aws:s3:::files.mastiff.party",
    "arn:aws:s3:::files.mastiff.party/*"
    ]
    }
    ]
    }
  10. mheffner created this gist Nov 21, 2022.
    367 changes: 367 additions & 0 deletions guide.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,367 @@
    # Running a Mastodon server on Fly.io (plus some stuff)

    With the recent chaos at Twitter and the future of the platform
    looking unclear, many are looking for alternatives. One popular
    alternative that has seen tremendous growth over the last several
    weeks has been Mastodon. Mastodon builds on a federated set of
    individual communities, each backed by their own running server
    instance. It's too early to tell if Mastodon will be the "replacement"
    for Twitter, but it does appear to be facilitating a different kind of
    discussion format.

    If you are looking to join Mastodon, it is best to jump into
    one of the [existing communities](https://joinmastodon.org/servers)
    and join the conversations ongoing
    there. However, you can also start your own community and host your
    own Mastodon instance. You can always follow people in different
    communities as part of the fediverse.

    This document briefly describes how to get Mastodon running on
    Fly.io. That's actually not 100% accurate because it actually uses a
    combination of services. For a small server with few users, you can
    operate a Mastodon instance with minor spend by leveraging the free
    tiers of many services. This is somewhat of a cobbled mess, so if you
    want a quick solution the Digital Ocean
    [pre-built](https://marketplace.digitalocean.com/apps/mastodon)
    instances are
    likely easier to get started with.

    ## Setup

    For this setup I'm using the following active services. I don't have
    any skin in the game on any of these, so it should be relatively
    unbiased.

    - Google Domains: domain registrar
    - Fly.io: VMs and Postgres
    - Redis.com: Redis instance
    - AWS: S3
    - Cloudflare: DNS and cached hosting of Mastodon Assets and
    user-uploaded content
    - Sendgrid: email delivery

    The current service costs I'm incurring at the moment are:

    - Redis.com: $5-7/month for the >30 concurrent connection tier
    - AWS S3: ~$0.10/month - not enough data to calculate full cost, but
    minimal

    This document walks through the approximate order I would use to set
    up these services. However, I built this with a lot of trial and
    error, so this was not the order of operations I followed initially.

    ## Buy a domain

    You will need a custom domain for your community. Chose your favorite
    domain registrar, it shouldn't matter too much which you chose. I used
    [Google Domains](https://domains.google/) because I had bought domains
    from there before.

    For the remainder of this doc I'll assume a domain of `mastiff.party`,
    a Mastodon dedicated to the love of Mastiffs. (I have no idea if this
    is an active community or not)


    ## Sendgrid

    You will need a service that can reliably deliver emails for sign up
    email verification, notifications and other transactional updates. You
    should be able to use any service that offers SMTP delivery. I chose
    Sendgrid, they have a [free plan](https://sendgrid.com/pricing/) that
    allows up to 100
    emails/day. I have few users on my instance, so that felt like plenty.

    Sign up for an account and enter your domain name. You will need to
    authenticate your domain and verify a sender address. Follow the
    instructions for domain authentication, it will require adding some
    records to your DNS settings from the registrar. Then you'll want to
    setup a single sender that emails will be sent from. Following the
    example here, I used `[email protected]` as my sender
    address.

    Ensure that link branding is turned off, even if the domain
    verification says that it should work. SSL problems mean that the
    links won't work. Just use the direct sendgrid links for now (see
    Hiccup below).

    Lastly, setup an API key in your settings and remember the key
    value. You'll use this when setting up the configuration for your
    site.

    ## AWS S3

    We're going to run stateless VMs for Mastodon, so we need somewhere
    to dump user uploaded content (images, custom emojis). Mastodon
    supports S3 compatible stores to keep this content. I ended up going
    back to AWS after finding that other providers, like Cloudflare's R2,
    were not fully compatible with the ACL's Mastodon uses.

    For a server name of `mastiff.party`, you will want to create an S3
    bucket of `files.mastiff.party`. This will allow the files to load
    when accessed by CNAME from Cloudflare. This
    [guide](https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md#setting-up-aws-and-an-s3-bucket)
    has good instructions for how to setup the S3 bucket for
    Mastodon including the permissions to enable on the bucket creation
    page. However, ignore the concerns about the bucket name syntax, just
    use `files.mastiff.party`.

    You'll want to create an IAM user that can access this bucket. See the
    attached file `s3policy.json` in this gist for an example policy for
    this user. This policy appears to cover all that's required, but it
    likely contains a few more that aren't strictly required.

    ## Redis

    Mastodon relies on Redis as part of its architecture, for example
    to run Sidekiq jobs. For a small Mastodon instance the load is very
    small from what I can tell.

    I initially tried the public beta Redis support
    [offered](https://fly.io/docs/reference/redis/) by Fly.io
    since I was already using Fly. Fly uses a
    Redis-compatible implementation based on the work by
    [Upstash](https://docs.upstash.com/redis). Unfortunately, Mastodon
    relies on the HyperLogLog support of Redis and this is not
    [supported](https://docs.upstash.com/redis/overall/rediscompatibility)
    yet by Upstash, so Mastodon fails when trying to run a `PFCOUNT`. Once
    this is available I would be curious to try again.

    I ended up using the Redis support from
    [Redis.com](https://redis.com/) since it was the
    easiest one I could find. I originally boot up the free tier service
    instance and it appeared to work fine for my case, however I quickly
    found that I was exceeding the 30 connection count limitation. It was
    often exceeded when deploying since I was running twice the number of
    instances. I had to upgrade to the [next
    tier](https://redis.com/redis-enterprise-cloud/pricing/) at
    $5-7/month. While I don't require the additional capacity at the
    moment, I did need >30 connections.

    ## Cloudflare

    My setup uses Cloudflare for a few aspects:

    - HTTPS CNAME hosting of the AWS S3 bucket, `files.mastiff.party`
    - Caching of the Mastodon static assets, as `assets.mastiff.party`
    - General DNS

    These could easily be handled by AWS Route53 and Cloudfront. I
    originally looked at Cloudflare because I was interested in kicking
    the tires of their R2 service. R2 provides object storage at
    competitive pricing and has an S3 compatible API. Unfortunately, the
    Mastodon ACL `public-read` use is not compatible with R2's
    implementation and failed to upload anything.

    Once you get setup with Cloudflare and migrate your DNS hosting to
    them, you will want to setup the following records:

    - A record: `mastiff.party`(`@` root) -> Fly.io IPv4 address
    (configured next)
    - Do not configure Proxying
    - AAAA record: `mastiff.party` -> Fly.io IPv6 address configured next)
    - Do not configure Proxying
    - CNAME record: `assets` -> `mastiff.party`
    - Configure Proxying (caching)
    - CNAME record: `files` -> `files.mastiff.party.s3.us-east-2.amazonaws.com`
    - Configure Proxying
    - (Use the correct AWS region name where you created your bucket)

    Under the *SSL/TLS* settings for your domain, enable **Full (strict)**
    encryption mode. Otherwise Cloudflare will attempt an HTTP connection
    to your origin server on Fly and get an HTTPS redirect. This will end
    up causing a redirect loop.

    Second, under *Rules -> Transform Rules*, select the *Header
    Modification* box on the right. You will need to create a *HTTP Response
    Header Modification* rule. These are the settings I used:
    - rule name: *access-control-allow-origin*
    - Under matches:
    - Field: *Request method*
    - Operator: *is in*
    - Value: *GET, POST, HEAD, OPTIONS*
    - Then,
    - Enable *Set static*
    - Header name: `Access-Control-Allow-Origin`
    - Value `*`

    This CORS header rule will permit Javascript resources loaded from
    `assets.mastiff.party` to access `mastiff.party`.


    ## Fly.io

    The remainder of the components will be setup on Fly.io. Mastodon
    [provides](https://hub.docker.com/r/tootsuite/mastodon) pre-built
    Docker images on Dockerhub so it is easy to simply use those
    directly. I've broken the steps out by the component pieces.

    An example `fly.toml` is attached to this gist with the relevant
    values that should match the rest of the config in this doc. I'm going
    to use the Fly app name of `mastiff` for this example.

    ### VMs

    The app is split between three VMs: web, sidekiq and streaming. This
    is my current scaling sizes for the VMs. You will definitely need more
    than the default 256MB for web and sidekiq or you'll see OOM's even at
    small scale. Streaming doesn't appear to need as much so can be kept
    at 256MB.

    - web
    - Size: shared-cpu-1x
    - Memory: 1GB
    - sidekiq
    - Size: shared-cpu-1x
    - Memory: 1GB
    - streaming
    - Size: shared-cpu-1x
    - Memory: 256MB

    ### Postgres

    To get started, the `shared-cpu-1x` Postgres VM with 256MB of memory
    should be fine. Provision the DB with `flyctl` and once it is running
    attach it to your Mastodon application. This will set the
    `DATABASE_URL` in the app allowing Rails to connect.

    ### Custom domain

    You'll want to follow the steps
    [here](https://fly.io/blog/how-to-custom-domains-with-fly/) to setup
    your Fly application with your custom `mastiff.party` domain. This
    will be where you'll plug the IPv4 and IPv6 addresses into your
    Cloudflare DNS.

    ### SSL Certs

    When you add your custom domain Fly will auto-provision an SSL cert
    for your domain `mastiff.party` using Let's Encrypt. However,
    Cloudflare will need to proxy requests for `assets.mastiff.party` to
    your site and hence will need a wildcard cert as well. Use the
    `flyctl` command to create a [wildcard
    cert](https://fly.io/docs/app-guides/custom-domains-with-fly/#adding-the-certificate). (Unfortunately
    you can not change the name of the origin Cloudflare uses to proxy to
    in their free plans).

    ### Secrets

    Along with the configuration in the provided `fly.toml`, you will need
    to set the following secrets with `flyctl secrets set`. I've tried to
    limit this to only values that are actually secret. All other values
    are specified in the `fly.toml`.

    ```
    AWS_SECRET_ACCESS_KEY The secret IAM user key for the S3 bucket
    DATABASE_URL Set automatically when the Postgres DB is attached
    OTP_SECRET Generated below
    REDIS_PASSWORD From your Redis.com API credentials
    SECRET_KEY_BASE Generated below
    SMTP_PASSWORD From Sendgrid's API auth creds
    VAPID_PRIVATE_KEY Generated below
    VAPID_PUBLIC_KEY Generated below
    ```

    ### Rails setup

    For these commands you should deploy the Mastodon container and then
    use the following to access the VM: `flyctl ssh console`. Once you
    have a terminal in the app, cd to `/mastodon`. This is where the app
    code exists in the VM.

    1. Setup the DB:
    ```
    RAILS_ENV=production bundle exec rake db:create db:schema:load
    ```
    2. Generate the secrets, this will provide `SECRET_KEY_BASE` and `OTP_SECRET`
    ```
    bundle exec rake secret
    ```
    3. Generate the web push secrets, will provide `VAPID_*_KEY` values
    ```
    bundle exec rake mastodon:webpush:generate_vapid_key
    ```

    ### fly.toml

    See the provided `fly.toml` example for the remaining settings. A few
    mentions:

    - ALTERNATE_DOMAINS: any potential name that may be used in a request
    should be listed here or else Rails will 403 the request
    - CDN_HOST: Domain that will be used to cache and offload the static
    assets, make sure to include the scheme (`https://`) here or else
    you'll have problems
    - S3_ALIAS_HOST: This is the Cloudflare proxy rule that will expose
    the S3 assets under your custom domain name in case you want to move
    them in the future
    - STREAMING_API_BASE_URL: This allows websocket streaming to work by
    requesting the streaming on a different port. In theory you would
    have nginx in front of your app to redirect those requests to the
    node.js app, but since we're running that in a separate VM this was
    a work around. Would love to know if there's a better way to handle
    this on Fly.

    This should be all you need to get the site loading. Deploy and check
    out `https://mastiff.party`. The monitoring logs on Fly are also
    critical to debugging any potential issues.

    ## Post setup

    Once you have the server running and have created a user, you can give
    yourself admin privileges by logging into the rails console (see
    above) and running the following (replace `myusername`):

    ```
    RAILS_ENV=production ./bin/tootctl accounts modify myusername --role admin
    ```

    ## Wrap up

    I believe that is the full setup, but again I arrived here somewhat
    through trial and error. Let me know if anything needs clarification.


    ## Hiccups

    ### Deleting DB doesn't remove DATABASE_URL

    During my earlier trial and error I messed up the DB enough that I
    figured I'd just delete it and start over. Unfortunately, deleting the
    DB doesn't appear to automatically detach it from the running
    apps. This isn't terrible until you attempt to reattach a new DB
    because it will complain that `DATABASE_URL` is already defined.

    I had to set the experimental `auto_rollback = false` option in
    `fly.toml` because I couldn't delete the secret
    `DATABASE_URL`. Removing it clearly failed the app so it would
    continually rollback to the previous version. Once I deleted this I
    could attach a new DB back again.

    ### Refresh account images

    During my early trail and error I found that account images were not
    working. This was when I was trying to find an S3 solution before
    settling on the original. I found this command that appeared to fix
    things and redownload the right media:

    Log into the VM with `flyctl ssh console`
    ```
    cd /mastodon
    ./bin/tootctl accounts refresh --all --verbose
    ```

    ### Link branding SSL problems

    Sendgrid allows link branding by having you setup a custom subdomain
    like `<foo>.mastiff.party` that will point back to Sendgrid. Email
    links will appear under your domain instead of
    Sendgrid's. These links don't use SSL, which normally would
    be fine. However, Fly sets the response header
    `strict-transport-security: max-age=63072000; includeSubDomains` when
    accessing your Mastodon site at `mastiff.party`. Browsers will respect
    that and try to access the email links using https, but Sendgrid won't
    have an accurate SSL cert and you'll see the big danger page.

    I haven't explored this much since I was OK with the Sendgrid branded
    email links. I'm guessing I could use Cloudflare to host these links
    behind https if needed.