mattes · July 29, 2024 04:08 · Nov 1, 2020 · Oct 16, 2020 · Oct 16, 2020 · Oct 16, 2020
diff --git a/rails-google-compute-deploy.md b/rails-google-compute-deploy.md
@@ -57,6 +57,8 @@ Create a Google Service Account for deploys, it will need the following roles:
 * Service Account User
 * Cloud SQL Client
 * Storage Admin
+* Secret Manager Secret Accessor
+* Secret Manager Viewer
 
 Create a JSON key for the service account you just created and copy and paste the contents to a Github Secret called `GOOGLE_APPLICATION_CREDENTIALS`. No need to base64 encode the secret.
 

diff --git a/rails-google-compute-deploy.md b/rails-google-compute-deploy.md
@@ -197,7 +197,7 @@ The deploy action will:
 2) Update metadata config of the newly created instance template to run a [cloud-init instructions file](https://github.com/mattes/gce-boot-scripts/tree/master/rails).
 3) Tell the instance group manager to perform a rolling update with the new instance template.
 
-Please review the [configuration options](https://github.com/mattes/gce-deploy-action) and get a better understanding how this step works in detail. Please also review the [rails helper utility](https://github.com/mattes/gce-boot-scripts/tree/master/rails) used in `deploys.*.vars.RUN` which starts a Rails Docker Image.
+Please review the [configuration options](https://github.com/mattes/gce-deploy-action) and get a better understanding how this step works in detail. Please also review the [rails helper utility](https://github.com/mattes/gce-boot-scripts/tree/master/rails/RAILS.md) used in `deploys.*.vars.RUN` which starts a Rails Docker Image.
 
 ```yaml
 common:

diff --git a/rails-google-compute-deploy.md b/rails-google-compute-deploy.md
@@ -39,7 +39,7 @@ Create Google Secrets which will be available to the Rails app as ENV variables.
 * Set your prefix for `GOOGLE_SECRETS_PREFIX` in the `rails.yml` Github Action (see below)
 * Set your prefix for `SECRETS_PREFIX` in the `deploy.yml` file (see below)
 
-To connect the CloudSQL proxy to your CLoudSQL instance later, please also set `<prefix>_CLOUDSQL_INSTANCE`, i.e. `MY_APP_CLOUDSQL_PREFIX=my-project:region:my-instance`.
+To connect the CloudSQL proxy to your CLoudSQL instance later, please also set `<prefix>_CLOUDSQL_INSTANCE`, i.e. `MY_APP_CLOUDSQL_INSTANCE=my-project:region:my-instance`.
 
 ### 2) Prepare Rails App
 

diff --git a/rails-google-compute-deploy.md b/rails-google-compute-deploy.md
@@ -39,6 +39,7 @@ Create Google Secrets which will be available to the Rails app as ENV variables.
 * Set your prefix for `GOOGLE_SECRETS_PREFIX` in the `rails.yml` Github Action (see below)
 * Set your prefix for `SECRETS_PREFIX` in the `deploy.yml` file (see below)
 
+To connect the CloudSQL proxy to your CLoudSQL instance later, please also set `<prefix>_CLOUDSQL_INSTANCE`, i.e. `MY_APP_CLOUDSQL_PREFIX=my-project:region:my-instance`.
 
 ### 2) Prepare Rails App
 
@@ -201,11 +202,10 @@ Please review the [configuration options](https://github.com/mattes/gce-deploy-a
 ```yaml
 common:
   region: us-central1
-  cloud_init: https://raw.githubusercontent.com/mattes/gce-boot-scripts/v1.0.0/rails/dist/cloud-init.yml
+  cloud_init: https://raw.githubusercontent.com/mattes/gce-boot-scripts/v1.0.1/rails/dist/cloud-init.yml
   vars:
     DOCKER_IMAGE: gcr.io/my-project/my-app:${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
     SECRETS_PREFIX: MY_APP
-    CLOUDSQL_INSTANCE: my-project:region:my-instance
   labels:
     version: ${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
     git-sha: ${{GITHUB_SHA}}

diff --git a/rails-google-compute-deploy.md b/rails-google-compute-deploy.md
@@ -205,6 +205,7 @@ common:
   vars:
     DOCKER_IMAGE: gcr.io/my-project/my-app:${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
     SECRETS_PREFIX: MY_APP
+    CLOUDSQL_INSTANCE: my-project:region:my-instance
   labels:
     version: ${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
     git-sha: ${{GITHUB_SHA}}

diff --git a/rails-google-compute-deploy.md b/rails-google-compute-deploy.md
@@ -111,7 +111,7 @@ jobs:
     - uses: actions/checkout@v2
 
     - name: Fetch Dockerfiles
-      run: wget -q -P /tmp https://raw.githubusercontent.com/mattes/dockerfile-buildpacks/v1.0.6/rails/Dockerfile.{Base,Build,Deploy}
+      run: wget -q -P /tmp https://raw.githubusercontent.com/mattes/dockerfile-buildpacks/v1.0.0/rails/Dockerfile.{Base,Build,Deploy}
 
     - name: Build Dockerfile.Base
       uses: mattes/cached-docker-build-action@v1

diff --git a/rails-google-compute-deploy.md b/rails-google-compute-deploy.md
@@ -0,0 +1,296 @@
+# Deploy Rails apps to Google Cloud Compute Engine
+
+* Zero Downtime
+* Graceful shutdowns
+* Via Github Actions
+* Zero infrastructure management overhead
+
+## Overview
+
+The general idea is to have Github Actions test, build and deploy a Rails app to Google Cloud Compute Engine.
+It works like this:
+
+1. Push rails app to master branch.
+2. Github Actions test, build and push a Rails app Dockerfile to the Google Container Registry.
+3. Github Actions run database migrations.
+4. Github Actions instruct Google Instance Group Manager to do a rolling update to the latest Dockerfile.
+
+It's a poor man's solution to avoid and having to manage Kubernetes or using Google App Engine with computational restrictions. At the end of the day, this approach just deploys a Dockerfile to a Google Compute Instance running Google's Container Optimized OS. 
+
+## Components
+
+* Github Actions, including the [deploy action](https://github.com/mattes/gce-deploy-action) and [Dockerfile Rails Buildpacks](https://github.com/mattes/dockerfile-buildpacks/tree/master/rails)
+* [Google Secret Manager](https://cloud.google.com/secret-manager), including [google_cloud_env_secrets gem](https://github.com/mattes/rails_google_cloud_env_secrets) to load secrets at runtime
+* [Google Container Registry](https://cloud.google.com/container-registry)
+* [Google Instance Groups](https://cloud.google.com/compute/docs/instance-groups) to manage deploys
+* [Google Compute Engine](https://cloud.google.com/compute) virtual machines
+* [Google Container Optimized OS](https://cloud.google.com/container-optimized-os)
+* [Google Instance Templates](https://cloud.google.com/compute/docs/instance-templates)
+* [Google Load Balancer](https://cloud.google.com/load-balancing)
+* [Google CloudSQL Postgres Database](https://cloud.google.com/sql/docs/postgres)
+* [Google Memorystore](https://cloud.google.com/memorystore) for Redis (optional)
+
+## Getting started
+
+### 1) Create Google Secrets
+
+Create Google Secrets which will be available to the Rails app as ENV variables. Choose a prefix for every secret, like `MY_APP`. For example: `MY_APP_RAILS_DATABASE` would save the connection string for the Postgres database in the Google Secrets Manager, but would still be available as `RAILS_DATABASE` to the Rails app.
+
+* Set your prefix for `GOOGLE_SECRETS_PREFIX` in the `rails.yml` Github Action (see below)
+* Set your prefix for `SECRETS_PREFIX` in the `deploy.yml` file (see below)
+
+
+### 2) Prepare Rails App
+
+Add [`gem 'google_cloud_env_secrets'`](https://github.com/mattes/rails_google_cloud_env_secrets) to your `Gemfile` and run `bundle install`. This Rubygem will load Google Secrets as ENV vars at runtime. 
+
+Since we are using Google Secrets, `config/credentials.yml.enc` won't be used anymore. Delete `config/credentials.yml.enc` and in `config/environments/production.rb` set `config.require_master_key = false`. You won't need to provide a `RAILS_MASTER_KEY` going forward, but you will have to create a Google Secret for `MY_APP_SECRET_KEY_BASE` which was previously provided by the credentials file.
+
+Should secrets be stored as ENV vars? [HackerNews](https://news.ycombinator.com/item?id=8826024) discussed it. We are following the [12factor](https://12factor.net/config) approach here. Using Google Secrets Manager also helps with tooling. For example: If you're using [Terraform](https://www.terraform.io/) to provision your CloudSQL Postgres database, you can use it to easily write the connection string to a Google Secret `MY_APP_RAILS_DATABASE`. This would be exponentially harder if the secrets would be stored inside your Git repository. 
+
+
+### 3) Create Google Service Account
+
+Create a Google Service Account for deploys, it will need the following roles:
+* Compute Admin
+* Service Account User
+* Cloud SQL Client
+* Storage Admin
+
+Create a JSON key for the service account you just created and copy and paste the contents to a Github Secret called `GOOGLE_APPLICATION_CREDENTIALS`. No need to base64 encode the secret.
+
+### 4) Set up Github Actions
+
+Copy the following file to `.github/workflows/rails.yml`. It will:
+
+1. Create a local Postgres database (port 5432) and redis instance (port 6379) for testing.
+2. Download [Dockerfile Rails Buildpacks](https://github.com/mattes/dockerfile-buildpacks/tree/master/rails)
+3. Build Dockerfile.Base which is a base image with Ruby and Node installed
+4. Build Dockerfile.Build which is based on Dockerfile.Base and also has Rubygems and NPM packages installed
+5. Run rspec
+6. If it's not the `master` branch, skip the next steps.
+7. Precompile Rails assets
+8. Build Dockerfile.Deploy which is a [distroless](https://github.com/GoogleContainerTools/distroless) image that contains the final Rails app.
+9. Push the image to the Google Cloud Container Registry
+10. Run database migrations
+11. Start rolling update of the Google instance group
+
+```yaml
+name: Rails 
+
+on: [push]
+
+jobs:
+  pipeline:
+    name: Build, Test & Deploy  
+    runs-on: ubuntu-latest
+
+    services:
+      postgres:
+        image: postgres:11
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: rails_test
+          POSTGRES_HOST_AUTH_METHOD: trust
+        ports:
+        - 5432:5432
+        options: --health-cmd pg_isready --health-interval 3s --health-timeout 5s --health-retries 5
+
+      redis:
+        image: redis
+        ports:
+        - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 3s
+          --health-timeout 5s
+          --health-retries 5
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Fetch Dockerfiles
+      run: wget -q -P /tmp https://raw.githubusercontent.com/mattes/dockerfile-buildpacks/v1.0.6/rails/Dockerfile.{Base,Build,Deploy}
+
+    - name: Build Dockerfile.Base
+      uses: mattes/cached-docker-build-action@v1
+      with:
+        args: "--file /tmp/Dockerfile.Base --tag rails:base ."
+
+    - name: Build Dockerfile.Build
+      uses: mattes/cached-docker-build-action@v1
+      with:
+        args: "--file /tmp/Dockerfile.Build --tag rails:build ."
+        cache_key: ${{hashFiles('*.lock')}}
+
+    - name: rspec
+      run: >
+          docker run --rm --net host -v $PWD:/app
+          -e RAILS_ENV='test' -e NODE_ENV='test'
+          rails:build
+          'bundle exec rails db:create && bundle exec rspec'
+
+    - name: Deploy - rails assets:precompile
+      if: github.ref == 'refs/heads/master'
+      run: >
+          docker run --rm --net host -v $PWD:/app
+          -e GOOGLE_APPLICATION_CREDENTIALS='${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
+          -e GOOGLE_SECRETS_PREFIX=MY_APP
+          rails:build
+          'bundle exec rails assets:precompile'
+
+    - name: Deploy - Build Dockerfile.Deploy
+      if: github.ref == 'refs/heads/master'
+      run: docker build -f /tmp/Dockerfile.Deploy --build-arg VERSION="${{github.run_number}}-${{github.sha}}" -t rails:deploy .
+
+    - name: Deploy - Push Docker Image
+      if: github.ref == 'refs/heads/master'
+      uses: mattes/gce-docker-push-action@v1
+      with:
+        creds: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
+        src: rails:deploy
+        dst: gcr.io/my-project/my-app:${{github.run_number}}-${{github.sha}}
+
+    - name: Deploy - Google Cloud SQL Proxy
+      if: github.ref == 'refs/heads/master'
+      uses: mattes/gce-cloudsql-proxy-action@v1
+      with:
+        creds: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
+        instance: my-project:us-central1:my-instance
+        port: 5433
+
+    - name: Deploy - rails db:migrate
+      if: github.ref == 'refs/heads/master'
+      run: >
+          docker run --rm --net host -v $PWD:/app
+          -e GOOGLE_APPLICATION_CREDENTIALS='${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
+          -e GOOGLE_SECRETS_PREFIX=MY_APP
+          -e RAILS_DATABASE_PORT=5433
+          rails:build
+          'bundle exec rails db:create; bundle exec rails db:migrate'
+
+    - name: Deploy - Start rolling update
+      uses: mattes/gce-deploy-action@v5
+      if: github.ref == 'refs/heads/master'
+      with:
+        creds: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
+```
+
+This Github Action workflow uses a couple of helper files and actions. You can read more about them here:
+
+* [Dockerfile Buildpacks for Rails](https://github.com/mattes/dockerfile-buildpacks/tree/master/rails)
+* [mattes/cached-docker-build-action](https://github.com/mattes/cached-docker-build-action)
+* [mattes/gce-docker-push-action](https://github.com/mattes/gce-docker-push-action)
+* [mattes/gce-cloudsql-proxy-action](https://github.com/mattes/gce-cloudsql-proxy-action)
+* [mattes/gce-deploy-action](https://github.com/mattes/gce-deploy-action)
+
+
+### 5) Create deploy.yml
+
+Please copy the following file to the root directory of your Git repository `./my-repo/deploy.yml` where it will be picked up by the `gce-deploy-action` step and deploys `app` and `worker`.
+
+The deploy action will:
+
+1) Clone an existing instance template (using it as a base).
+2) Update metadata config of the newly created instance template to run a [cloud-init instructions file](https://github.com/mattes/gce-boot-scripts/tree/master/rails).
+3) Tell the instance group manager to perform a rolling update with the new instance template.
+
+Please review the [configuration options](https://github.com/mattes/gce-deploy-action) and get a better understanding how this step works in detail. Please also review the [rails helper utility](https://github.com/mattes/gce-boot-scripts/tree/master/rails) used in `deploys.*.vars.RUN` which starts a Rails Docker Image.
+
+```yaml
+common:
+  region: us-central1
+  cloud_init: https://raw.githubusercontent.com/mattes/gce-boot-scripts/v1.0.0/rails/dist/cloud-init.yml
+  vars:
+    DOCKER_IMAGE: gcr.io/my-project/my-app:${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
+    SECRETS_PREFIX: MY_APP
+  labels:
+    version: ${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
+    git-sha: ${{GITHUB_SHA}}
+
+deploys:
+  - name: app
+    instance_group: my-instance-group
+    instance_template_base: my-base-instance-template
+    instance_template: my-app-${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA:0:7}}
+    vars:
+      RUN: >
+        rails --service my-app --run "rails server -b 0.0.0.0 -p 3000" --port 3000 -e RAILS_SERVE_STATIC_FILES=true;
+
+  - name: worker
+    instance_group: my-worker-instance-group
+    instance_template_base: my-base-worker-instance-template
+    instance_template: my-app-worker-${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA:0:7}}
+    update_policy:
+      type: OPPORTUNISTIC
+    vars:
+      RUN: >
+        rails --service worker --container worker-1 --run "rails resque:work QUEUE='*'" --healthcheck-port 7000 --graceful-shutdown 300 --stop-signal SIGQUIT;
+        rails --service worker --container worker-2 --run "rails resque:work QUEUE='*'" --healthcheck-port 7001 --graceful-shutdown 300 --stop-signal SIGQUIT;
+```
+
+#### Default Service Account Roles
+
+Please make sure that the default service account used by the VM has role `Secret Manager Secret Accessor`. It must be added explicitly. This is for the default service account, not the service account you created for the Github Actions.
+
+#### Graceful shutdowns
+
+You'll notice how `deploys.worker.update_policy.type` is set to `OPPORTUNISTIC`. For app it defaults to `PROACTIVE`. What does that mean? 
+
+First it's important to understand that `app` is a *request-based* application serving *short-lived* HTTP requests, whereas `worker` runs *long-running* background jobs. Heroku would call `app` a Dyno, and `worker` just worker. How does it affect graceful shutdowns?
+
+##### App
+
+App and its Instance Group `my-instance-group` needs to use a Google Load Balancer with [connection draining](https://cloud.google.com/load-balancing/docs/enabling-connection-draining) enabled. This ensures that existing, in-progress requests are given time to complete when a VM is removed from an instance group. During a deploy new instances are started, while the old instances finish their pending requests and then shut down. This enables zero downtime deployments. Easy.
+
+##### Worker
+
+For workers it's a bit more complex, because unfortunately Google Shutdown scripts are not reliable and at best only can run for a couple of seconds. Using a [graceful-shutdown helper](https://github.com/mattes/gce-graceful-shutdown) and setting the update policy to `OPPORTUNISTIC` we can use `--graceful-shutdown [sec]` to manually handle the shutdown procedure ourselves.
+
+In the example shown above, `--graceful-shutdown 300` is set to 5 minutes. When a new deploy happens the Docker container will receive `SIGQUIT` (set by `--stop-signal`) and then has 5 minutes guaranteed to stop. If the container doesn't stop after this, it will receive a `SIGTERM` signal but is essentially allowed to finish to run until the instance is finally deleted by Google. Please note that the "delete this instance" call is made by the graceful-shutdown helper, not the Google Instance Group Manager itself, hence the `OPPORTUNISTIC` deploy type.
+
+To trap a signal inside Ruby, you could use:
+
+```ruby
+should_exit = false
+Signal.trap('TERM') { should_exit = true }
+
+while true; do
+  break if should_exit
+  # process
+end
+```
+
+#### Healthcheck helper
+
+If what you're running doesn't provide a health check out of the box, you can use `--healthcheck-port [port]` to start a [helper utility](https://github.com/mattes/healthcheck-cmd) which returns HTTP status code `200` if the docker container is running. If it's not running it will return  HTTP status code `500`.
+
+
+### 6) Create Google components
+
+Following the example `deploy.yml` from above, you will need to create:
+
+#### app
+
+* Google Cloud Load Balancer with Connection Draining enabled
+* Instance Template `my-base-instance-template`
+* Instance Group `my-instance-group` (with or without autoscaler)
+* Auto healing healtcheck for port 3000
+
+#### worker
+
+* Instance Template `my-base-worker-instance-template`
+* Instance Group `my-worker-instance-group` (with or without autoscaler)
+* Auto healing healtcheck for port 7000
+* Auto healing healtcheck for port 7001
+
+
+## Troubleshooting
+
+* __I need a rails console__  
+  SSH into any VM and just run `console`.
+
+* __Log files?__  
+  SSH into a VM and `tail -f /var/log/cloud-init-output.log` or check the Google Logs UI.
+
No results found