Skip to content

Instantly share code, notes, and snippets.

@dzintars

dzintars/project-quay-setup.md

Last active May 15, 2023 14:08
Show Gist options
  • Save dzintars/c93beed43f4eb1610af8307cb235eba3 to your computer and use it in GitHub Desktop.
Save dzintars/c93beed43f4eb1610af8307cb235eba3 to your computer and use it in GitHub Desktop.
Download ZIP
How to setup Quay Image Registry for the local development and testing
Raw
project-quay-setup.md

[UPDATE] by Dzintars Klavins @ 2023-05-15

These instructions is no more quite accurate. Current setup is much more simpler. Probably I will update these instructions at some point.

[WIP] by Dzintars Klavins @ 2021-08-28

Project Quay Setup

These instructions is written for myself as an reminder guide. It is higly possible that they are wrong, outdated, un-maintained. I do not carry any responsibility about following these instructions. Use at your own risk! If you have any suggestions, i will be happy to improve this document.

This setup utilizes podman play kube tool on Fedora 34 WorkStation. Intention of this setup is to use it for Kubernetes (Minikube) and Bazel (build system) for temporary image storage. The alternatives is to use Minikubes built-in simple registry or some of the public offerings. I don't like public services as I like to save some money and tinker with the new stuff.

General TODO

  • Make it rootless
  • Handle volume persistance corectly
  • Automate?

Prerequisites

  • Podman
  • Local storage (ideally would be neat to use Minio S3 from my oswee.ansible.minio role, but I left this for later)
  • Clean up the mounted volume directories if starting over. Don't forget to sudo ranger because files in those directories belongs to sudo user.

Bootstrap

Firewall

Get the active firewall zone

sudo firewall-cmd --get-active-zones

See which zone is used for your network interface. Most likely it will be public zone. Open the ports required for this setup

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --permanent --zone=public --add-port=9981/tcp
sudo firewall-cmd --reload

Add the image registry domain to the /etc/hosts file

echo '192.168.0.2 registry.domain.local' | sudo tee -a /etc/hosts

Or if you willing to add to the specific line then

sudo sed -i "2i192.168.0.2 registry.domain.local" /etc/hosts

where 2i prefix is the line nuber in which you want that record to appear.

Create the ~/quay/config-pod.yaml file with the content:

---
apiVersion: v1
kind: Pod
metadata:
  name: quay
  labels:
    app: quay
spec:
  restartPolicy: Always
  containers:
    - name: postgres
      image: docker.io/postgres:latest
      env:
        - name: POSTGRES_USER
          value: quayuser
        - name: POSTGRES_PASSWORD
          value: quaypass
        - name: POSTGRES_DB
          value: quay
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/postgresql/data:Z
          name: postgres-data

    - name: redis
      image: registry.redhat.io/rhel8/redis-5:1
      env:
        - name: REDIS_PASSWORD
          value: strongpassword
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/redis/data:Z
          name: redis-data

    - name: config
      image: quay.io/projectquay/quay:latest
      ports:
        - containerPort: 8080
          hostPort: 9981
          protocol: TCP
      args:
        - 'config'
        - 'secret'

  volumes:
    - name: postgres-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
        type: Directory
    - name: redis-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
        type: Directory

Make sure the volume directories exists

mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/config
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/storage

TODO: The volume location could be more conventional/generic

Run the sudo podaman play kube ~/quay/config-pod.yaml

Enable TRGM module

sudo podman exec -it quay-postgres /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm" | psql -d quay -U quayuser'

At this point you should have healthy Quay-Config, Redis and Postgresql containers running inside single quay pod.

Now it's time to generate configuration file for the Quay registry.

Access the configuration page at http://localhost:9981

Because I am using HAProxy with TLS termination (my proxy encrypts all traffic with walid TLS certificates) in this setup i will skip any TLS part. Thou... in general it is bad idea to run unencripted traffic inside of the perimeter and probably i will fix that some day soon.

For the Redis and Postgresql hosts use 127.0.0.1 IP address because that is how containers inside the Pod can comunicate. For "Database Server:" field shold be 127.0.0.1:5432. You will instantly see the "Please specify a non-localhost hostname" warning. We will deal with that later. For the "Redis Hostname:" field should be 127.0.0.1 Also add your username you will be using for Quay registry to the "Super Users:" field at the bottom of the page.

Now, the tricky part: In order for get the "Validate Configuration Changes" button available, we need temporary to replace 127.0.0.1 IP address for Redis and Postgresql with any valid domain, like example.com Then we need to open Chrome or Firefox Developer Tools and to select the "Validate Configuration Changes" button. In the source code we need to find the lines which looks like

<button class="btn btn-warning ng-scope ng-hide" ng-click="checkValidateAndSave()" ng-show="!configform.$valid">
  <i class="fa fa-lg fa-sort"></i>
  <!-- ngIf: configform.$error['required'].length -->
  <!-- ngIf: !configform.$error['required'].length --><span ng-if="!configform.$error['required'].length" class="ng-scope">
    Invalid configuration field
  </span><!-- end ngIf: !configform.$error['required'].length -->
</button>

and we need to change ng-show="!configform.$valid" to ng-show="configform.$valid" This will prevent from hiding the "Validate Configuration Changes" button when we use 127.0.0.1 IP address as databases host addresses.

Once validation is successful you need to download the config tar.

Export environment variable

export QUAY=/home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data

Move the tar to the Quay config directory

sudo tar -xvf ~/Downloads/quay-config.tar.gz -C $QUAY/config

Remove the Quay Config container from the pod

sudo podman stop quay-config

This should stop and remove the config container form the pod but leave the pod networking intact.

So, now inject the real registry container in the same pod

  sudo podman run --pod quay -d \
  --name=registry \
  -v $QUAY/config:/quay-registry/conf/stack:Z \
  -v $QUAY/storage:/datastorage:Z \
  quay.io/projectquay/quay:latest

As you see, I do not expose any ports there, because those ports already was exposed by Config container. We are lucky that Config and Registry containers uses 8080 port inside. :)

Check the logs

sudo podman logs quay-redis

If you see something like

1:M 29 Aug 2021 00:12:03.113 # Background saving error
1:M 29 Aug 2021 00:12:09.028 * 1 changes in 900 seconds. Saving...
1:M 29 Aug 2021 00:12:09.029 * Background saving started by pid 514
514:C 29 Aug 2021 00:12:09.029 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis/data) for saving: Permission denied

this means we have some problem with volume mount permissions. Most likely we need to use podman unshare and sudo setfacl -m u:26:-wx /some/volume/dir to set the right permissions. (UNVALIDATED/TODO)

Why all this magic?

My main intention was to expose as less ports as possible and make the setup more "portable" if i could say so. Managing single Pod is much more easier thand 3 or 4 separate conatiners. And I do really like play kube :)

Restore

To restore the existing setup we need another Pod manifest because we will use real Quay Registry image there. All other paramaters we leave intact.

Create new file

touch ~/quay/registry-pod.yaml

with the content of

---
apiVersion: v1
kind: Pod
metadata:
  name: quay
  labels:
    app: quay
spec:
  restartPolicy: Always
  containers:
    - name: postgres
      image: docker.io/postgres:latest
      env:
        - name: POSTGRES_USER
          value: quayuser
        - name: POSTGRES_PASSWORD
          value: quaypass
        - name: POSTGRES_DB
          value: quay
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/postgresql/data:Z
          name: postgres-data

    - name: redis
      image: registry.redhat.io/rhel8/redis-5:1
      env:
        - name: REDIS_PASSWORD
          value: strongpassword
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/redis/data:Z
          name: redis-data

    - name: registry
      image: quay.io/projectquay/quay:latest
      ports:
        - containerPort: 8080
          hostPort: 9981
          protocol: TCP
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /quay-registry/conf/stack:Z
          name: quay-config
        - mountPath: /datastorage:Z
          name: quay-storage

  volumes:
    - name: postgres-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
        type: Directory
    - name: redis-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
        type: Directory
    - name: quay-config
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/config
        type: Directory
    - name: quay-storage
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/storage
        type: Directory

Now, because we used volume mounts, all our data was persisted in those directories and new Pod will use the same configs and data. In theory... I haven't checked it yet. Writing this while i am doing all this. :)

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment