A few things I have learned during the setup of my Gitlab/Registry System:

• The TLS section int config.yml is optional. Gitlab acts as a Reverse Proxy/SSL Termination for the registry. If you do drop the TLS section, change the GITLAB_REGISTRY_API_URL to `http://.
• rootcertbundle can point to ANY certificate, the only requirement is that GITLAB_REGISTRY_KEY_PATH points to the corrosponding private key of this certificate. I tested this with an expired key-pair for another domain and it worked just fine. I created my key-pair (certificate) like this:

openssl req -new -newkey rsa:4096 > registry.csr
openssl rsa -in privkey.pem -out registry.key
openssl x509 -in registry.csr -out registry.crt -req -signkey registry.key -days 3650

• It doesn't matter what information you put in the csr. I set the expiring time to 10 years, but it doesn't really matter as well.
• The values of issuer and GITLAB_REGISTRY_ISSUER have to match

Deployment Option 1: (I use this one, that doesn't mean you should)
Registry and Gitlab are running as Containers on the same host. Both are connected to a shared docker network. Neither are externally reachable. There is a Reverse Proxy handling all incoming connections to the host (I use HAProxy).

docker-compose.yml: (only the relevant parts, this will not give you a functional setup)

version: '2'

networks:
  default:
    ipam:
      config:
        - subnet: 172.50.0.0/24

services:
  gitlab:
    image: sameersbn/gitlab:8.8.2-registry
    ports:
      - 172.50.0.1:8022:22
      - 172.50.0.1:8080:80
    environment:
      - [...]
      - GITLAB_REGISTRY_ENABLED=true
      - GITLAB_REGISTRY_HOST=registry.example.com
      - GITLAB_REGISTRY_PORT=443
      - GITLAB_REGISTRY_API_URL=http://registry:5000
      - GITLAB_REGISTRY_KEY_PATH=/certs/registry.key

      - SSL_REGISTRY_KEY_PATH=/certs/registry.key
      - SSL_REGISTRY_CERT_PATH=/certs/registry.crt
    volumes:
      - ./certs:/certs
  registry:
    image: registry:2
    ports:
      - 172.50.0.1:5000:5000
    volumes:
      - ./registry/config/config.yml:/etc/docker/registry/config.yml
      - ./certs:/certs

note: GITLAB_REGISTRY_PORT could be empty, but even than the Gitlab UI shows a colon in the docker login/push/pull commands. Also, Gitlab fails to start because the (unused) reverse proxy fails to start. This may change in the future. For the same reason SSL_REGISTRY_KEY_PATH and SSL_REGISTRY_CERT_PATH could be omitted (but currently can't).

config.yml:

version: 0.1
log:
  fields:
    service: registry
http:
  secret: <some random string>
storage:
    cache:
        blobdescriptor: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: :5000
    headers:
        X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
auth:
  token:
    realm: https://gitlab.example.com/jwt/auth
    service: container_registry
    issuer: gitlab-issuer
    rootcertbundle: /certs/registry.crt

With this setup, Gitlab and the Registry are reachable only from the host. To actually use them, you need to setup a Reverse Proxy.
I use HAProxy
haproxy.cfg:

global
    [...]
    crt-base /etc/letsencrypt/live

frontend https
    bind <public-ipv4>:443 ssl crt example.com/haproxy.pem
    bind <public-ipv6>:443 ssl crt example.com/haproxy.pem
    mode http

    http-request add-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port 443

    use_backend bk_gitlab if { ssl_fc_sni gitlab.example.com }
    use_backend bk_registry if { ssl_fc_sni registry.example.com }

backend bk_gitlab
    server gitlab 172.50.0.1:8080

backend bk_registry
    server registry 172.50.0.1:5000

listen gitlab
    bind <public-ipv4>:22
    bind <public-ipv6>:22

    mode tcp
    option tcplog
    timeout client 1h
    timeout server 1h
    server gitlab_registry 172.50.0.1:8022

With this setup you don't have to include any ports in any commands:

git clone [email protected]/user/fancy-project
docker login registry.example.com
docker push registry.example.com/user/fancy-project:1.0

Deployment Option 2:
I haven't tried this one so I don't have examples.
Here you don't use docker networks but expose Gitlab and the Registry directly to the outside world.

gitlab:
  ports:
    - <your_public_ip>:22
    - <your_public_ip>:443
registry:
  ports:
    - <your_public_ip>:5000

In this case, you need to specify the TLS section in config.yml and point the API URL to https://example.com:5000
Again, the Reverse Proxy in the Gitlab Container for the Registry is not used.

Deployment Option 3:
Let Gitlab handle all connections, Registry is not accessible directly from the outside:

gitlab:
  links:
   - registry:registry
  ports:
    - <your_public_ip>:22
    - <your_public_ip>:443
    - <your_public_ip>:5005
  environment:
    - GITLAB_REGISTRY_API_URL=http://registry:5000
registry:
  [...]

In this scenario, you don't need the TLS section in config.yml

Deployment Option n:

Maybe Gitlab and Registry on different Hosts? I don't know, but I'm sure there are many more choices ;-)

on dev option3 can i push pull directly to the gitlab which handles those commands?

You can create the signing key pair non-interactively using:

#/bin/bash
echo "Create Signing Key and CSR"
openssl req -nodes -newkey rsa:2048 -keyout registry-auth.key -out registry-auth.csr -subj "/CN=gitlab-issuer"

echo "Self-Sign Certificate"
openssl x509 -in registry-auth.csr -out registry-auth.crt -req -signkey registry-auth.key -days 3650

@boonkerz sorry, I did not get notified, but yes you can

@cvle thanks! I think the /CN=gitlab-issuer is optional though, the certificate itself never get's checked for anything, only the private part must match the public part.

with self signed certs you have to do https://docs.docker.com/registry/insecure/

If its possible to use letsenrypt?

@boonkerz: yes.

Newer versions of Gitlab (>=9.4) now overwrite the key you generated with openssl, and you need to supply it in the registry['internal_key'] variable on startup (in addition to the file). It's horrible, but the workaround described in there works.

https://gitlab.com/gitlab-org/gitlab-ce/issues/35461

There's four hours of my life I won't get back.