Using Registry Mirror for Container Images

What is a Container Registry

A container registry is a catalogue of storage locations where you can push and pull container images.

However, the actual physical locations where images are stored are known as repositories. Each repository stores a collection of related images with the same name.

Each image within a repository represents a different version of the same container deployment. For example, on the popular container registry Docker Hub, nginx is the name of the repository that contains different versions of the Docker image for open-source web server installation NGINX.

A specific image is identified by either its tag or its own unique reference.

What is a Registry Mirror

A registry mirror is a local copy of a public registry that copies or mirrors the file structure of the public registry. For example, let's say that the path to your registry mirror is 10.101.101.100:443. When containerd encounters a request to pull an image such as gcr.io/etcd-development/etcd:v3.5.5, containerd attempts to pull that image, not from gcr.io, but from your registry at the following path: 10.101.101.100:443/etcd-development/etcd:v3.5.5. If the image isn't in this registry path, the image is pulled automatically from the public registry gcr.io.

Using a registry mirror provides the following benefits:

Can be used as custom registry for images.
Protects from public registry outages.
Speeds up pod creation.
Allows to conduct image vulnerability scanning.
Avoids limits imposed by public registries on how frequently you can issue commands to them.

Using Registry Mirror with PMK clusters

Pre-requisites

Container registry server set up in your network.
If your registry server runs a private TLS certificate, you must have the certificate authority (CA) file.
If your registry server needs authentication, you must have the login credentials.
Must use containerd as a runtime.
Supported only with PMK Qbert API and must be used to configure clusters.
PMK support only the following repositories with this feature. Images with any other registry URL will be directly downloaded from the mentioned registry (See in figure - anyOtherRegistry).
- platform9.io
- docker.io
- gcr.io
- quay.io
- registry.k8s.io

Understanding the API Body changes in cluster API

API spec located at https://platform9.com/docs/qbert/ref

Example:

Bash
    
 
curl --request POST \ --url https://<account>.platform9.net/qbert/v4/{projectId}/clusters \ --header "X-Auth-Token: {X-Auth-Token}" \ --data '{"customRegistryUrl": "https://10.101.101.100:443","customRegistryRepoPath": "infra/k8s""customRegistryUsername": "username","customRegistryPassword": "password","customRegistrySkipTls": false,"customRegistrySelfSignedCerts": true,"customRegistryCertPath": "/usr/local/share/reg-ca.crt",...
Copy

API Parameter	Type	Description	default	Example
customRegistryUrl	string	Endpoint URL for the registry.	null	Eg: https://10.101.101.100:443, https://user.jfrog.io
customRegistryRepoPath		Path where all images are cached.	null	Can be empty, meaning all images are cached at root. Can be like `infra` where all images will be cached in `infra` repository. Can be like `infra/k8s` where all images will be cached under `infra/k8s`
customRegistryUsername	string	Username for the registry.	null
customRegistryPassword	string	Password for the registry.	null
customRegistrySkipTls	boolean	To be used to skip TLS.	false
customRegistrySelfSignedCerts	boolean	To be used if the the registry has a self-signed certificate.	false
customRegistryCertPath	string	Path to the cert file on the nodes or the endpoint URL from where it can be fetched.	null	Can be of the following formats: https://example.com/path/to/cert http://example.com/path/to/cert file:///path/to/cert /path/to/cert

Note: Before using the API

Add the CA certificate file (ca.cert file) at specified location "customRegistryCertPath": "/usr/local/share/reg-ca.crt" on the node. (this path should be readable by pf9 user) OR
Store the CA certificate file at some endpoint say https://example.com/cacert and set this field to "customRegistryCertPath": "https://example.com/cacert" so that the nodelet can fetch the cert from this endpoint and place it on nodes and configure it into containerd.

Example Registry

As a user if you have setvup a private Jfrog registry like the below image:

API request body will have following parameters to use this feature:

Bash
    
 
"customRegistryUrl": "https://schoudhari.jfrog.io","customRegistryPath": "infra/k8s""customRegistryUsername": "username","customRegistryPassword": "password",
Copy

How this feature affects containerd?

Containerd configurations for registries are stored at /etc/containerd/certs.d on the nodes.

For example, if 10.101.101.100:443 is used as the registry mirror and in PMK there are multiple images from various registries, the /etc/containerd/certs.d will look like:

Bash
    
 
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# pwd/etc/containerd/certs.d[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# tree.|-- 10.101.101.100:443|   `-- hosts.toml|-- docker.io|   `-- hosts.toml|-- gcr.io|   `-- hosts.toml|-- platform9.io|   `-- hosts.toml|-- quay.io|   `-- hosts.toml`-- registry.k8s.io    `-- hosts.toml
Copy

PMK currently has above 5 registries. Along with that, we configure the custom registry 10.101.101.100:443 . The pull requests to all the registries will go through config in their hosts.toml.

Listed are examples of all registry hosts.toml files for this case:

10.101.101.100:443 ← custom registry

Bash
    
​x
 
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat 10.101.101.100\:443/hosts.toml server = "https://10.101.101.100:443"​[host."https://10.101.101.100:443/v2/infra/k8s"]  capabilities = ["pull", "resolve"]  ca = "/usr/local/share/reg-ca.crt"  override_path = true  [host."https://10.101.101.100:443/v2/infra/k8s".header]  authorization = "Basic d1h1YmhhbTpwZjck"
Copy

docker.io

Bash
    
 
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat docker.io/hosts.toml server = "https://docker.io"​[host."https://10.101.101.100:443/v2/infra/k8s"]  capabilities = ["pull", "resolve"]  ca = "/usr/local/share/reg-ca.crt"  override_path = true  [host."https://10.101.101.100:443/v2/infra/k8s".header]  authorization = "Basic d1h1YmhhbTpwZjck"​[host."https://dockermirror.platform9.io"]  capabilities = ["pull", "resolve"]​[host."https://registry-1.docker.io"]  capabilities = ["pull", "resolve"]
Copy

platform9.io

Bash
    
 
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat platform9.io/hosts.toml server = "https://platform9.io"​[host."https://10.101.101.100:443/v2/infra/k8s"]  capabilities = ["pull", "resolve"]  ca = "/usr/local/share/reg-ca.crt"  override_path = true  [host."https://10.101.101.100:443/v2/infra/k8s".header]  authorization = "Basic d1h1YmhhbTpwZjck"​[host."https://dockermirror.platform9.io"]  capabilities = ["pull", "resolve"]
Copy

quay.io

Bash
    
 
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat quay.io/hosts.toml server = "https://quay.io"​[host."https://10.101.101.100:443/v2/infra/k8s"]  capabilities = ["pull", "resolve"]  ca = "/usr/local/share/reg-ca.crt"  override_path = true  [host."https://10.101.101.100:443/v2/infra/k8s".header]  authorization = "Basic d1h1YmhhbTpwZjck"  [host."https://dockermirror.platform9.io"]  capabilities = ["pull", "resolve"]
Copy

gcr.io

Bash
    
 
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat gcr.io/hosts.toml server = "https://gcr.io"​[host."https://10.101.101.100:443/v2/infra/k8s"]  capabilities = ["pull", "resolve"]  ca = "/usr/local/share/reg-ca.crt"  override_path = true  [host."https://10.101.101.100:443/v2/infra/k8s".header]  authorization = "Basic d1h1YmhhbTpwZjck"  [host."https://dockermirror.platform9.io"]  capabilities = ["pull", "resolve"]
Copy

registry.k8s.io

Bash
    
 
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat registry.k8s.io/hosts.toml server = "https://registry.k8s.io"​[host."https://10.101.101.100:443/v2/infra/k8s"]  capabilities = ["pull", "resolve"]  ca = "/usr/local/share/reg-ca.crt"  override_path = true  [host."https://10.101.101.100:443/v2/infra/k8s".header]  authorization = "Basic d1h1YmhhbTpwZjck"  [host."https://dockermirror.platform9.io"]  capabilities = ["pull", "resolve"]
Copy

Using this feature during cluster creation

Cluster Create (API Reference)

Use the Cluster POST API with the following parameters to use this feature. Example:

Bash
    
 
curl --request POST \ --url https://<account>.platform9.net/qbert/v4/{projectId}/clusters \ --header "X-Auth-Token: {X-Auth-Token}" \ --data '{  "customRegistryUrl": "https://10.101.101.100:443",  "customRegistryUsername": "user",  "customRegistryRepoPath": "infra/k8s",  "customRegistryPassword": "password",  "customRegistrySkipTls": false,  "customRegistrySelfSignedCerts": true,  "customRegistryCertPath": "http://10.101.101.100:8080/domain.crt",  .  .  .
Copy

Using this feature with an existing cluster with cluster Kubernetes version upgrade

Cluster Create (API Reference)

Use the Cluster POST API with the following parameters to use this feature. Example:

Bash
    
 
curl --request POST \ --url https://<account>.platform9.net/qbert/v4/{project_uuid}/clusters/{uuid}/upgrade?type={type}&force={force} \ --header "X-Auth-Token: {X-Auth-Token}" \ --data '{  "customRegistryUrl": "{string}",  "customRegistryRepoPath": "{string}",  "customRegistryUsername": "{string}",  "customRegistryPassword": "{string}",  "customRegistrySkipTls": "{boolean}",  "customRegistrySelfSignedCerts": "{boolean}",  "customRegistryCertPath": "{string}",  .  .  .
Copy

This shall update all the the nodes with new containerd configuration.

Using this feature with an existing cluster with cluster update (no Kubernetes version upgrade)

Cluster Create (API Reference)

Use the Cluster PUT API with the following parameters to use this feature. Example:

Bash
    
 
curl --request PUT \ --url https://<account>.platform9.net/qbert/v4/{project_uuid}/clusters/{uuid} \ --header "X-Auth-Token: {X-Auth-Token}" \ --data '{  "customRegistryUrl": "{string}",  "customRegistryRepoPath": "{string}",  "customRegistryUsername": "{string}",  "customRegistryPassword": "{string}",  "customRegistrySkipTls": "{boolean}",  "customRegistrySelfSignedCerts": "{boolean}",  "customRegistryCertPath": "{string}",  .  .  .
Copy

Note: With cluster update, following commands have to be run on each node of the cluster to restart the nodelet phases. With cluster version upgrade, following takes places automatically during node upgrades.

Example:

Bash
    
 
root@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# systemctl stop pf9-hostagentroot@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# systemctl stop pf9-nodeletdroot@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# su pf9 -s /bin/bashpf9@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d$ /opt/pf9/nodelet/nodeletd phases stoproot@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# systemctl start pf9-hostagent
Copy

Last updated by Anmol Sachan on Mar 19, 2024

Was this page helpful?