Pull Images from Internet in a limited Internet Access Environment.

Problem

• Unable to pull images from Internet(Eg. docker.io) in an environment having limited access to Internet.

• Even after whitelisting docker.io in the proxy, the image pull fails with the below events. Example image shown here is _docker.io/library/nginx:latest_

Bash

Type     Reason   Age                From     Message
  ----     ------   ----               ----     -------
  Warning  Failed   21s                kubelet  Failed to pull image "nginx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head "https://dockermirror.platform9.io/v2/library/nginx/manifests/latest?ns=docker.io": dial tcp 34.212.72.27:443: i/o timeout
  Warning  Failed   21s                kubelet  Error: ErrImagePull
  Normal   BackOff  20s                kubelet  Back-off pulling image "nginx"

Environment

• Platform9 Managed Kubernetes

• Platform9 Edge Cloud

Cause

• Platform9 caches Images in _https://dockermirror.platform9.io_ for all the pulls done from docker.io . This registry mirror is to avoid rate-limiting from docker.io and for faster pulls.

• If an environment with limited Internet access having docker.io whitelisted in the proxy, tries to pull the images, it will eventually fail as it will initially try to reach _https://dockermirror.platform9.io_ , which the environment would probably not have access to.

Resolution

• Whitelist the URL _https://dockermirror.platform9.io_ in the proxy that the cluster is configured with.

• In order for the container runtime to be aware of the proxy while pulling images, create a /etc/environment file on each node and add the below content in it

bash

$ cat /etc/environment
export LC_ALL=en_US.UTF-8
export http_proxy=http://<proxy-url>:<proxy-port>
export https_proxy=http://<proxy-url>:<proxy-port>
export no_proxy=<list of node IPs/CIDR that doen't need proxy to be routed through. Eg. Internal nodes of a cluster>

• Once the above is done, restart the Nodeletd phases in a rolling fashion

Bash

$ systemctl stop pf9-hostagent pf9-nodeletd
$ /opt/pf9/nodelet/nodeletd phases restart
$ systemctl start pf9-hostagent pf9-nodeletd

• Try deploying a pod using image from docker.io