Pull Images from Internet in a limited Internet Access Environment.

Problem

• Unable to pull images from Internet(Eg. docker.io) in an environment having limited access to Internet.
• Even after whitelisting docker.io in the proxy, the image pull fails with the below events. Example image shown here is _docker.io/library/nginx:latest_

Type     Reason   Age                From     Message

  ----     ------   ----               ----     -------

  Warning  Failed   21s                kubelet  Failed to pull image "nginx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head "https://dockermirror.platform9.io/v2/library/nginx/manifests/latest?ns=docker.io": dial tcp 34.212.72.27:443: i/o timeout

  Warning  Failed   21s                kubelet  Error: ErrImagePull

  Normal   BackOff  20s                kubelet  Back-off pulling image "nginx"

Environment

• Platform9 Managed Kubernetes
• Platform9 Edge Cloud

Cause

• Platform9 caches Images in _https://dockermirror.platform9.io_ for all the pulls done from docker.io . This registry mirror is to avoid rate-limiting from docker.io and for faster pulls.
• If an environment with limited Internet access having docker.io whitelisted in the proxy, tries to pull the images, it will eventually fail as it will initially try to reach _https://dockermirror.platform9.io_ , which the environment would probably not have access to.

Resolution

• Whitelist the URL _https://dockermirror.platform9.io_ in the proxy that the cluster is configured with.
• In order for the container runtime to be aware of the proxy while pulling images, create a /etc/environment file on each node and add the below content in it

$ cat /etc/environment

export LC_ALL=en_US.UTF-8

export http_proxy=http://<proxy-url>:<proxy-port>

export https_proxy=http://<proxy-url>:<proxy-port>

export no_proxy=<list of node IPs/CIDR that doen't need proxy to be routed through. Eg. Internal nodes of a cluster>

• Once the above is done, restart the Nodeletd phases in a rolling fashion

$ systemctl stop pf9-hostagent pf9-nodeletd

$ /opt/pf9/nodelet/nodeletd phases restart

$ systemctl start pf9-hostagent pf9-nodeletd

• Try deploying a pod using image from docker.io