Troubleshooting Pod Issues
ImagePullBackOff
If you encounter an ImagePullBackOff error with one or more pods, the image may no longer be available upstream (i.e. no longer on DockerHub) or within the specified private repository.
It may be possible to still leverage this image if it is present one or more nodes, indicated by running pods for the same StatefulSet or Deployment. For example:
$ kubectl get pods -n <namespace> -o wide -l app=minioNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESminio-0 0/1 ImagePullBackOff 0 141m 10.3.70.76 10.2.99.39 <none> <none>minio-1 1/1 Running 0 146m 10.3.4.4 10.2.99.42 <none> <none>minio-2 1/1 Running 0 152m 10.3.92.8 10.2.99.43 <none> <none>minio-3 1/1 Running 0 148m 10.3.17.4 10.2.99.44 <none> <none>Save/Load Docker Image from Cache
- Identify the image – either by name, or image ID, which can be found in the
kubectl describe podorkubectl get podoutput – for anyRunningpod.
$ kubectl get pod minio-1 -n <namespace> -o json | jq '.spec.containers[].image'"minio/minio:RELEASE.2021-05-27T22-06-31Z"- SSH to the
Nodeon which the targetedRunningpod resides, and list the Docker images.
$ sudo docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEminio/minio RELEASE.2021-05-27T22-06-31Z 375bd7b2cc2c 4 weeks ago 264MB- Save the Docker image – specifying the
Image ID,Repository, andTag.
$ sudo docker save <image-id> -o <filename.tar <repository>:<tag>Note: If the <repository>:<tag> is not specified, the Docker image will be imported without any identifying information and Kubernetes will continue to fail to spawn the pod as a result.
- Transfer the file (i.e. using
rsyncorscp) to the destinationNodewhere the pod is inImagePullBackOffstate. - Load the Docker image on that
Node.
$ sudo docker load -i <filename>.tarExample
$ sudo docker load -i <filename>.tarLoaded image ID: sha256:375bd7b2cc2ce76d6216ae68f68b54913a8f2df4a3d2c34919bd33d23a02f409Was this page helpful?