Troubleshooting Pod Issues

ImagePullBackOff

If you encounter an ImagePullBackOff error with one or more pods, the image may no longer be available upstream (i.e. no longer on DockerHub) or within the specified private repository.

It may be possible to still leverage this image if it is present one or more nodes, indicated by running pods for the same StatefulSet or Deployment. For example:

Bash
    
 
$ kubectl get pods -n <namespace> -o wide -l app=minioNAME      READY   STATUS             RESTARTS   AGE    IP            NODE          NOMINATED NODE   READINESS GATESminio-0   0/1     ImagePullBackOff   0          141m   10.3.70.76   10.2.99.39   <none>           <none>minio-1   1/1     Running            0          146m   10.3.4.4     10.2.99.42   <none>           <none>minio-2   1/1     Running            0          152m   10.3.92.8    10.2.99.43   <none>           <none>minio-3   1/1     Running            0          148m   10.3.17.4    10.2.99.44   <none>           <none>
Copy

Save/Load Docker Image from Cache

Identify the image – either by name, or image ID, which can be found in the kubectl describe pod or kubectl get pod output – for any Running pod.

Bash
    
 
$ kubectl get pod minio-1 -n <namespace> -o json | jq '.spec.containers[].image'"minio/minio:RELEASE.2021-05-27T22-06-31Z"
Copy

SSH to the Node on which the targeted Running pod resides, and list the Docker images.

Bash
    
 
$ sudo docker imagesREPOSITORY  TAG                            IMAGE ID            CREATED             SIZEminio/minio RELEASE.2021-05-27T22-06-31Z   375bd7b2cc2c        4 weeks ago         264MB
Copy

Save the Docker image – specifying the Image ID , Repository , and Tag .

Bash
    
 
$ sudo docker save <image-id> -o <filename.tar <repository>:<tag>
Copy

Note: If the <repository>:<tag> is not specified, the Docker image will be imported without any identifying information and Kubernetes will continue to fail to spawn the pod as a result.

Transfer the file (i.e. using rsync or scp ) to the destination Node where the pod is in ImagePullBackOff state.
Load the Docker image on that Node .

Bash
    
 
$ sudo docker load -i <filename>.tar
Copy

Example

Bash
    
 
$ sudo docker load -i <filename>.tarLoaded image ID: sha256:375bd7b2cc2ce76d6216ae68f68b54913a8f2df4a3d2c34919bd33d23a02f409
Copy

CrashLoopBackOff

NodeAffinity

Bash
    
 
$ kubectl get pods -A | grep -i sentryplatform9-system       pf9-sentry-688cfcf9-h52v6                    1/1     Running            1          3h22m*platform9-system       pf9-sentry-688cfcf9-s4fjm                    0/1     NodeAffinity       0          3h33m*
Copy

Bash
    
----     ------            ----   ----               -------  Warning  FailedScheduling  11m    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.  Warning  FailedScheduling  11m    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.  Normal   Scheduled         11m    default-scheduler  Successfully assigned platform9-system/pf9-sentry-688cfcf9-s4fjm to 10.128.230.29  Normal   Pulled            11m    kubelet            Container image "platform9/pf9-sentry:1.0.0" already present on machine  Normal   Created           11m    kubelet            Created container pf9-sentry  Normal   Started           11m    kubelet            Started container pf9-sentry  Warning  Unhealthy         6m29s  kubelet            Liveness probe failed: Get http://10.20.244.65:8080/v1/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)  Warning  NodeAffinity      87s    kubelet            Predicate NodeAffinity failed
Copy

You may delete the pod stuck in node affinity status as another pod is spawned for the same application.

Last updated by Chris Jones on Oct 21, 2021

Was this page helpful?