Whereabouts Pods Failing After Cluster Upgrade (1.24.7 to 1.25)

Problem

An issue has been observed where post upgrading from (PMK) 5.7.3 with Kubernetes v1.24.7 to PMK 5.9.4 with Kubernetes v1.25, the whereabouts pods enter CrashLoopBackOff state.

whereabouts-4dkvd                          1/1     Running            1             93m

whereabouts-fzrn5                          0/1     CrashLoopBackOff   5 (16s ago)   3m16s

whereabouts-kg9d6                          1/1     Running            1             93m

whereabouts-lbxcd                          1/1     Running            1             93m

whereabouts-swxqm                          0/1     CrashLoopBackOff   5 (50s ago)   3m48s

Environment

• Platform9 Managed Kubernetes 5.7.3 and 5.9.4.
• Kubernetes versions: v1.24.7-pmk.240 and v1.25.
• Whereabouts version: v0.4.10

Cause

After the upgrade process, the Whereabouts pods attempt to restart but fails with the following error:

# kubectl logs -f --tail 10 -n kube-system whereabouts-fzrn5  

Done configuring CNI.  Sleep=false  

/bin/sh: /ip-control-loop: not found

This occurs because the whereabouts Image field in the NetworkPlugins (CRD) still points to an older image (v0.4.10). Meanwhile, new arguments (SLEEP=false /install-cni.sh && /ip-control-loop -log-level debug) have been introduced in Whereabouts v0.6.3. However, since the image is not updated automatically, the outdated image does not support these arguments, leading to a failure.

Resolution

The recommended fix is to remove the whereaboutsImage field from the NetworkPlugins CRD. This allows Luigi Controller to update Whereabouts to the latest compatible version (v0.6.3), which includes the required changes.

1. Edit the NetworkPlugins CRD:

# kubectl edit networkplugins networkplugins-<NAME>

2. Locate and remove the following line:

whereaboutsImage: docker.io/platform9/whereabouts:v0.4.10

3. Save and exit the editor.
4. Wait for the Whereabouts pods to perform a rolling update. Further validate:

kubectl get pods -n kube-system | grep whereabouts

5. Ensure the pods are running without CrashLoopBackOff status.
6. Check the updated NetworkPlugins CRD:

kubectl get networkplugins networkplugins-<NAME> -o yaml

Expected output should no longer contain whereaboutsImage under spec.plugins.whereabouts, allowing Luigi Controller to update it automatically to v0.6.3.

Additional Information

The issue is researched under internal bug PMK-6650.