Unable Allocate IP to Pods From Specific IP-Pool After Node Reboot

Problem

IPs are not getting assigned to pods from the specified IP address pool after a node restart.

May 18 04:43:23 master1 kubelet: E0518 04:43:23.090622   39927 cni.go:366] Error adding pod-0/356eff34ccd2f3ad4c30238cd5fbc6d60892efa260eb159cc8187112075cf99a to network multus/multus-cni-network: [ns/pod-0:mh0]: error adding container to network "mh0": failed to allocate for range 0: no IP addresses available in range set: 2401:4900:14:434:0:b7:0:1d05-2401:4900:14:434:0:b7:0:1d05

Environment

Platform9 Edge Cloud - LTS-2 and below.
Whereabouts version v-0.4.10 and below.

Answer

The Engineering team has confirmed that the state is maintained in the dataDir in the ipam config. And you will have to manually cleanup the content from dataDir after every reboot .

This is an upstream bug which is fixed in whereabouts v0.4.10 in LTS2 release.

Info

Allocated IP addresses are stored as files in /var/lib/cni/networks/$NETWORK_NAME. The path can be customized with the dataDir option listed above. Environments where IPs are released automatically on reboot (e.g. running containers are not restored) may have to specify /var/run/cni or another tmpfs mounted directory instead.