IP-Reconciler Skips Next Pods if it Fails to Add IP to a Pod from the List

Problem

• IP-Reconciler skips next pods if it fails to add IP to a pod from the list.

• Whereabout adds IPs to the pods like below:

Pod logs

2023-05-25T20:55:38.000Z        DEBUG   listing IP pools
2023-05-25T20:55:38.300Z        DEBUG   Added IP 10.177.156.2 for pod ttmst417/ttmst417-mstore-0
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.66 for pod ttmst417/ttmst417-mstore-0
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.3 for pod ttmst417/ttmst417-mstore-1

• But, if adding IP to any of the pod fails then it does not add IP any of the further(successor) pods from that set. In the traces shown below we can see that it was able to add IPs till pod ttmst417-mstore-13 and then it failed to add IP on pod ttmst417-mstore-14.

• So, instead of continuing to add IPs on further(successor) pods (mstore-15, mstore-16, etc), it jumped to next set of pods which is ttmst417-mstore-2. Therefore, here it seems like it sorts with naming convention (alphanumeric) and hence, after mstore-14 failed, next sequence it found was mstore-2

Pod logs

2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.78 for pod ttmst417/ttmst417-mstore-12
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.15 for pod ttmst417/ttmst417-mstore-13
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.79 for pod ttmst417/ttmst417-mstore-13
2023-05-25T20:55:38.301Z        ERROR   could not parse network annotation  for pod: ttmst417/ttmst417-mstore-14; error: unexpected end of JSON input    <<==========
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.4 for pod ttmst417/ttmst417-mstore-2
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.68 for pod ttmst417/ttmst417-mstore-2
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.5 for pod ttmst417/ttmst417-mstore-3
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.69 for pod ttmst417/ttmst417-mstore-3
2023-05-25T20:55:38.301Z        DEBUG   Added IP 10.177.156.7 for pod ttmst417/ttmst417-mstore-5

Environment

• Platform9 Managed Kubernetes - v5.3 and Higher

Cause

This is known issue with ip-reconciler and is being discussed as per JIRA AIR-1079.

Resolution

• As a workaround restart the affected pod so that new IP is added to the pod.

• Upgrade to whereabouts version 0.4.11 as a permanent fix.

Additional Information

More details mentioned in the below function:

https://github.com/k8snetworkplumbingwg/whereabouts/blob/master/pkg/reconciler/wrappedPod.go#L57