How to Clean Luigi and Kubevirt Stale Entries After disabling the Luigin Plugins

Problem

After disabling the Luigin plugins, stale network erslated entries are present, which results in pod creation failures with below errors:

Events:

  Type     Reason          Age                    From     Message

  ----     ------          ----                   ----     -------

  Normal   SandboxChanged  34m (x141 over 64m)    kubelet  Pod sandbox changed, it will be killed and re-created.

  Warning  FailedKillPod   3m52s (x130 over 31m)  kubelet  error killing pod: failed to "KillPodSandbox" for "exxxxxxxx" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"xxxx\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): Multus: [default/testpod1]: error getting pod: Unauthorized"

Environment

• Platform9 Managed Kubernetes- 5.7.0 and higher.

Procedure

The workaround steps to be followed are:

1. Using network plugin CRD, first disable the networks that are not needed.

# kubectl edit networkplugins -n luigi-system -o yaml

2. Comment out the plugins that are not needed and save the CRD. Check for those plugin pods and daemonsets to be deleted automatically.

Note- Delete the network plugin CRD for unwanted plugins.

# kubectl get networkplugins -A

# kubectl edit networkplugins -n luigi-system -o yaml

3. Delete the '- teardownPlugins' under finalizers and run the below delete:

# kubectl delete networkplugins -n luigi-system   networkplugins-sample-default

4. And remove multus folders and conf file under /etc/cni/net.d/ on all the nodes:

# rm -rf 00-multus.conf multus.d/

5. After all the plugins are commented out and are been deleted, disable the Luigi add-on from UI.

Wait for the add-on to be removed.

# kubectl get all -n luigi-systemx

6. Re-enabling the luigi addon works fine after this.

Additional Information

An internal jira- PMK-6521 has been file to automate this process. And the fix will be available in upcoming PMK-5.11 version.