Kubevirt Cluster Add-ons Failing To Uninstall With Error "kubevirt-operator-webhook not found"
Problem
reflector.go:436] storage/cacher.go:/sunpike.platform9.com/clusteraddon: watch of *sunpike.ClusterAddon ended with: Internal error occurred: Object 'Kind' is missing in '{"metadata":{"name":"<cluster-uuid>-kubevirt","namespace":"default","uid":"<cluster-uuid>","creationTimestamp":"2022-06-28T11:11:45Z","deletionTimestamp":"2022-06-30T15:43:01Z","deletionGracePeriodSeconds":0,"labels":{"sunpike.pf9.io/cluster":"<cluster-uuid>","type":"kubevirt"},"finalizers":["addons.pf9.io"],"managedFields":[{"manager":"unknown","operation":"Update","apiVersion":"sunpike.platform9.com/v1alpha2","time":"2022-06-28T11:11:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:sunpike.pf9.io/cluster":{},"f:type":{}}},"f:spec":{"f:clusterID":{},"f:type":{},"f:version":{},"f:watch":{}}}},{"manager":"sunpike-controller-manager","operation":"Update","apiVersion":"sunpike.platform9.com/v1alpha2","time":"2022-06-28T11:13:32Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:finalizers":{".":{},"v:\"addons.pf9.io\"":{}}}}},{"manager":"manager","operation":"Update","apiVersion":"sunpike.platform9.com/v1alpha2","time":"2022-07-01T04:11:40Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:healthy":{},"f:message":{},"f:phase":{}}}}]},"spec":{"clusterID":"<cluster-uuid>","version":"0.45.0","type":"kubevirt","override":{},"watch":true},"status":{"phase":"UninstallAddonError","message":"could not delete object (cdi.kubevirt.io/v1beta1, Kind=CDI) /cdi: Internal error occurred: failed calling webhook \"cdi-validate.cdi.kubevirt.io\": Post \"https://cdi-api.cdi.svc:443/cdi-validate?timeout=30s\": x509: certificate has expired or is not yet valid: current time 2022-06-30T15:44:34Z is after 2022-06-29T05:49:15Z","healthy":true,"lastChecked":null}}
message: 'could not delete object (kubevirt.io/v1, Kind=KubeVirt) kubevirt/kubevirt: Internal error occurred: failed calling webhook "kubevirt-validator.kubevirt.io": Post "https://kubevirt-operator-webhook.kubevirt.svc:443/kubevirt-validate-delete?timeout=10s": service "kubevirt-operator-webhook" not found'
Environment
- Platform9 Managed Kubernetes - v5.4 and Higher
- Kubevirt Cluster Add-ons
Cause
This is a known issue and Platform9 Engineering is working on the fix in KVIRT-131. Please reach out to Support to know the status of the JIRA
Workaround
- Scale down the
pf9-addon-operator
pod:
$ kubectl -n pf9-addons scale --replicas=0 deployment pf9-addon-operator
Remove finalizer from addon from pf9-addons namespace if kubevirt addons is stuck.
- Edit pf9-addons deployment and set
DISABLE_SUNPIKE_SYNC
value as"true"
$ kubectl edit deploy pf9-addon-operator -n pf9-addons
- Make sure the all the resources are cleaned up from
cdi
andkubevirt
namespaces
$ kubectl get all -n cdi
$ kubectl get all -n kubevirt
- Once the resources are deleted scale the replica of
pf9-addon-operator
to 1 and setDISABLE_SUNPIKE_SYNC
value as"false"
Please reach out to Platform9 Support for any assistance or queries.
Was this page helpful?