Nodelet Initiating Stop Chain on Failure due to Third Party Webhook

Problem

• Nodelet phases getting failed due to a failure while reaching a third party webhook.

C

{"L":"ERROR","T":"2024-07-04T03:07:35.903+0700","C":"misc/uncordon_node.go:103","M":"failed to remove annotation UserNodeCordon from node : failed to update node with removed annotations : Internal error occurred: failed calling webhook \"nodes.capsule.clastix.io\": failed to call webhook: Post \"https://capsule-webhook-service.capsule-system.svc:443/nodes?timeout=30s\": dial tcp 10.13.24.239:443: connect: connection refused"}

Environment

• Platform9 Managed Kubernetes - v5.0 and Higher

Cause

• Platform9 is still investigating the nodelet behavior via PMK-6529

Resolution

• The current workaround is to restart the nodelet stack.

$ /opt/pf9/nodelet/nodeletd phases stop --force
$ /opt/pf9/nodelet/nodeletd phases start --force