Namespace stuck in "Terminating" state

Problem

• Namespace stuck in terminating state and unable to delete it.

Environment

• Platform9 Edge Cloud

• Platform9 Managed Kubernetes

Cause

• There is a finalizer defined for the affected namespace.

• After executing commands like, kubectl delete namespace <namespace>, Kubernetes checks for a finalizer in the metadata.finalizers field. A finalizer is a special metadata key that tells Kubernetes to wait until a specific condition is met before it fully deletes a resource. If the condition defined in the finalizer cannot be deleted for any reason, then the namespace is not deleted either. This puts the namespace into a terminating state awaiting the removal of the resource, which never occurs.

• Check for the kube-controller-manager log files present on the master nodes to identify what are the resources that is failed.

sample - /var/log/pf9/kube/kube-controller-manager.log

E0124 11:31:59.292922       1 namespaced_resources_deleter.go:161] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0124 11:32:07.190571       1 namespace_controller.go:162] deletion of namespace mtcil failed: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Resolution

• Identify the resource which is waited up on or any error using below command:

command

# kubectl get ns <namespace> -oyaml

• Start by checking the condition field. If any error or condition pointing towards the api-resouces get the list of all api-resources.

command

# kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace-name>

Example

# kubectl api-resources --verbs=list --namespaced -o name 
....
error: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

• If the issue is due to the api-resource that is not able to handle the request then identify the issue with that resource:

command

# kubectl get APIService <version>.<api-resource>
# kubectl describe APIService <version>.<api-resource>

• If the apiservices and its corresponding pods/deployment is no longer running on the node then this could be deleted using below command and later point reinstall (if required) the corresponding addons/application that provides this api-resource

command

kubectl delete APIService <version>.<api-resource>

• If there are no failed api-resources associated then, Dump the contents of the namespace in a temporary file, edit the temporary file to remove the kubernetes value from the finalizers field and save the file:

command

# kubectl get namespace <NAMESPACE> -o json > tmp.json
# vi tmp.json

• Set a temporary proxy IP and port, using the following command. Be sure to keep your terminal window open until you delete the stuck namespace:

command

# kubectl proxy
Starting to serve on 127.0.0.1:8001

• From a new terminal window, make an API call with your temporary proxy IP and port :

command

# curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize

• Now verify if the namespace is removed.

command

# kubectl get namespaces