Why Namespace Deletion is Stuck Due to Finalizers

Problem

Kubernetes namespace is stuck in terminating state due to finalizers.

Javascript

# kubectl get ns test-ns
NAME           STATUS          AGE
test-ns        Terminating     15m

Environment

• Platform9 Managed Kubernetes - All Versions

Answer

• Sometimes the process to delete namespaces gets stuck, and the command never completes. While the command returns a message showing that the namespace was deleted, querying it indicates that it's actually in a Terminating state.

• On some occasions, namespaces have a finalizer defined under spec. A finalizer is a special metadata key that tells Kubernetes to wait until a specific condition is met before it fully deletes a resource.

• So when one runs a command like kubectl delete namespace abcd, Kubernetes checks for a finalizer in the metadata.finalizers field. If the resource defined in the finalizer cannot be deleted for any reason, then the namespace is not deleted either. This puts the namespace into a terminating state awaiting the removal of the resource, which never occurs.

$ kubectl delete namespace test-ns
namespace "test-ns" deleted

$ kubectl get namespace
NAME           STATUS    	      AGE
test-ns    	   Terminating      15m

• The correct way to avoid this issue is to make sure that all the resources in the namespace are deleted before you delete a namespace. As a workaround, removing the finalizers will allow the stuck namespace to be deleted.

• The finalizers can be removed for the affected namespace using the steps mentioned below.

Steps to remove Finalizers From Stuck Namespace

# kubectl get namespace <stuck_namespace> -o json > <stuck_namespace>.json
# kubectl replace --raw "/api/v1/namespaces/<stuck_namespace>/finalize" -f ./<stuck_namespace>.json
# kubectl get namespace