# Namespace stuck in "Terminating" state

## Problem

* Namespace stuck in terminating state and unable to delete it.

## Environment

* Platform9 Edge Cloud
* Platform9 Managed Kubernetes

## Cause

* There is a finalizer defined for the affected namespace.
* After executing commands like, `kubectl delete namespace <namespace>`, Kubernetes checks for a finalizer in the `metadata.finalizers` field. A finalizer is a special metadata key that tells Kubernetes to wait until a specific condition is met before it fully deletes a resource. If the condition defined in the finalizer cannot be deleted for any reason, then the namespace is not deleted either. This puts the namespace into a terminating state awaiting the removal of the resource, which never occurs.
* Check for the `kube-controller-manager` log files present on the master nodes to identify what are the resources that is failed.

{% tabs %}
{% tab title="sample - /var/log/pf9/kube/kube-controller-manager.log" %}

```bash
E0124 11:31:59.292922       1 namespaced_resources_deleter.go:161] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0124 11:32:07.190571       1 namespace_controller.go:162] deletion of namespace mtcil failed: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
```

{% endtab %}
{% endtabs %}

## Resolution

* Identify the resource which is waited up on or any error using below command:

{% tabs %}
{% tab title="command" %}

```javascript
# kubectl get ns <namespace> -oyaml
```

{% endtab %}
{% endtabs %}

* Start by checking the condition field. If any error or condition pointing towards the api-resouces get the list of all api-resources.

{% tabs %}
{% tab title="command" %}

```javascript
# kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace-name>
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example" %}

```bash
# kubectl api-resources --verbs=list --namespaced -o name 
....
error: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
```

{% endtab %}
{% endtabs %}

* If the issue is due to the api-resource that is not able to handle the request then identify the issue with that resource:

{% tabs %}
{% tab title="command" %}

```javascript
# kubectl get APIService <version>.<api-resource>
# kubectl describe APIService <version>.<api-resource>
```

{% endtab %}
{% endtabs %}

* If the apiservices and its corresponding pods/deployment is no longer running on the node then this could be deleted using below command and later point reinstall (if required) the corresponding addons/application that provides this api-resource

{% tabs %}
{% tab title="command" %}

```javascript
kubectl delete APIService <version>.<api-resource>
```

{% endtab %}
{% endtabs %}

* If there are no failed `api-resources` associated then, Dump the contents of the namespace in a temporary file, edit the temporary file to **remove** the `kubernetes` value from the `finalizers` field and save the file:

{% tabs %}
{% tab title="command" %}

```javascript
# kubectl get namespace <NAMESPACE> -o json > tmp.json
# vi tmp.json
```

{% endtab %}
{% endtabs %}

* Set a temporary proxy IP and port, using the following command. Be sure to keep your terminal window open until you delete the stuck namespace:

{% tabs %}
{% tab title="command" %}

```javascript
# kubectl proxy
Starting to serve on 127.0.0.1:8001
```

{% endtab %}
{% endtabs %}

* From a new terminal window, make an API call with your temporary proxy IP and port :

{% tabs %}
{% tab title="command" %}

```javascript
# curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize
```

{% endtab %}
{% endtabs %}

* Now verify if the namespace is removed.

{% tabs %}
{% tab title="command" %}

```javascript
# kubectl get namespaces
```

{% endtab %}
{% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/smcp/solution/namespace-stuck-in-terminating-state.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.