Revert a PV Stuck in a Deleting State

Problem

You accidentally deleted a PV but now it is stuck in a deleting state due to it being attached to a pod.

Environment

• Platform9 Managed Kubernetes - All Versions

Cause

The reason for PV not being deleted successfully is because the kubernetes.io/pv-protection finalizer protects it from being deleted while being attached to a pod.

Resolution

Warning

Please create an ETCD backup before proceeded with the below steps. If you are not sure about the commands that need to be run, reach out to our Support Team for requesting assistance.

1. Login to the master node of the canary cluster and run the following command.

Javascript

# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key get / --prefix --keys-only | grep -i 'persistentvolume'

1. In the output, you will find the name of the PV that is stuck in a Terminating state. Run the following command to get the PV data from ETCD:

Javascript

# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key get /registry/persistentvolumes/<pv_name>

1. Copy the JSON output from the above command and use a text editor to remove the following 2 key value pairs:

Javascript

DeletionTimestamp
DeletionGracePeriodSeconds

1. Once you have the edited JSON, use the following command to update the ETCD using the following command.

Javascript

# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key put /registry/persistentvolumes/<pv_name> '<edited_JSON>'