Revert a PV Stuck in a Deleting State
Problem
You accidentally deleted a PV but now it is stuck in a deleting state due to it being attached to a pod.
Environment
- Platform9 Managed Kubernetes - All Versions
Cause
The reason for PV not being deleted successfully is because the kubernetes.io/pv-protection finalizer protects it from being deleted while being attached to a pod.
Resolution
Please create an ETCD backup before proceeded with the below steps. If you are not sure about the commands that need to be run, reach out to our Support Team for requesting assistance.
- Login to the master node of the canary cluster and run the following command.
# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key get / --prefix --keys-only | grep -i 'persistentvolume'- In the output, you will find the name of the PV that is stuck in a Terminating state. Run the following command to get the PV data from ETCD:
# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key get /registry/persistentvolumes/<pv_name>- Copy the JSON output from the above command and use a text editor to remove the following 2 key value pairs:
DeletionTimestampDeletionGracePeriodSeconds- Once you have the edited JSON, use the following command to update the ETCD using the following command.
# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key put /registry/persistentvolumes/<pv_name> '<edited_JSON>'Was this page helpful?