Recover Persistent Volumes from "Terminating" State

Problem

• Accidentally performed delete operation on PV instead of PVC.

$ kubectl delete pv --all -n <Namespace>

• All PV's across the cluster are in Terminating state. Need to recover or set it back to "Bound" state.

Environment

• Platform9 Managed Kubernetes - All versions

Cause

• Default behaviour as a part of the annotation kubernetes.io/pv-protection . It doesn't let controllers to delete PV till the time it is Bound with a PVC.

Resolution

As a workaround, the below operations can be carried out to try recover the PV back to bound state.

• Take the backup of the existing PV

$ kubectl get pv <pvname> -o yaml > pvname.yaml

• Patch the respective PV and remove the spec.claimRef section. This will delete the PV completely. The PVC would be in Lost status initially when the PV is deleted. Make sure the ReclaimPolicy is set to Retain to avoid data loss.

$ kubectl patch pv <PV-name> -p '{"spec":{"claimRef": null}}'

• Edit the backed up PV YAML and delete the metadata.deletionTimestamp line.

$ vi pvname.yaml

• Recreate the PV and wait for the status to be updated. After recreating the PV, the PVC should be back in Bound status.

$ kubectl create -f pvname.yaml

$ kubectl get pv

$ kubectl get pvc -n <namespace>