Recover Persistent Volumes from "Terminating" State

Problem

• Accidentally performed delete operation on PV instead of PVC.

Bash

$ kubectl delete pv --all -n <Namespace>

• All PV's across the Cluster are in Terminating state. Need to recover or set it back to "Bound" state.

Environment

• Platform9 Managed Kubernetes - All versions

]Cause

• Default behaviour having the annotation kubernetes.io/pv-protection, it doesn't let controllers to delete PV till the time it is Bound with a PVC.

Resolution

• As a workaround, the below operations can be carried out to try recover the PV back to bound state.

  • Take the backup of the existing PV

Bash

kubectl get pv <pvname> -o yaml > pv.yaml

- Edit the existing PV and delete the `spec.claimRef`  section. This will delete the PV completely. The PVC would be in lost status initially when the PV is deleted

Bash

$ kubectl edit pv <pvname>

- Edit the backed up PV YAML and delete the `metadata.deletionTimestamp` line.

Bash

$ vi pv.yaml and delete the "metadata.deletionTimestamp" line

- Create the PV and wait for the status to be updated. After recreating the PV, the PVC should be back in bound status.

Bash

$ kubectl create -f pv.yaml
$ kubectl get pv
$ kubectl get pvc -n <namespace>