Unable to Delete Stacks Stuck in "DELETE_FAILED" State

Problem

Delete operation for OpenStack stack fails, leaving the stack in a DELETE FAILED state.

None

$ openstack stack list | grep DELETE_FAILED| c33fdc4b-36bb-431e-aa18-c8fa2ae60eef | demo_stack | DELETE_FAILED | 2019-08-16T07:55:26Z | 2019-08-17T15:15:06Z |

Environment

• Platform9 Managed OpenStack - v3.6.0 and Higher

• Heat

Cause

OpenStack Heat fails to delete a stack, leaving it in a DELETE_FAILED state, because the request timed out.

Resolution

1. Verify that all the servers in the stack which failed to delete are in a DELETING state.

None

$ openstack stack list  --all  | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack resource list -n 2 $i | grep -i server | awk '{ print $4}' | while read s; do openstack server show $s -c OS-EXT-STS:task_state -f value ; done; done

1. Perform a state reset for all the servers in these stacks.

None

$ openstack stack list  --all  | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack resource list -n 2 $i | grep -i server | awk '{ print $4}' | while read s; do openstack server set --state active $s ; done; done

1. Verify that there are no more servers in the DELETING state.

None

$ openstack stack list  --all  | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack resource list -n 2 $i | grep -i server | awk '{ print $4}' | while read s; do STATE=$(openstack server show $s -c OS-EXT-STS:task_state -f value); if [ "$STATE" == "deleting" ]; then echo $n; fi; done; done

1. Delete the DELETE_FAILED stacks.

None

$ openstack stack list  --all  | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack delete $i --yes; done