Stack deletion fails with 'DELETE_FAILED'

Problem

• When trying to delete stack, it stays in DELETE IN PROGRESS state and fails with DELETE_FAILED:

Javascript

# openstack stack list  --all  | grep -i test
| d86ced7e-aaaa-4c08-854f-6cd772aa831a | test           | b8ba4cd05ecf4e47a3002622574cd815 | DELETE_FAILED   | 2022-12-06T06:27:02Z | 2022-12-07T03:56:56Z |

Environment

• Platform9 Managed OpenStack - v5.4 and Higher

Cause

• openstack-heat-engine service was not responding to delete requests.

• Heat logs shows below error:

Javascript

$ cat /var/log/heat/heat-api-daemon.log:

2022-12-07 06:22:49,602.602 24651 ERROR heat.engine.worker [req-5bba22e9-aaaa-4483-a0da-e749af7d2b0b - admin - default default] Failed to stop all workers of stack test-kma_server_resource_group-ahdhfnfns, stack cancel not complete
2022-12-07 06:22:49,743.743 24651 INFO heat.engine.stack [req-5bba22e9-aaaa-4483-a0da-e749af7d2b0b - admin - default default] Triggering resource 349871 for cleanup

Resolution

• Restart heat-engine service and then try to delete the stack:

Javascript

# systemctl restart openstack-heat-engine.service