Stack Stuck in "Delete-In-Progress" State

Problem

• Deleting stack gets stuck withDELETE_IN_PROGRESS state

Bash

openstack stack list
+--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+
| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+
| d86ced7e-ba0a-4c08-854f-6cd772aa831a | lmfv_c426_kma_0 | b8ba4cd05ecf4e47a3002622574cd815 | DELETE_IN_PROGRESS | 2022-12-06T06:27:02Z | 2022-12-06T14:18:45Z |
| fe48b5d2-28ad-4912-9a5e-a16751032588 | lmfv_c426_base | b8ba4cd05ecf4e47a3002622574cd815 | DELETE_FAILED | 2022-12-06T06:00:57Z | 2022-12-06T14:18:45Z |
+--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+

• Below lines can be seen in the heat logs

Bash

cat /var/log/heat/heat-api-daemon.log:
...
2022-12-07 06:22:49,602.602 24651 ERROR heat.engine.worker [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Failed to stop all workers of stack lmfv_c426_km
a_0-kma_server_resource_group-7kilsgnv7trs, stack cancel not complete
2022-12-07 06:22:49,743.743 24651 INFO heat.engine.stack [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Triggering resource 349871 for cleanup
2022-12-07 06:22:49,822.822 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] deleting CinderVolumeAttachment "kma_volume_att
achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196]
2022-12-07 06:22:49,842.842 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_att
achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist
2022-12-07 06:22:49,945.945 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_att
achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist
2022-12-07 06:22:50,542.542 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_att
achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist

Environment

• Platform9 Managed OpenStack

Cause

• We can see there is a timeout while deleting the stack which leaves it in DELETE IN PROGRESS state**.**

Bash

2022-12-07 03:56:56Z [lmfv_c426_kma_0]: DELETE_IN_PROGRESS  Stack DELETE started
2022-12-07 03:56:56Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_IN_PROGRESS  state changed
2022-12-07 04:56:56Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_FAILED  DELETE aborted (Task delete from TemplateResource "kma_server_resource_group" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] Stack "lmfv_c426_kma_0" [d86ced7e-ba0a-4c08-854f-6cd772aa831a] Timed out)
2022-12-07 04:56:56Z [lmfv_c426_kma_0]: DELETE_FAILED  Timed out
2022-12-07 06:22:34Z [lmfv_c426_kma_0]: DELETE_IN_PROGRESS  Stack DELETE started
2022-12-07 06:22:34Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_IN_PROGRESS  state changed

$ cat /var/log/heat/heat-api-daemon.log:

2022-12-07 04:56:56,335.335 24650 INFO heat.engine.scheduler [req-0549a184-ce07-455f-b2b8-b2cd5b35e679 - admin - default default] Task delete from TemplateResource "kma_server_resource_group" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] Stack "lmfv_c426_kma_0" [d86ced7e-ba0a-4c08-854f-6cd772aa831a] timed out
2022-12-07 04:56:56,350.350 28263 INFO heat.common.wsgi [req-d7af259f-6281-4c52-bc02-017deadd610d - c413 - default default] Processing request: GET /heat//v1/1a37f3c84b7f4358846009ddd2e5ee4c/stacks/e911_c413_wrkr_4/4c23d329-3b7d-48bd-8805-fa9b66630f15/events
2022-12-07 04:56:56,377.377 24650 INFO heat.engine.stack [req-0549a184-ce07-455f-b2b8-b2cd5b35e679 - admin - default default] Stack DELETE FAILED (lmfv_c426_kma_0): Timed out
2022-12-07 04:56:56,404.404 28263 INFO eventlet.wsgi.server [req-d7af259f-6281-4c52-bc02-017deadd610d - c413 - default default] 199.165.246.9,127.0.0.1 - - [07/Dec/2022 04:56:56] "GET //v1/1a37f3c84b7f4358846009ddd2e5ee4c/stacks/e911_c413_wrkr_4/4c23d329-3b7d-48bd-8805-fa9b66630f15/events?marker=9af8e628-1add-4512-a84b-e625c0e61c41&sort_dir=asc HTTP/1.1" 200 188 0.071507

Resolution

• Try restarting the openstack-heat-engine service on the management plane.

Bash

systemctl restart openstack-heat-engine.service
systemctl status openstack-heat-engine.service -l