Stack Stuck in "Delete-In-Progress" State
Problem
- Deleting stack gets stuck with
DELETE_IN_PROGRESSstate
openstack stack list+--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |+--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+| d86ced7e-ba0a-4c08-854f-6cd772aa831a | lmfv_c426_kma_0 | b8ba4cd05ecf4e47a3002622574cd815 | DELETE_IN_PROGRESS | 2022-12-06T06:27:02Z | 2022-12-06T14:18:45Z || fe48b5d2-28ad-4912-9a5e-a16751032588 | lmfv_c426_base | b8ba4cd05ecf4e47a3002622574cd815 | DELETE_FAILED | 2022-12-06T06:00:57Z | 2022-12-06T14:18:45Z |+--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+- Below lines can be seen in the heat logs
cat /var/log/heat/heat-api-daemon.log:...2022-12-07 06:22:49,602.602 24651 ERROR heat.engine.worker [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Failed to stop all workers of stack lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs, stack cancel not complete2022-12-07 06:22:49,743.743 24651 INFO heat.engine.stack [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Triggering resource 349871 for cleanup2022-12-07 06:22:49,822.822 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] deleting CinderVolumeAttachment "kma_volume_attachment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196]2022-12-07 06:22:49,842.842 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_attachment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist2022-12-07 06:22:49,945.945 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_attachment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist2022-12-07 06:22:50,542.542 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_attachment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not existEnvironment
- Platform9 Managed OpenStack
Cause
- We can see there is a timeout while deleting the stack which leaves it in DELETE IN PROGRESS state.
x
2022-12-07 03:56:56Z [lmfv_c426_kma_0]: DELETE_IN_PROGRESS Stack DELETE started2022-12-07 03:56:56Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_IN_PROGRESS state changed2022-12-07 04:56:56Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_FAILED DELETE aborted (Task delete from TemplateResource "kma_server_resource_group" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] Stack "lmfv_c426_kma_0" [d86ced7e-ba0a-4c08-854f-6cd772aa831a] Timed out)2022-12-07 04:56:56Z [lmfv_c426_kma_0]: DELETE_FAILED Timed out2022-12-07 06:22:34Z [lmfv_c426_kma_0]: DELETE_IN_PROGRESS Stack DELETE started2022-12-07 06:22:34Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_IN_PROGRESS state changed$ cat /var/log/heat/heat-api-daemon.log:2022-12-07 04:56:56,335.335 24650 INFO heat.engine.scheduler [req-0549a184-ce07-455f-b2b8-b2cd5b35e679 - admin - default default] Task delete from TemplateResource "kma_server_resource_group" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] Stack "lmfv_c426_kma_0" [d86ced7e-ba0a-4c08-854f-6cd772aa831a] timed out2022-12-07 04:56:56,350.350 28263 INFO heat.common.wsgi [req-d7af259f-6281-4c52-bc02-017deadd610d - c413 - default default] Processing request: GET /heat//v1/1a37f3c84b7f4358846009ddd2e5ee4c/stacks/e911_c413_wrkr_4/4c23d329-3b7d-48bd-8805-fa9b66630f15/events2022-12-07 04:56:56,377.377 24650 INFO heat.engine.stack [req-0549a184-ce07-455f-b2b8-b2cd5b35e679 - admin - default default] Stack DELETE FAILED (lmfv_c426_kma_0): Timed out2022-12-07 04:56:56,404.404 28263 INFO eventlet.wsgi.server [req-d7af259f-6281-4c52-bc02-017deadd610d - c413 - default default] 199.165.246.9,127.0.0.1 - - [07/Dec/2022 04:56:56] "GET //v1/1a37f3c84b7f4358846009ddd2e5ee4c/stacks/e911_c413_wrkr_4/4c23d329-3b7d-48bd-8805-fa9b66630f15/events?marker=9af8e628-1add-4512-a84b-e625c0e61c41&sort_dir=asc HTTP/1.1" 200 188 0.071507Resolution
- Try restarting the openstack-heat-engine service on the management plane.
systemctl restart openstack-heat-engine.servicesystemctl status openstack-heat-engine.service -lWas this page helpful?