# Stack Stuck in "Delete-In-Progress" State ## Problem * Deleting stack gets stuck with`DELETE_IN_PROGRESS` state {% tabs %} {% tab title="Bash" %} ```bash openstack stack list +--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+ | d86ced7e-ba0a-4c08-854f-6cd772aa831a | lmfv_c426_kma_0 | b8ba4cd05ecf4e47a3002622574cd815 | DELETE_IN_PROGRESS | 2022-12-06T06:27:02Z | 2022-12-06T14:18:45Z | | fe48b5d2-28ad-4912-9a5e-a16751032588 | lmfv_c426_base | b8ba4cd05ecf4e47a3002622574cd815 | DELETE_FAILED | 2022-12-06T06:00:57Z | 2022-12-06T14:18:45Z | +--------------------------------------+-----------------+----------------------------------+--------------------+----------------------+----------------------+ ``` {% endtab %} {% endtabs %} * Below lines can be seen in the heat logs {% tabs %} {% tab title="Bash" %} ```bash cat /var/log/heat/heat-api-daemon.log: ... 2022-12-07 06:22:49,602.602 24651 ERROR heat.engine.worker [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Failed to stop all workers of stack lmfv_c426_km a_0-kma_server_resource_group-7kilsgnv7trs, stack cancel not complete 2022-12-07 06:22:49,743.743 24651 INFO heat.engine.stack [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Triggering resource 349871 for cleanup 2022-12-07 06:22:49,822.822 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] deleting CinderVolumeAttachment "kma_volume_att achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] 2022-12-07 06:22:49,842.842 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_att achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist 2022-12-07 06:22:49,945.945 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_att achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist 2022-12-07 06:22:50,542.542 24648 INFO heat.engine.resource [req-5bba22e9-efb9-4483-a0da-e749af7d2b0b - admin - default default] Resource CinderVolumeAttachment "kma_volume_att achment_0" [7960cd50-42e7-4784-95af-da424a7fa665] Stack "lmfv_c426_kma_0-kma_server_resource_group-7kilsgnv7trs" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] is locked or does not exist ``` {% endtab %} {% endtabs %} ## Environment * Platform9 Managed OpenStack ## Cause * We can see there is a timeout while deleting the stack which leaves it in **DELETE *****IN***** PROGRESS** state\*\*.\*\* {% tabs %} {% tab title="Bash" %} ```bash 2022-12-07 03:56:56Z [lmfv_c426_kma_0]: DELETE_IN_PROGRESS Stack DELETE started 2022-12-07 03:56:56Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_IN_PROGRESS state changed 2022-12-07 04:56:56Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_FAILED DELETE aborted (Task delete from TemplateResource "kma_server_resource_group" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] Stack "lmfv_c426_kma_0" [d86ced7e-ba0a-4c08-854f-6cd772aa831a] Timed out) 2022-12-07 04:56:56Z [lmfv_c426_kma_0]: DELETE_FAILED Timed out 2022-12-07 06:22:34Z [lmfv_c426_kma_0]: DELETE_IN_PROGRESS Stack DELETE started 2022-12-07 06:22:34Z [lmfv_c426_kma_0.kma_server_resource_group]: DELETE_IN_PROGRESS state changed $ cat /var/log/heat/heat-api-daemon.log: 2022-12-07 04:56:56,335.335 24650 INFO heat.engine.scheduler [req-0549a184-ce07-455f-b2b8-b2cd5b35e679 - admin - default default] Task delete from TemplateResource "kma_server_resource_group" [b6b5bce1-9441-447a-bee0-a6a9d7ea0196] Stack "lmfv_c426_kma_0" [d86ced7e-ba0a-4c08-854f-6cd772aa831a] timed out 2022-12-07 04:56:56,350.350 28263 INFO heat.common.wsgi [req-d7af259f-6281-4c52-bc02-017deadd610d - c413 - default default] Processing request: GET /heat//v1/1a37f3c84b7f4358846009ddd2e5ee4c/stacks/e911_c413_wrkr_4/4c23d329-3b7d-48bd-8805-fa9b66630f15/events 2022-12-07 04:56:56,377.377 24650 INFO heat.engine.stack [req-0549a184-ce07-455f-b2b8-b2cd5b35e679 - admin - default default] Stack DELETE FAILED (lmfv_c426_kma_0): Timed out 2022-12-07 04:56:56,404.404 28263 INFO eventlet.wsgi.server [req-d7af259f-6281-4c52-bc02-017deadd610d - c413 - default default] 199.165.246.9,127.0.0.1 - - [07/Dec/2022 04:56:56] "GET //v1/1a37f3c84b7f4358846009ddd2e5ee4c/stacks/e911_c413_wrkr_4/4c23d329-3b7d-48bd-8805-fa9b66630f15/events?marker=9af8e628-1add-4512-a84b-e625c0e61c41&sort_dir=asc HTTP/1.1" 200 188 0.071507 ``` {% endtab %} {% endtabs %} ## Resolution * Try restarting the **openstack-heat-engine** service on the management plane. {% tabs %} {% tab title="Bash" %} ```bash systemctl restart openstack-heat-engine.service systemctl status openstack-heat-engine.service -l ``` {% endtab %} {% endtabs %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://platform9.com/kb/pmo/solution/stack-stuck-in--delete-in-progress--state.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.