Reenabling the monitoring fails in v5.2

Problem

With the latest release, we can disable the monitoring, but now we are not able to enable it back.

None

Request URL: https://airctl-1.pf9.localnet/appbert/v1/clusters/f15e9366-3808-4897-8d01-450f98a183ee/8Request Method: PUTStatus Code: 403 ForbiddenRemote Address: 10.128.233.165:443​Ignoring install task, delete task in progress for cluster: f15e9366-3808-4897-8d01-450f98a183ee

appbert.log:

It seems that the delete task is still in progress according to appbert.

None

{"level":"info","msg":"Ignoring install task 0: Install, pf9-mon, f15e9366-3808-4897-8d01-450f98a183ee in DB, delete in progress","time":"2021-06-01T07:47:41Z"}{"level":"error","msg":"Ignoring install task, delete task in progresswhile adding install task","time":"2021-06-01T07:47:41Z"}{"level":"error","msg":"Error deleteing: CustomResourceDefinition, output: Error from server (InternalError): an error on the server (\"\u003chtml\u003e\\r\<br>\u003chead\u003e\u003ctitle\u003e502 Bad Gateway\u003c/title\u003e\u003c/head\u003e\\r\<br>\u003cbody\u003e\\r\<br>\u003ccenter\u003e\u003ch1\u003e502 Bad Gateway\u003c/h1\u003e\u003c/center\u003e\\r\<br>\u003chr\u003e\u003ccenter\u003enginx/1.18.0\u003c/center\u003e\\r\<br>\u003c/body\u003e\\r\<br>\u003c/html\u003e\") has prevented the request from succeeding<br>customresourcedefinition.apiextensions.k8s.io \"clusterserviceversions.operators.coreos.com\" deleted<br>","time":"2021-06-01T07:47:56Z"}{"level":"error","msg":"Failed to run kubectl delete, retry: 1 out of 15, exit status 1","time":"2021-06-01T07:47:56Z"}{"level":"info","msg":"deleteing type: CustomResourceDefinition","time":"2021-06-01T07:48:06Z"}{"level":"error","msg":"Error deleteing: CustomResourceDefinition, output: Error from server (NotFound): error when deleting \"STDIN\": customresourcedefinitions.apiextensions.k8s.io \"clusterserviceversions.operators.coreos.com\" not found<br>","time":"2021-06-01T07:48:07Z"}{"level":"info","msg":"Already deleted, ignoring error","time":"2021-06-01T07:48:07Z"}{"level":"info","msg":"deleteing type: CustomResourceDefinition","time":"2021-06-01T07:48:08Z"}

None

mysql> select * from tasks_v2 where type='Delete';+----+--------+---------+--------------------------------------+----------------------------------+------+----------+| id | type   | pkgname | cluster                              | project                          | data | status   |+----+--------+---------+--------------------------------------+----------------------------------+------+----------+|  2 | Delete | pf9-mon | f15e9366-3808-4897-8d01-450f98a183ee | 9910aa195c5d4bdb9303981917b890b3 | NULL | Complete |+----+--------+---------+--------------------------------------+----------------------------------+------+----------+1 row in set (0.00 sec)

Checking the appbert logs again, we can see that the delete task has now been completed.

None

{"level":"info","msg":"Already deleted, ignoring error","time":"2021-06-01T07:50:20Z"}{"level":"info","msg":"Received a call to update tags on cluster: f15e9366-3808-4897-8d01-450f98a183ee","time":"2021-06-01T07:50:21Z"}{"level":"info","msg":"Finished delete task for package pf9-mon on cluster f15e9366-3808-4897-8d01-450f98a183ee","time":"2021-06-01T07:50:21Z"}

Now if we enable monitoring from the UI it gets enabled without any issues. So immediately after disabling the monitoring, we cannot enable monitoring again as the Delete task was still InProgress. Once the delete task was in a Complete state we can enable monitoring.

Environment

• Platform9 Managed Kubernetes - v4.0 and Higher