Node Attach via U/I Returns Error "StatusCodeError: 404 - Not Found"

Problem

Node attachment via U/I returns error "StatusCodeError: 404 - "Not Found".

Environment

• Platform9 Managed Kubernetes - All Versions

Cause

Errors like the one below are recorded in the qbert log.

None

[2021-04-29T18:37:33.574] [DEBUG] api - incoming request for /v3/07f73965c6bc485499d2c373a6af6744/clusters/e310f40f-e979-4046-8908-d95648655671/attach[2021-04-29T18:37:33.811] [INFO] api - Starting with process of syncing host and cluster kubeRoleVersion[2021-04-29T18:37:33.811] [INFO] api - Checking for the node: 2544ec03-485a-4801-a88d-349ffd6205ee[2021-04-29T18:37:33.832] [INFO] verify - checking node swap space, firewalld and selinux status on host 2544ec03-485a-4801-a88d-349ffd6205ee[2021-04-29T18:37:33.832] [DEBUG] tackboard - queueing entry with uuid 431f4d29-351a-491e-93e8-67b181f8506c[2021-04-29T18:37:40.067] [DEBUG] api - incoming request for /v1[2021-04-29T18:37:40.067] [INFO] keystoneAuth - No authentication token in request[2021-04-29T18:37:40.068] [INFO] api - GET /v1 401 - 0ms - Unauthorized[2021-04-29T18:37:41.490] [DEBUG] api - incoming request for /v1/clusters[2021-04-29T18:37:41.499] [DEBUG] db - _getCloudProvider called with uuid: 5543a729-d6f2-44ef-88b6-951e90f800ff[2021-04-29T18:37:41.528] [INFO] api - GET /v1/clusters 200 - 38ms[2021-04-29T18:37:41.838] [DEBUG] tackboard - removing entry with uuid 431f4d29-351a-491e-93e8-67b181f8506c[2021-04-29T18:37:41.841] [ERROR] api - StatusCodeError: 404 - "Not Found"    at Object.wrapPromiseError (/opt/pf9/qbert/node_modules/qbert-utils/index.js:53:8)    at wrapPromiseError.throw ([anonymous])    at Object.wrappedPromiseFunction [as attachNode] (/opt/pf9/qbert/node_modules/qbert-utils/index.js:64:5)    at wrappedPromiseFunction.throw ([anonymous])    at genAttachNodes (/opt/pf9/qbert/api/handlers/v3/index.js:115:5)    at genAttachNodes.throw ([anonymous])    at genWrap (/opt/pf9/qbert/api/utils.js:21:9)    at genWrap.throw ([anonymous])    at onRejected (/opt/pf9/qbert/node_modules/co/index.js:81:24)    at tryCatcher (/opt/pf9/qbert/node_modules/bluebird/js/release/util.js:16:23)    at Promise._settlePromiseFromHandler (/opt/pf9/qbert/node_modules/bluebird/js/release/promise.js:547:31)    at Promise._settlePromise (/opt/pf9/qbert/node_modules/bluebird/js/release/promise.js:604:18)    at Promise._settlePromise0 (/opt/pf9/qbert/node_modules/bluebird/js/release/promise.js:649:10)    at Promise._settlePromises (/opt/pf9/qbert/node_modules/bluebird/js/release/promise.js:725:18)    at _drainQueueStep (/opt/pf9/qbert/node_modules/bluebird/js/release/async.js:93:12)    at _drainQueue (/opt/pf9/qbert/node_modules/bluebird/js/release/async.js:86:9)    at Async._drainQueues (/opt/pf9/qbert/node_modules/bluebird/js/release/async.js:102:5)    at Immediate.Async.drainQueues (/opt/pf9/qbert/node_modules/bluebird/js/release/async.js:15:14)    at runCallback (timers.js:810:20)    at tryOnImmediate (timers.js:768:5)    at processImmediate [as _immediateCallback] (timers.js:745:5)[2021-04-29T18:37:41.843] [INFO] api - POST /v3/07f73965c6bc485499d2c373a6af6744/clusters/e310f40f-e979-4046-8908-d95648655671/attach 400 - 8269ms - Bad Request

This is observed as the pf9-sidekick service on the node is in a failed state.

None

$ systemctl status pf9-sidekick● pf9-sidekick.service - Platform9 Sidekick ServiceLoaded: loaded (/lib/systemd/system/pf9-sidekick.service; enabled; vendor preset: enabled)Active: inactive (dead) since Fri 2021-05-07 04:22:29 UTC; 2min 7s agoCondition: start condition failed at Fri 2021-05-07 04:22:29 UTC; 2min 7s ago

Resolution

1. Restart the pf9-sidekick service on the node.

None

$ systemctl restart pf9-sidekick

1. Retry attaching the node.