How To Renew Vault Token LTS1 Setup

Problem

• Unable to run the kubectl commands as the vault token was expired on the master nodes.

• Nodelet phases getting stuck on Gen-Certs Phase

Errors in kubelet.log

File \"<string>\", line 1, in <module>"}
  KeyError: 'data'"}  
  Certificate is not signed by CA"}
  Error loading file /tmp/authbs-certs.abc/apiserver/
  error sending status update to sunpike: rpc error: code = Unknown desc = apiserver storage error: an error on the server

   "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate has expired or is not yet valid\""}

While checking the directory/tmp/authbs-certs.abc/apiserver/ mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.

Affected nodes

[root@master0 apiserver]# cat /tmp/authbs-certs.abc/apiserver/request.json
{"errors":["permission denied"]}

Environment

• Platform9 Edge Cloud v-5.3.

Solution

To recover from this issue, it is required to regenerate the vault token.

Validation

Steps to validate the token expiry:

1. SSH into the DU VM as root user.

2. Export the required details.

Command

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

export VAULT_ADDR=http://127.0.0.1:8200

CLUSTER_UUID=<CLUSTER_UUID>

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

1. Run the below command to know token expiry details:

Command

/usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN

Example:

Command

SAMPLE:

#  /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Key                 Value
---                 -----
accessor            [ACCESSOR-ID]
creation_time       [CREATION TIMESTAMP]
creation_ttl        26280h
display_name        token
entity_id           n/a
expire_time         [EXPIRY TIMESTAMP]
explicit_max_ttl    0s
id                  [ID]
issue_time          [ISSUE TIMESTAMP]
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [POLICIES]
renewable           true
ttl                 26215h49m50s
type                service

Procedure

The steps to regenerate the vault token are:

Perform below 1 to 7 steps as a root user in the DU VM:

1. Retrieve the affected cluster's UUID

DU VM Terminal

mysql  qbert -e "select name,id,uuid,status,lastOk,lastOp,taskStatus,kubeRoleVersion from clusters;"

2. Check the current vault token for the affected cluster

DU VM Terminal

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='$CLUSTER_UUID';"

3. Set the value for the following variables to regenerate the token:

DU VM Terminal

CLUSTER_UUID=<Cluster-UUID>

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

4. Verify the values of the variables $OLD_VAULT_TOKENand $ROOT_VAULT_TOKEN

DU VM Terrminal

echo $OLD_VAULT_TOKEN

echo $ROOT_VAULT_TOKEN

5. Generate new vault token using the below commands:

DU VM Terminal

NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)

NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')

echo "New Vault-Token generated - $NEW_TOKEN"

6. Update the new token in the qbert database:

DU VM Terminal

mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"

7. Verify the newly generated token using:

DU VM

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<uuid of the affected cluster>';"

1. Restart the nodeletd phases on each master nodes/affected nodes one at a time.

Phases Restart

# systemctl stop pf9-hostagent pf9-nodeletd 
# /opt/pf9/nodelet/nodeletd phases restart
# systemctl start pf9-hostagent pf9-nodeletd

9. Check the new token in the file /etc/pf9/kube.env

Master node

# grep -i vault

Additional Information

Post pf9-vault token renewal, if the new token generated in the DU VM is not propagated to the nodes in file /etc/pf9/kube.env perform the below workaround:

1. Manually copy the vault token from the DU VM:

DU VM

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<Cluster-ID>';"

1. Replace the token in the node in file /etc/pf9/kube.env

Node

export VAULT_TOKEN=<REPLACE_WITH_NEW_TOKEN>

1. Restart the nodeletd phases in the nodes. This should pick up the new pf9-vault token.