How To Renew Vault Token LTS1 Setup
Problem
- Unable to run the kubectl commands as the vault token was expired on the master nodes.
- Nodelet phases getting stuck on Gen-Certs Phase
File \"<string>\", line 1, in <module>"} KeyError: 'data'"} Certificate is not signed by CA"} Error loading file /tmp/authbs-certs.abc/apiserver/ error sending status update to sunpike: rpc error: code = Unknown desc = apiserver storage error: an error on the server "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate has expired or is not yet valid\""}While checking the directory/tmp/authbs-certs.abc/apiserver/ mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.
[root@master0 apiserver]# cat /tmp/authbs-certs.abc/apiserver/request.json{"errors":["permission denied"]}Environment
- Platform9 Edge Cloud v-5.3.
Solution
To recover from this issue, it is required to regenerate the vault token.
Validation
Steps to validate the token expiry:
- SSH into the DU VM as root user.
- Export the required details.
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# export VAULT_ADDR=http://127.0.0.1:8200# CLUSTER_UUID=<CLUSTER_UUID># OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")# ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")- Run the below command to know token expiry details:
# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENExample:
SAMPLE:# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENKey Value--- -----accessor [ACCESSOR-ID]creation_time [CREATION TIMESTAMP]creation_ttl 26280hdisplay_name tokenentity_id n/aexpire_time [EXPIRY TIMESTAMP]explicit_max_ttl 0sid [ID]issue_time [ISSUE TIMESTAMP]meta <nil>num_uses 0orphan falsepath auth/token/createpolicies [POLICIES]renewable truettl 26215h49m50stype serviceProcedure
The steps to regenerate the vault token are:
Perform below 1 to 7 steps as a root user in the DU VM:
1. Retrieve the affected cluster's UUID
mysql qbert -e "select name,id,uuid,status,lastOk,lastOp,taskStatus,kubeRoleVersion from clusters;"2. Check the current vault token for the affected cluster
mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<Cluster-ID>';"3. Set the value for the following variables to regenerate the token:
CLUSTER_UUID=<Cluster-UUID>OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")4. Verify the values of the variables $OLD_VAULT_TOKENand $ROOT_VAULT_TOKEN
echo $OLD_VAULT_TOKENecho $ROOT_VAULT_TOKEN5. Generate new vault token using the below commands:
NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')echo "New Vault-Token generated - $NEW_TOKEN"6. Update the new token in the qbert database:
mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"7. Verify the newly generated token using:
mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<uuid of the affected cluster>';"- Restart the nodeletd phases on each master nodes/affected nodes one at a time.
# systemctl stop pf9-hostagent pf9-nodeletd # /opt/pf9/nodelet/nodeletd phases restart# systemctl start pf9-hostagent pf9-nodeletd9. Check the new token in the file /etc/pf9/kube.env
# grep -i vault /etc/pf9/kube.env