How To Renew Vault Token LTS1 Setup

Problem

Unable to run the kubectl commands as the vault token was expired on the master nodes.
Nodelet phases getting stuck on Gen-Certs Phase

Errors in kubelet.log
    
​x
    
File \"<string>\", line 1, in <module>"}  KeyError: 'data'"}    Certificate is not signed by CA"}  Error loading file /tmp/authbs-certs.abc/apiserver/  error sending status update to sunpike: rpc error: code = Unknown desc = apiserver storage error: an error on the server​   "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate has expired or is not yet valid\""}
Copy

While checking the directory/tmp/authbs-certs.abc/apiserver/ mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.

Affected nodes
    
 
[root@master0 apiserver]# cat /tmp/authbs-certs.abc/apiserver/request.json{"errors":["permission denied"]}
Copy

Environment

Platform9 Edge Cloud v-5.3.

Solution

To recover from this issue, it is required to regenerate the vault token.

Validation

Steps to validate the token expiry:

SSH into the DU VM as root user.
Export the required details.

Command
    
 
export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")​export VAULT_ADDR=http://127.0.0.1:8200​CLUSTER_UUID=<CLUSTER_UUID>​OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")​ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")​CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")
Copy

Run the below command to know token expiry details:

Command
    
 
/usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Copy

Example:

Command
    
 
SAMPLE:​#  /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENKey                 Value---                 -----accessor            [ACCESSOR-ID]creation_time       [CREATION TIMESTAMP]creation_ttl        26280hdisplay_name        tokenentity_id           n/aexpire_time         [EXPIRY TIMESTAMP]explicit_max_ttl    0sid                  [ID]issue_time          [ISSUE TIMESTAMP]meta                <nil>num_uses            0orphan              falsepath                auth/token/createpolicies            [POLICIES]renewable           truettl                 26215h49m50stype                service
Copy

Procedure

The steps to regenerate the vault token are:

Perform below 1 to 7 steps as a root user in the DU VM:

1. Retrieve the affected cluster's UUID

DU VM Terminal
    
 
mysql  qbert -e "select name,id,uuid,status,lastOk,lastOp,taskStatus,kubeRoleVersion from clusters;"
Copy

2. Check the current vault token for the affected cluster

DU VM Terminal
    
 
mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='$CLUSTER_UUID';"
Copy

3. Set the value for the following variables to regenerate the token:

DU VM Terminal
    
 
CLUSTER_UUID=<Cluster-UUID>​OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")​ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")
Copy

4. Verify the values of the variables $OLD_VAULT_TOKENand $ROOT_VAULT_TOKEN

DU VM Terrminal
    
 
echo $OLD_VAULT_TOKEN​echo $ROOT_VAULT_TOKEN
Copy

5. Generate new vault token using the below commands:

DU VM Terminal
    
NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)​NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')​echo "New Vault-Token generated - $NEW_TOKEN"
Copy

6. Update the new token in the qbert database:

DU VM Terminal
    
 
mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"
Copy

7. Verify the newly generated token using:

DU VM
    
 
mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<uuid of the affected cluster>';"
Copy

Restart the nodeletd phases on each master nodes/affected nodes one at a time.

Phases Restart
    
 
# systemctl stop pf9-hostagent pf9-nodeletd # /opt/pf9/nodelet/nodeletd phases restart# systemctl start pf9-hostagent pf9-nodeletd
Copy

9. Check the new token in the file /etc/pf9/kube.env

Master node
    
 
# grep -i vault
Copy

Additional Information

Post pf9-vault token renewal, if the new token generated in the DU VM is not propagated to the nodes in file /etc/pf9/kube.env perform the below workaround:

Manually copy the vault token from the DU VM:

DU VM
    
 
mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<Cluster-ID>';"
Copy

Replace the token in the node in file /etc/pf9/kube.env

Node
    
 
export VAULT_TOKEN=<REPLACE_WITH_NEW_TOKEN>
Copy

Restart the nodeletd phases in the nodes. This should pick up the new pf9-vault token.

Last updated on

Was this page helpful?