How To Renew Vault Token LTS1 Setup

Problem

Unable to run the kubectl commands as the vault token was expired on the master nodes.
Nodelet phases getting stuck on Gen-Certs Phase

File \"<string>\", line 1, in <module>"}
  KeyError: 'data'"}  
  Certificate is not signed by CA"}
  Error loading file /tmp/authbs-certs.abc/apiserver/
  error sending status update to sunpike: rpc error: code = Unknown desc = apiserver storage error: an error on the server

   "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate has expired or is not yet valid\""}

While checking the directory/tmp/authbs-certs.abc/apiserver/ mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.

[root@master0 apiserver]# cat /tmp/authbs-certs.abc/apiserver/request.json
{"errors":["permission denied"]}

Environment

Platform9 Edge Cloud v-5.3.

Solution

To recover from this issue, it is required to regenerate the vault token.

Validation

Steps to validate the token expiry:

SSH into the DU VM as root user.
Export the required details.

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

export VAULT_ADDR=http://127.0.0.1:8200

CLUSTER_UUID=<CLUSTER_UUID>

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

Run the below command to know token expiry details:

/usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN

Example:

SAMPLE:

#  /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Key                 Value
---                 -----
accessor            [ACCESSOR-ID]
creation_time       [CREATION TIMESTAMP]
creation_ttl        26280h
display_name        token
entity_id           n/a
expire_time         [EXPIRY TIMESTAMP]
explicit_max_ttl    0s
id                  [ID]
issue_time          [ISSUE TIMESTAMP]
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [POLICIES]
renewable           true
ttl                 26215h49m50s
type                service

Procedure

The steps to regenerate the vault token are:

Perform below 1 to 7 steps as a root user in the DU VM:

1. Retrieve the affected cluster's UUID

mysql  qbert -e "select name,id,uuid,status,lastOk,lastOp,taskStatus,kubeRoleVersion from clusters;"

2. Check the current vault token for the affected cluster

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='$CLUSTER_UUID';"

3. Set the value for the following variables to regenerate the token:

CLUSTER_UUID=<Cluster-UUID>

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

4. Verify the values of the variables $OLD_VAULT_TOKENand $ROOT_VAULT_TOKEN

echo $OLD_VAULT_TOKEN

echo $ROOT_VAULT_TOKEN

5. Generate new vault token using the below commands:

NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)

NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')

echo "New Vault-Token generated - $NEW_TOKEN"

6. Update the new token in the qbert database:

mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"

7. Verify the newly generated token using:

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<uuid of the affected cluster>';"

Restart the nodeletd phases on each master nodes/affected nodes one at a time.

# systemctl stop pf9-hostagent pf9-nodeletd 
# /opt/pf9/nodelet/nodeletd phases restart
# systemctl start pf9-hostagent pf9-nodeletd

9. Check the new token in the file /etc/pf9/kube.env

# grep -i vault

Additional Information

Post pf9-vault token renewal, if the new token generated in the DU VM is not propagated to the nodes in file /etc/pf9/kube.env perform the below workaround:

Manually copy the vault token from the DU VM:

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<Cluster-ID>';"

Replace the token in the node in file /etc/pf9/kube.env

export VAULT_TOKEN=<REPLACE_WITH_NEW_TOKEN>

Restart the nodeletd phases in the nodes. This should pick up the new pf9-vault token.

PreviousIngress-Nginx Vulnerability (CVE-2025-1974)

NextWhat Is the TLS Version Used For UI In EDGE releases?

Last updated 8 months ago