How to Renew vault Token?
Problem
The vault token is expired.
Environment
- Platform9 Managed Kubernetes v5.9.3
Validation
Steps to validate the token expiry:
- Exec into pf9-vault pod in Management Plane namespace.
$ kubectl exec -it -n <MANAGEMENT_PLANE_NAMESPACE> --kubeconfig <KUBECONFIG> $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash- Export the required details.
x
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# export VAULT_ADDR=http://127.0.0.1:8200# CLUSTER_UUID=<CLUSTER_UUID># OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")# ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")- Run the below command to know token expiry details:
# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENExample:
SAMPLE:# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENKey Value--- -----accessor [ACCESSOR-ID]creation_time [CREATION TIMESTAMP]creation_ttl 26280hdisplay_name tokenentity_id n/aexpire_time <>explicit_max_ttl 0sid [ID]issue_time [ISSUE TIMESTAMP]meta <nil>num_uses 0orphan falsepath auth/token/createpolicies [POLICIES]renewable truettl 26215h49m50stype serviceProcedure
- Exec Into pf9-vault pod in customer namespace
export KUBECONFIG=kplane-enterprise-2.yamlexport NS=<customer_shortname>kubectl exec -it -n $NS $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash- Export required details (in pf9-vault pod)
CLUSTER_UUID=<cluster_UUID>OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")echo $OLD_VAULT_TOKENecho $ROOT_VAULT_TOKEN- Generate New Token (in
pf9-vaultpod)
NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')echo "New Vault-Token generated - $NEW_TOKEN"- Update the new token in qbert Database (in
pf9-vaultpod)
mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"- Open a new terminal window and Verify if the new token is updated at cluster and node level (Execute on a new terminal window)
export KUBECONFIG=kplane-enterprise-2.yamlexport NS=<customer_shortname>kubectl -n $NS exec -it deploy/mysqld-exporter -- mysql qbert -e "select name,uuid,vaultToken from clusters where uuid=<cluster_uuid>"kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts <HOST_UUID> -o yaml | grep -i vault- If the token in Sunpike does not match the token in Qbert, run the following command to patch the Sunpike host object.(from the same terminal window as Step 5 )
export VAULT_TOKEN=<TOKEN_FROM_QBERT_DB>export CLUSTER_UUID=<CLUSTER_UUID>for i in $(kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts --no-headers | grep $CLUSTER_UUID | awk '{print $1}'); do kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl patch host $i -p '{"spec":{"pf9":{"vaultToken":"'${VAULT_TOKEN}'"}}}'; done- restart the full stack restart on nodes that got stuck at the Cert Generation phase (if any).
- revoke the old Token (in pf9-vault pod) - once all nodes are working fine.
curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"token": "'$OLD_VAULT_TOKEN'"}' http://localhost:8200/v1/auth/token/revokeAdditional Information
To check if the vault token is expired follow these steps:
- From the problematic master node, perform this.
#grep -i vault /etc/pf9/kube.envYou will get an output like this: export VAULT_TOKEN="<token_name>" <<--------- Come to the DU VM and perform this.
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# export VAULT_ADDR=http://127.0.0.1:8200# vault token lookup <token_name_from_above>Was this page helpful?