How to Renew vault Token?

Problem

The vault token is expired.

Environment

• Platform9 Managed Kubernetes v5.9.3

Validation

Steps to validate the token expiry:

1. Exec into pf9-vault pod in Management Plane namespace.

Command

$ kubectl exec -it -n <MANAGEMENT_PLANE_NAMESPACE> --kubeconfig <KUBECONFIG> $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash

1. Export the required details.

Command

# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

# export VAULT_ADDR=http://127.0.0.1:8200

# CLUSTER_UUID=<CLUSTER_UUID>

# OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

# ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

# CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

1. Run the below command to know token expiry details:

Command

# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN

Example:

Command

SAMPLE:

#  /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Key                 Value
---                 -----
accessor            [ACCESSOR-ID]
creation_time       [CREATION TIMESTAMP]
creation_ttl        26280h
display_name        token
entity_id           n/a
expire_time         <>
explicit_max_ttl    0s
id                  [ID]
issue_time          [ISSUE TIMESTAMP]
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [POLICIES]
renewable           true
ttl                 26215h49m50s
type                service

Procedure

1. Exec Into pf9-vault pod in customer namespace

Management cluster

export KUBECONFIG=kplane-enterprise-2.yaml
export NS=<customer_shortname>
kubectl exec -it -n $NS $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash

1. Export required details (in pf9-vault pod)

Management Cluster

CLUSTER_UUID=<cluster_UUID>
OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")
ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")
echo $OLD_VAULT_TOKEN
echo $ROOT_VAULT_TOKEN

1. Generate New Token (in pf9-vault pod)

Management Cluster

NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)
NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')
echo "New Vault-Token generated - $NEW_TOKEN"

1. Update the new token in qbert Database (in pf9-vault pod)

Management Cluster

mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"

1. Open a new terminal window and Verify if the new token is updated at cluster and node level (Execute on a new terminal window)

Management Cluster

export KUBECONFIG=kplane-enterprise-2.yaml
export NS=<customer_shortname>
kubectl -n $NS exec -it deploy/mysqld-exporter -- mysql qbert -e "select name,uuid,vaultToken from clusters where uuid=<cluster_uuid>"
kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts <HOST_UUID> -o yaml | grep -i vault

1. If the token in Sunpike does not match the token in Qbert, run the following command to patch the Sunpike host object.(from the same terminal window as Step 5 )

Management cluster

export VAULT_TOKEN=<TOKEN_FROM_QBERT_DB>
export CLUSTER_UUID=<CLUSTER_UUID>
for i in $(kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts --no-headers | grep $CLUSTER_UUID | awk '{print $1}'); do kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl patch host $i -p '{"spec":{"pf9":{"vaultToken":"'${VAULT_TOKEN}'"}}}'; done

1. restart the full stack restart on nodes that got stuck at the Cert Generation phase (if any).

2. revoke the old Token (in pf9-vault pod) - once all nodes are working fine.

Management Cluster

curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"token": "'$OLD_VAULT_TOKEN'"}' http://localhost:8200/v1/auth/token/revoke

Additional Information

To check if the vault token is expired follow these steps:

1. From the problematic master node, perform this.

#grep -i vault /etc/pf9/kube.env

You will get an output like this:   
export VAULT_TOKEN="<token_name>" <<--------

1. Come to the DU VM and perform this.

DU VM

# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")
# export VAULT_ADDR=http://127.0.0.1:8200
# vault token lookup <token_name_from_above>