How to Monitor Vault Token and Expiry Dates of Various certs?
Problem
How to check the status of vault token pro-actively
Environment
- Platform9 Edge Cloud - v5.3 and Higher
Answer
Scenario-1: How to check the vault token status specific to a cluster?
Run the below commands from the DU Vm as the root user:
- Fetch the vault token:
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")- Export the vault address:
# export VAULT_ADDR=http://127.0.0.1:8200- Export the cluster UUID:
# CLUSTER_UUID=<cluster-UUID>- Export the cluster vault token:
# CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")- Execute the vault token lookup
# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENExample output:
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# export VAULT_ADDR=http://127.0.0.1:8200# mysql qbert -e "select name,uuid from clusters;"+--------------+--------------------------------------+| name | uuid |+--------------+--------------------------------------+| test-cluster | [CLUSTER UUID] |+--------------+--------------------------------------+ # CLUSTER_UUID=<CLUSTER UUID># CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENKey Value--- -----accessor [ACCESSOR-ID]creation_time [CREATION TIMESTAMP]creation_ttl 26280hdisplay_name tokenentity_id n/aexpire_time [EXPIRY TIMESTAMP]explicit_max_ttl 0sid [ID]issue_time [ISSUE TIMESTAMP]meta <nil>num_uses 0orphan falsepath auth/token/createpolicies [POLICIES]renewable truettl 26215h49m50stype serviceScenario-2: Monitoring Sunpike CA and Certs
Run the below commands from the DU Vm as the root user:
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# export VAULT_ADDR=http://127.0.0.1:8200# /usr/local/bin/vault read pki/cert/ca -format=json | jq -r '.data.certificate'| openssl x509 -noout -dates# /usr/local/bin/vault read pki/roles/sunpike | grep ttlIf the Sunpike CA is expiring, please follow the manual steps mentioned in the following article to regenerate it: https://platform9.com/kb/PEC/refresh-sunpike-ca
To check the sunpike-kube-apiserver certificate expiry:
# docker exec -it <sunpike-kube-apiserver-container-ID> sh -c "openssl s_client -connect localhost -port 6443 -showcerts 2>/dev/null </dev/null | openssl x509 -noout -dates"Scenario-3: Monitoring Cluster CA and Certs
Run the below commands from the DU Vm as the root user:
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# export VAULT_ADDR=http://127.0.0.1:8200# export PATH=$PATH:/usr/local/bin# secrets_list=$(vault secrets list | awk '$2 == "pki" {print $1}' | grep "pmk-ca")echo "$secrets_list" | while read secret; do echo -n "Certificate for Cluster: "; echo -n "$secret" | awk -F 'pmk-ca-' '{print $2}'; vault read "${secret}cert/ca" -format=json | jq -r '.data.certificate'| openssl x509 -noout -dates; done# /usr/local/bin/vault read pki/roles/sunpike | grep ttlThis outputs the date on which the CA is created and the date on which it expires, per cluster. If the certs are expiring, please follow the manual steps mentioned in the following article to regenerate it: https://platform9.com/kb/PEC/refresh-cluster-ca
To verify the cluster certificate details from the node:
# openssl x509 -text -noout -in /etc/pf9/kube.d/certs/etcd/client/ca.crt | grep -A2 -i validityOr# openssl x509 -text -noout -in /tmp/authbs-certs.[LATEST-FILE-ID]/etcd/client/ca.crt | grep -A2 -i validityScanario-4 : Vault token renewal Steps:
Follow the steps mentioned in the KB Renew expired vault token from the DU VM.