How to Monitor Vault Token and Expiry Dates of Various certs?

Problem

How to check the status of vault token pro-actively

Environment

• Platform9 Edge Cloud - v5.3 and Higher

Answer

Scenario-1: How to check the vault token status specific to a cluster?

Run the below commands from the DU Vm as the root user:

1. Fetch the vault token:

Command

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

1. Export the vault address:

Command

export VAULT_ADDR=http://127.0.0.1:8200

1. Export the cluster UUID:

Command

mysql  qbert -e “select name,id,uuid,status,lastOk,lastOp,taskStatus,kubeRoleVersion from clusters;”

CLUSTER_UUID=<cluster-UUID>

1. Export the cluster vault token:

Command

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

1. Execute the vault token lookup

Command

/usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN

Example output:

Commands

# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

# export VAULT_ADDR=http://127.0.0.1:8200

# mysql qbert -e "select name,uuid from clusters;"
+--------------+--------------------------------------+
| name         | uuid                                 |
+--------------+--------------------------------------+
| test-cluster | [CLUSTER UUID]                       |
+--------------+--------------------------------------+
  
# CLUSTER_UUID=<CLUSTER UUID>

# CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")


# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Key                 Value
---                 -----
accessor            [ACCESSOR-ID]
creation_time       [CREATION TIMESTAMP]
creation_ttl        26280h
display_name        token
entity_id           n/a
expire_time         [EXPIRY TIMESTAMP]
explicit_max_ttl    0s
id                  [ID]
issue_time          [ISSUE TIMESTAMP]
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [POLICIES]
renewable           true
ttl                 26215h49m50s
type                service

Scenario-2: Monitoring Sunpike CA and Certs

Run the below commands from the DU Vm as the root user:

Commands

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

export VAULT_ADDR=http://127.0.0.1:8200

/usr/local/bin/vault read pki/cert/ca -format=json | jq -r '.data.certificate'| openssl x509 -noout -dates

usr/local/bin/vault read pki/roles/sunpike | grep ttl

If the Sunpike CA is expiring, please follow the manual steps mentioned in the following article to regenerate it: https://platform9.com/kb/PEC/refresh-sunpike-ca

To check the sunpike-kube-apiserver certificate expiry:

Bash

docker exec -it <sunpike-kube-apiserver-container-ID> sh -c "openssl s_client -connect localhost -port 6443 -showcerts 2>/dev/null </dev/null | openssl x509 -noout -dates"

Scenario-3: Monitoring Cluster CA and Certs

Run the below commands from the DU Vm as the root user:

Commands

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

export VAULT_ADDR=http://127.0.0.1:8200

export PATH=$PATH:/usr/local/bin

secrets_list=$(vault secrets list | awk '$2 == "pki" {print $1}' | grep "pmk-ca")
echo "$secrets_list" | while read secret; do echo -n "Certificate for Cluster: "; echo -n "$secret" | awk -F 'pmk-ca-' '{print $2}'; vault read "${secret}cert/ca" -format=json | jq -r '.data.certificate'| openssl x509 -noout -dates; done

usr/local/bin/vault read pki/roles/sunpike | grep ttl

This outputs the date on which the CA is created and the date on which it expires, per cluster. If the certs are expiring, please follow the manual steps mentioned in the following article to regenerate it: https://platform9.com/kb/PEC/refresh-cluster-ca

To verify the cluster certificate details from the node:

Bash

openssl x509 -text -noout -in /etc/pf9/kube.d/certs/etcd/client/ca.crt | grep -A2 -i validity
Or
 openssl x509 -text -noout -in /tmp/authbs-certs.[LATEST-FILE-ID]/etcd/client/ca.crt | grep -A2 -i validity

Scanario-4 : Vault token renewal Steps:

Follow the steps mentioned in the KB Renew expired vault token from the DU VM.