Kubectl command failing with certificate error

Problem

• kubectl commands are failing and throwing the below certificate error in kubelet logs:

Errors in kubelet.log

x509: certificate has expired or is not yet valid:

• The nodelet phase failing on the first cert-gen phase with the below error:

nodelet logs

cat /tmp/authbs-certs.xxxx/kubelet/apiserver/request.json
extract_vault_json certificate
/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''certificate'\''])'
Traceback (most recent call last):
File "<string>", line 1, in <module>
KeyError: 'data'
cat /tmp/authbs-certs.fL0T/kubelet/apiserver/request.json
extract_vault_json issuing_ca
/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''issuing_ca'\''])'
openssl verify -CAfile /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt /tmp/authbs-certs.xxxx/kubelet/apiserver/request.crt
Error loading file /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt
+ echo 'Certificate is not signed by CA'
Certificate is not signed by CA

• While checking the directory/tmp/authbs-certs.xxxx/kubelet/apiserver mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.

Impacted node

$ cat request.json
{"errors":["permission denied"]}

Environment

• Platform9 Edge Cloud - 5.3.0 or Higher

Cause

• The vault token is expired.

Procedure

• Retrieve the token using below command from the affected node:

Affected node

$ grep -i vault /etc/pf9/kube.env

• To check the validity of the vault token, run the below command:

Affected node

$ export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

$export VAULT_ADDR=http://127.0.0.1:8200

$vault token lookup <token id>

• If the above command return like below instead of the token information, then its confirmed that the vault token is expired and need to renew the vault token:

Affected node

Error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup
Code: 403. Errors:

* bad token

Resolution

• To resolve the issue, renew the vault token as per the KB