Kubectl command failing with certificate error
Problem
kubectlcommands are failing and throwing the below certificate error in kubelet logs:
x509: certificate has expired or is not yet valid:- The nodelet phase failing on the first
cert-genphase with the below error:
cat /tmp/authbs-certs.xxxx/kubelet/apiserver/request.jsonextract_vault_json certificate/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''certificate'\''])'Traceback (most recent call last):File "<string>", line 1, in <module>KeyError: 'data'cat /tmp/authbs-certs.fL0T/kubelet/apiserver/request.jsonextract_vault_json issuing_ca/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''issuing_ca'\''])'openssl verify -CAfile /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt /tmp/authbs-certs.xxxx/kubelet/apiserver/request.crtError loading file /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt+ echo 'Certificate is not signed by CA'Certificate is not signed by CA- While checking the directory/tmp/authbs-certs.xxxx/kubelet/apiserver mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.
$ cat request.json{"errors":["permission denied"]}Environment
- Platform9 Edge Cloud - 5.3.0 or Higher
Cause
- The vault token is expired.
Procedure
- Retrieve the token using below command from the affected node:
$ grep -i vault /etc/pf9/kube.env- To check the validity of the vault token, run the below command:
x
$ export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")$export VAULT_ADDR=http://127.0.0.1:8200$vault token lookup <token id>- If the above command return like below instead of the token information, then its confirmed that the vault token is expired and need to renew the vault token:
Error looking up token: Error making API request.URL: POST http://127.0.0.1:8200/v1/auth/token/lookupCode: 403. Errors:* bad tokenResolution
- To resolve the issue, renew the vault token as per the KB
Was this page helpful?