Kubectl command failing with certificate error
Problem
kubectl
commands are failing and throwing the below certificate error in kubelet logs:
x509: certificate has expired or is not yet valid:
- The nodelet phase failing on the first
cert-gen
phase with the below error:
cat /tmp/authbs-certs.xxxx/kubelet/apiserver/request.json
extract_vault_json certificate
/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''certificate'\''])'
Traceback (most recent call last):
File "<string>", line 1, in <module>
KeyError: 'data'
cat /tmp/authbs-certs.fL0T/kubelet/apiserver/request.json
extract_vault_json issuing_ca
/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''issuing_ca'\''])'
openssl verify -CAfile /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt /tmp/authbs-certs.xxxx/kubelet/apiserver/request.crt
Error loading file /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt
+ echo 'Certificate is not signed by CA'
Certificate is not signed by CA
- While checking the directory/tmp/authbs-certs.xxxx/kubelet/apiserver mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.
$ cat request.json
{"errors":["permission denied"]}
Environment
- Platform9 Edge Cloud - 5.3.0 or Higher
Cause
- The vault token is expired.
Procedure
- Retrieve the token using below command from the affected node:
$ grep -i vault /etc/pf9/kube.env
- To check the validity of the vault token, run the below command:
x
$ export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")
$export VAULT_ADDR=http://127.0.0.1:8200
$vault token lookup <token id>
- If the above command return like below instead of the token information, then its confirmed that the vault token is expired and need to renew the vault token:
Error looking up token: Error making API request.
URL: POST http://127.0.0.1:8200/v1/auth/token/lookup
Code: 403. Errors:
* bad token
Resolution
- To resolve the issue, renew the vault token as per the KB
Was this page helpful?