How to Regenerate Hostagent Certificates of Nodes in SMCP

Problem

• Hostagent certificates of nodes in the workload cluster expired or near expiry.

Environment

• Self Managed Cloud Platform - v5.10.0-3248001 and Higher
• Hostagent Certificates.

Cause

• Platform9 has identified this as a known bug with ID: AIR-1503.
• Refer the Workaround section to renew the certificates.

Diagnostics

• Check for pf9-comms service logs located at /var/log/pf9/comms/comms.log on the affected node/s.

/var/log/pf9/comms/comms.log

[ERROR] [URL]-::1-8111-28 - TLS socket for client 93508 error: Error: 140582915938176:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate

• Check the vouch-noauth pod logs in the relevant region namespace.

Logs of socat container in vouch-noauth pod

socat[7] E SSL_accept(): error:0A000086:SSL routines::certificate verify failed

• Check hostagent certificate expiry.

On affected node

$ openssl x509 -in /etc/pf9/certs/hostagent/cert.pem -text -noout

Workaround

There are 2 scenarios for certificates

• Certificates near expiration:

• Certificates already expired:

Certificates near expiration:

• Check the hostagent certificate using the command in Diagnostics section.
• Run the below script on node/s for which the hostagent certificate are expiring.

On affected node

/opt/pf9/hostagent/bin/python /opt/pf9/hostagent/bin/host-certs.py refresh --vouch-url http://localhost:8558

• The script will renew the hostagent certificates automatically.

Certificates already expired:

Follow the below steps in order on ALL affected nodes. Prerequisite: Ensure to export the admin.yaml kubeconfig on the nodes.

On affected node

export KUBECONFIG=/etc/pf9/kube.d/kubeconfigs/admin.yaml

1. Read the current certificate subject/CN from the existing hostagent certificate. Identify the CN in this certificate. This CN is referred to as <OLD_CN> in the further steps.

   On affected node

   openssl x509 -in /etc/pf9/certs/hostagent/cert.pem -noout -subject

2. Generate a new private key and CSR on the host using the same CN identified in Step 1.

   On affected node

   openssl req -newkey rsa:2048 -nodes \
     -keyout /tmp/new_key.pem \
     -out /tmp/new_csr.pem \
     -subj "/CN=<OLD_CN>"

3. Copy the CSR into the vouch-noauth container of the vouch-noauth pod in the relevant region namespace.

   In vouch-noauth container of vouch-noauth pod

   kubectl cp new_csr.pem <REGION_NAMESPACE>/<VOUCH-NOAUTH-POD>:/tmp/new_csr.pem

4. Run the signing request from inside the vouch-noauth pod against the local vouch endpoint. Use the same <OLD_CN> as identified in Step 1.

   In vouch-noauth container of vouch-noauth pod

   python3 -c "
   import json, sys
   csr = open('/tmp/new_csr.pem').read()
   print(json.dumps({'common_name': '<OLD_CN>', 'csr': csr}))
   " | curl -s -X POST http://127.0.0.1:8558/v1/sign/cert \
     -H "Content-Type: application/json" \
     -d @- \
     -o /tmp/signed_response.json

5. Copy the signed response back to the host

   On affected node

   kubectl cp <REGION_NAMESPACE>/<VOUCH-NOAUTH-POD>:/tmp/signed_response.json /tmp/signed_response.json 

6. Extract and place the new hostagent certificate on the node.

   On affected node

     python3 -c "import json; d=json.load(open('/tmp/signed_response.json')); print(d['certificate'])" \
     > /etc/pf9/certs/hostagent/cert.pem

7. Copy the new private key into the hostagent key path and set the correct ownership.

   On affected node

     cp /tmp/new_key.pem /etc/pf9/certs/hostagent/key.pem
     chown pf9:pf9group /etc/pf9/certs/hostagent/cert.pem  /etc/pf9/certs/hostagent/key.pem

8. Restart pf9-comms and pf9-sidekick.

   On affected node

   sudo systemctl restart pf9-comms pf9-sidekick

Validation

• Check hostagent certificate expiry.

  On affected node

  openssl x509 -in /etc/pf9/certs/hostagent/cert.pem -text -noout

Additional Information

• Reach out to Platform9 Support in case of any questions/concerns regarding this process.
• This article will be updated in a timely manner with the new information on bug fix version and more.