How to Renew Percona and Minio Certificates

Problem

The Minio and Percona pods are seen in CrashLoopBackOff state with errors in the pod logs as:

Percona pod logs

{"log":"2024-10-02T09:04:27.612080Z 0 [ERROR] WSREP: handshake with remote endpoint ssl://10.20.3.13:4567 failed: asio.ssl:337047686: 'certificate verify failed' ( 337047686: 'error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed')<br>","file":"/var/lib/mysql/mysqld-error.log"}
...
{"log":"2024-10-02T09:04:27.614816Z 0 [ERROR] WSREP: Provider/Node (gcomm://10.20.3.174,percona-db-pxc-db-pxc-1.percona-db-pxc-db-pxc) failed to establish connection with cluster (reason: 7)<br>","file":"/var/lib/mysql/mysqld-error.log"}

Minio pod logs

{"level":"FATAL","errKind":"","time":"2024-10-02T07:19:49.168830203Z","message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","error":{"message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","source":["cmd/server-main.go:169:cmd.serverHandleCmdArgs()"]}}

Environment

• Platform9 Edge Cloud - v5.6 [LTS2].

Procedure

NOTE

Percona certs will get auto-renewed starting SMCP 5.11.

For previous versions, follow the steps mentioned below:

The steps to be followed for the Percona certificate rotation are:

1. Retrieve the list of secrets in the percona namespace and verify the validity of the certificates.

Management cluster

$ kubectl get secrets -n percona

$ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

1. Delete the Percona DB secret. (Note: No new secrets will be created at this step)

Management cluster

$ kubectl delete secrets percona-db-pxc-db-ssl -n percona
$ kubectl delete secrets percona-db-pxc-db-ssl-internal -n percona

1. Get the list of Helm Deployments and make a note of the chart name for percona-db.

Management cluster

$ helm ls -n percona

1. Perform the helm upgrade for Percona DB.

Management cluster

$ helm upgrade --install percona-db /opt/pf9/airctl/conf/helm_charts/pxc-db-1.11.6.tgz --namespace percona --reuse-values

1. The deleted secrets will be recreated, and the certificates will also be renewed. Please verify this.

Management cluster

$ kubectl get secrets -n percona

$ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

The steps to be followed for the Minio certificate rotation are:

1. Check the list of secrets in the Minio namespace and verify the certificate validity.

Management cluster

$ kubectl get secrets -n minio

$ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

1. Restart the nodeletd stack in each master node one at a time within the Management cluster.

Info

The stack restart is mandatory as this will refresh the CA which signs the Minio certificates.

Each master node in Mgmt cluster

# systemctl stop pf9-nodeletd 
# /opt/pf9/nodelet/nodeletd phases stop 
# /opt/pf9/nodelet/nodeletd phases start 
# systemctl start pf9-nodeletd

1. Disable _requestAutoCert_in the minio tenant. (set requestAutoCert to false): $plugin[3qq4mela3y2]

Management cluster

$ kubectl -n minio edit tenant minio1
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
…
spec:
…
requestAutoCert: false           <---------

1. List the minio Secrets:

Management cluster

$ kubectl get secrets -n minio

1. Delete the existing minio certs:

Management cluster

$ kubectl -n minio delete secret minio1-tls
$ kubectl -n minio delete secret operator-tls

1. List the CSRs:

Management cluster

$ kubectl get csr -n minio

1. Delete the minio CSR:

Management cluster

$ kubectl -n minio delete csr minio1-minio-csr

1. Enable requestAuthCert in the minio tenant. (set requestAutoCert to true)

Management cluster

$ kubectl -n minio edit tenant minio1
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
…
spec:
…
requestAutoCert: true           <---------

1. The minio-tls and operator secret will be regenerated, which the minio pods use. Also check certificate validity.

Management cluster

$ kubectl -n minio get secrets

$ kubectl -n minio get csr

$ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

Additional Information

Platform9 team has filed an internal jira AIR-1401 to track Minio cert issue, and jira AIR-1402 to track the Percona cert rotation issue. To know more details on the ETA for the fix and the release version, please reach out to support team mentioning the jira-ID.