# How to Renew Percona and Minio Certificates

## Problem

The Minio and Percona pods are seen in CrashLoopBackOff state with errors in the pod logs as:

{% tabs %}
{% tab title="Percona pod logs" %}

```javascript
{"log":"2024-10-02T09:04:27.612080Z 0 [ERROR] WSREP: handshake with remote endpoint ssl://10.20.3.13:4567 failed: asio.ssl:337047686: 'certificate verify failed' ( 337047686: 'error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed')<br>","file":"/var/lib/mysql/mysqld-error.log"}
...
{"log":"2024-10-02T09:04:27.614816Z 0 [ERROR] WSREP: Provider/Node (gcomm://10.20.3.174,percona-db-pxc-db-pxc-1.percona-db-pxc-db-pxc) failed to establish connection with cluster (reason: 7)<br>","file":"/var/lib/mysql/mysqld-error.log"}
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Minio pod logs" %}

```javascript
{"level":"FATAL","errKind":"","time":"2024-10-02T07:19:49.168830203Z","message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","error":{"message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","source":["cmd/server-main.go:169:cmd.serverHandleCmdArgs()"]}}
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Edge Cloud - v5.6 \[LTS2].

## Procedure

{% hint style="success" %}
**NOTE**

Percona certs will get auto-renewed starting SMCP 5.11.

For previous versions, follow the steps mentioned below:
{% endhint %}

**The steps to be followed for the Percona certificate rotation are:**

1. Retrieve the list of secrets in the percona namespace and verify the validity of the certificates.

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl get secrets -n percona

$ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
```

{% endtab %}
{% endtabs %}

2. Delete the Percona DB secret. (Note: No new secrets will be created at this step)

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl delete secrets percona-db-pxc-db-ssl -n percona
$ kubectl delete secrets percona-db-pxc-db-ssl-internal -n percona
```

{% endtab %}
{% endtabs %}

3. Get the list of Helm Deployments and make a note of the chart name for percona-db.

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ helm ls -n percona
```

{% endtab %}
{% endtabs %}

4. Perform the helm upgrade for Percona DB.

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ helm upgrade --install percona-db /opt/pf9/airctl/conf/helm_charts/pxc-db-1.11.6.tgz --namespace percona --reuse-values
```

{% endtab %}
{% endtabs %}

5. The deleted secrets will be recreated, and the certificates will also be renewed. Please verify this.

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl get secrets -n percona

$ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
```

{% endtab %}
{% endtabs %}

## **The steps to be followed for the Minio certificate rotation are:**

1. Check the list of secrets in the Minio namespace and verify the certificate validity.

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl get secrets -n minio

$ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
```

{% endtab %}
{% endtabs %}

2. Restart the nodeletd stack in each master node one at a time within the Management cluster.

{% hint style="info" %}
**Info**

The stack restart is mandatory as this will refresh the CA which signs the Minio certificates.
{% endhint %}

{% tabs %}
{% tab title="Each master node in Mgmt cluster" %}

```javascript
# systemctl stop pf9-nodeletd 
# /opt/pf9/nodelet/nodeletd phases stop 
# /opt/pf9/nodelet/nodeletd phases start 
# systemctl start pf9-nodeletd
```

{% endtab %}
{% endtabs %}

3. Disable \_**requestAutoCert**\_in the minio tenant. (set requestAutoCert to false): $plugin\[3qq4mela3y2]

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl -n minio edit tenant minio1
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
…
spec:
…
requestAutoCert: false           <---------
```

{% endtab %}
{% endtabs %}

4. List the minio Secrets:

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl get secrets -n minio
```

{% endtab %}
{% endtabs %}

5. Delete the existing minio certs:

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl -n minio delete secret minio1-tls
$ kubectl -n minio delete secret operator-tls
```

{% endtab %}
{% endtabs %}

6. List the CSRs:

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl get csr -n minio
```

{% endtab %}
{% endtabs %}

7. Delete the minio CSR:

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl -n minio delete csr minio1-minio-csr
```

{% endtab %}
{% endtabs %}

8. Enable requestAuthCert in the minio tenant. (set requestAutoCert to true)

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl -n minio edit tenant minio1
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
…
spec:
…
requestAutoCert: true           <---------
```

{% endtab %}
{% endtabs %}

9. The minio-tls and operator secret will be regenerated, which the minio pods use. Also check certificate validity.

{% tabs %}
{% tab title="Management cluster" %}

```javascript
$ kubectl -n minio get secrets

$ kubectl -n minio get csr

$ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
```

{% endtab %}
{% endtabs %}

## Additional Information

Platform9 team has filed an internal jira AIR-1401 to track Minio cert issue, and jira AIR-1402 to track the Percona cert rotation issue. To know more details on the ETA for the fix and the release version, please reach out to support team mentioning the jira-ID.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/smcp/how-to/how-to-renew-minio-certificates.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.