AddOn Operator Pod Restarting due to Error "Use SANs or Temporarily Enable Common Name Matching with

Problem

AddOn Operator Pod Restarting due to Error "Use SANs or Temporarily Enable Common Name Matching with GODEBUG=x509ignoreCN=0"

Environment

• Platform9 Edge Cloud - 5.3 LTS Patch #6 and Below
• AddOn Operator

Cause

AddOn operator certificate generation is using CN instead of SAN.

Bash

$ kubectl logs pf9-addon-operator-5f5cd7649b-5dgvz -n pf9-addons

{"level":"error","msg":"Error in healthcheck: Error listing ClusterAddon objects from sunpike","time":"2022-07-07T07:38:49Z"} {"level":"error","msg":"List ClusterAddons error count: 5 of 10","time":"2022-07-07T07:38:49Z"} {"level":"error","msg":"Get \"https://airctl-pximsp02.pf9.localnet/qbert/v4/a1c4d5887ce34c81a2c8696bd9d67171/sunpike/apis/sunpike.platform9.com/v1alpha2/namespaces/default/clusteraddons?labelSelector=sunpike.pf9.io%2Fcluster%3Db4877409-fe75-421a-890f-4faa49b5434d\": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0Unable to list ClusterAddons","time":"2022-07-07T07:40:19Z"}

{"level":"error","msg":"Unable to get ClusterAddon objects for cluster: b4877409-fe75-421a-890f-4faa49b5434d Error listing ClusterAddon objects from sunpike","time":"2022-07-07T07:40:19Z"}

Resolution

1. Scale the pf9-addon-operator deployment replica to 0.

Command

kubectl scale deployment/pf9-addon-operator --replicas=0 -n pf9-addons

1. Edit the pf9-addon-operator deployment and make the below changes in the pf9-addon-operator deployment.

Add the following environments variable for pf9-addon-operator container

env:
  - name: GODEBUG
    value: "x509ignoreCN=0"

Example:

• Edit the pf9-addon-operator deployment:

Edit Deployment

# kubectl edit deployment/pf9-addon-operator -n pf9-addons

• After modification, the changed spec content in spec.template should look like

Changed Spec

spec:
  containers:
  - env:
    - name: GODEBUG
      value: x509ignoreCN=0
    - name: LOGLEVEL
      value: INFO
...
    image: localhost:5100/platform9/pf9-addon-operator:3.2.3

1. On all the master nodes part of the cluster, edit file /opt/pf9/pf9-kube/conf/addons/pf9-addon-operator/pf9-addon-operator-deployment.yaml and add the changes made similar to the deployment object above to this file. These changes made to pf9-addon-operator-deployment.yaml file will ensure that the env vars are persisted when the stack restarts.

   • Edit thepf9-addon-operator-deployment.yaml file.

$ sudo vi /opt/pf9/pf9-kube/conf/addons/pf9-addon-operator/pf9-addon-operator-deployment.yaml

- After modification, the changes should look like:

Changed Spec

containers:
- name: pf9-addon-operator
...
  env:
  - name: GODEBUG
    value: x509ignoreCN=0
  - name: LOGLEVEL
    value: "INFO"

1. Scale the pf9-addon-operator deployment replica back to 1.

Command

kubectl scale deployment/pf9-addon-operator --replicas=1 -n pf9-addons

1. Check the status of the new pf9-addon-operator pod replica.

Additional Information

The issue is fixed starting 5.3 LTS Patch #10 & Patch #12 onwards.