Disabling pf9-managed cert-manager

Problem

After upgrading the cluster to version 1.28.6, cert-manager is getting installed in the luigi-system namespace is having conflict with the existing custom cert manager in the cert-manager namespace. As a result, the cert-manager pods are going into an error state.

Javascript

% kubectl get pods -A | grep cert-manager
cert-manager             cert-manager-89b545d6d-zstl8                             1/1     Running            2 (3h4m ago)       9h
cert-manager             cert-manager-cainjector-646bf69b85-xhbxp                 0/1     CrashLoopBackOff   64 (78s ago)       9h
cert-manager             cert-manager-webhook-796478777-qzzfs                     1/1     Running            0                  9h
luigi-system             cert-manager-5dcbbc765c-hpbql                            1/1     Running            2 (5h3m ago)       8h
luigi-system             cert-manager-cainjector-6db486b6b7-d8btt                 1/1     Running            2 (5h3m ago)       8h
luigi-system             cert-manager-webhook-57876b9fd-j4f6l                     1/1     Running            0                  8h

Environment

• Platform9 Managed Kubernetes 5.9.4

• Kubernetes version 1.28.6

Procedure

To completely disable pf9 managed cert-manager and continue using custom cert-manager:

1. Patch the pf9-addon-operator image to the custom private image which doesn't install/uninstall pf9-managed cert-manager.

2. Apply the below script, which will point all the CRB from luigi-system to cert-manager system.

Javascript

#!/bin/bash

# List of ClusterRoleBindings to update
CRBS=(
  cert-manager-cainjector
  cert-manager-controller-issuers
  cert-manager-controller-clusterissuers
  cert-manager-controller-certificates
  cert-manager-controller-orders
  cert-manager-controller-challenges
  cert-manager-controller-ingress-shim
  cert-manager-controller-approve:cert-manager-io
  cert-manager-controller-certificatesigningrequests
  cert-manager-webhook:subjectaccessreviews
)

# New namespace value
NEW_NAMESPACE="cert-manager"

echo "Updating ClusterRoleBinding subjects to use namespace: $NEW_NAMESPACE"

for crb in "${CRBS[@]}"; do
  echo "Patching $crb..."
  kubectl patch clusterrolebinding "$crb" \
    --type=json \
    -p='[{"op": "replace", "path": "/subjects/0/namespace", "value": "'"$NEW_NAMESPACE"'"}]'
done

echo "All ClusterRoleBindings updated successfully."

1. Edit the webhooks to point to the cert-manager namespace instead of luigi-system namespace.

Javascript

kubectl edit ValidatingWebhookConfiguration cert-manager-webhook
kubectl edit MutatingWebhookConfiguration cert-manager-webhook

And delete all the 3 cert-manager deployments from luigi-system.

Javascript

kubectl delete deploy cert-manager-webhook -n luigi-system
kubectl delete deploy cert-manager-cainjector -n luigi-system
kubectl delete deploy cert-manager -n luigi-system

Once this is done all the pf9-managed cert-manager will be completely cleaned and wont be applied again.