Custom CertManager Pod in CrashLoopBackoff During Luigi Installation

Problem

The custom cert-manager pod is crashing due to permissions issue.

Cluster

% kubectl get pods -A | grep cert-manager
cert-manager cert-manager-cainjector-646bf69b85-xhbxp 0/1 CrashLoopBackOff 64 (78s ago) 9h

Pod logs

% k logs cert-manager-cainjector-646bf69b85-z4ph9 -n cert-manager --tail 2
E0404 20:55:14.115006 1 main.go:45] "cert-manager: error executing command" err="customresourcedefinitions.apiextensions.k8s.io \"certificates.cert-manager.io\" is forbidden: User \"system:serviceaccount:cert-manager:cert-manager-cainjector\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"

Environment

• Platform9 Managed Kubernetes - v5.9.4

• Kubernetes version 1.28.6

Answer

This is a known issue, and it is being tracked in the jira PMK-6659.

Workaround

To completely disable pf9 managed cert-manager and continue using custom cert-manager:

1. Patch the pf9-addon-operator image to the custom private image platform9/pf9-addon-operator:8.0.5-hf1 which doesn't install/uninstall pf9-managed cert-manager.

2. Apply the below script, which will point all the CRB from luigi-system to cert-manager system.

Master node

#!/bin/bash

# List of ClusterRoleBindings to update
CRBS=(
  cert-manager-cainjector
  cert-manager-controller-issuers
  cert-manager-controller-clusterissuers
  cert-manager-controller-certificates
  cert-manager-controller-orders
  cert-manager-controller-challenges
  cert-manager-controller-ingress-shim
  cert-manager-controller-approve:cert-manager-io
  cert-manager-controller-certificatesigningrequests
  cert-manager-webhook:subjectaccessreviews
)

# New namespace value
NEW_NAMESPACE="cert-manager"

echo "Updating ClusterRoleBinding subjects to use namespace: $NEW_NAMESPACE"

for crb in "${CRBS[@]}"; do
  echo "Patching $crb..."
  kubectl patch clusterrolebinding "$crb" \
    --type=json \
    -p='[{"op": "replace", "path": "/subjects/0/namespace", "value": "'"$NEW_NAMESPACE"'"}]'
done

echo "All ClusterRoleBindings updated successfully."

1. Edit the below webhooks to set the namespace as cert-manager instead of luigi-system namespace.

Master node

kubectl edit ValidatingWebhookConfiguration cert-manager-webhook
kubectl edit MutatingWebhookConfiguration cert-manager-webhook

` 4. And delete all the three cert-manager deployments from luigi-system.

{% tabs %} {% tab language="javascript" title="Master node" %} {% code %}

kubectl delete deploy cert-manager-webhook -n luigi-system
kubectl delete deploy cert-manager-cainjector -n luigi-system
kubectl delete deploy cert-manager -n luigi-system

Once this is done all the pf9-managed cert-manager will be completely cleaned and wont be applied again.

Additional Information

The fix release version is on PMK version 5.14