Custom CertManager Pod in CrashLoopBackoff During Luigi Installation
Problem
The custom cert-manager pod is crashing due to permissions issue.
% kubectl get pods -A | grep cert-managercert-manager cert-manager-cainjector-646bf69b85-xhbxp 0/1 CrashLoopBackOff 64 (78s ago) 9h % k logs cert-manager-cainjector-646bf69b85-z4ph9 -n cert-manager --tail 2E0404 20:55:14.115006 1 main.go:45] "cert-manager: error executing command" err="customresourcedefinitions.apiextensions.k8s.io \"certificates.cert-manager.io\" is forbidden: User \"system:serviceaccount:cert-manager:cert-manager-cainjector\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"Environment
- Platform9 Managed Kubernetes - v5.9.4
- Kubernetes version 1.28.6
Answer
This is a known issue, and it is being tracked in the jira PMK-6659.
Workaround
To completely disable pf9 managed cert-manager and continue using custom cert-manager:
- Patch the pf9-addon-operator image to the custom private image
platform9/pf9-addon-operator:8.0.5-hf1which doesn't install/uninstall pf9-managed cert-manager. - Apply the below script, which will point all the CRB from luigi-system to cert-manager system.
x
# List of ClusterRoleBindings to updateCRBS=( cert-manager-cainjector cert-manager-controller-issuers cert-manager-controller-clusterissuers cert-manager-controller-certificates cert-manager-controller-orders cert-manager-controller-challenges cert-manager-controller-ingress-shim cert-manager-controller-approve:cert-manager-io cert-manager-controller-certificatesigningrequests cert-manager-webhook:subjectaccessreviews)# New namespace valueNEW_NAMESPACE="cert-manager"echo "Updating ClusterRoleBinding subjects to use namespace: $NEW_NAMESPACE"for crb in "${CRBS[@]}"; do echo "Patching $crb..." kubectl patch clusterrolebinding "$crb" \ --type=json \ -p='[{"op": "replace", "path": "/subjects/0/namespace", "value": "'"$NEW_NAMESPACE"'"}]'doneecho "All ClusterRoleBindings updated successfully."- Edit the below webhooks to set the namespace as
cert-managerinstead ofluigi-systemnamespace.
kubectl edit ValidatingWebhookConfiguration cert-manager-webhookkubectl edit MutatingWebhookConfiguration cert-manager-webhook` 4. And delete all the three cert-manager deployments from luigi-system.
kubectl delete deploy cert-manager-webhook -n luigi-systemkubectl delete deploy cert-manager-cainjector -n luigi-systemkubectl delete deploy cert-manager -n luigi-systemOnce this is done all the pf9-managed cert-manager will be completely cleaned and wont be applied again.
Additional Information
The fix release version is on PMK version 5.14
Was this page helpful?