Kubernetes Resource Quota Not Updating
Problem
The Resource quota for a namespace does not display or get updated even after pods are created. Changes in settings and usage are ignored, current usage looks frozen
kubectl describe quota -n test Name: test-quota Namespace: test Resource Used Hard
Environment
- Platform9 Managed Kubernetes - All Versions
Cause
A broken Customer Resource Definition will cause issues with the Kubernetes Garbage Collection. Specifically in this case a broken Cert Manager webhook.
Errors similar to this can be observed in the kube-controller-manager logs.
I0628 22:31:35.854601 1 shared_informer.go:223] Waiting for caches to sync for garbage collector0627 02:30:32.549664 1 reflector.go:178] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to list *v1.PartialObjectMetadata: conversion webhook for cert-manager.io/v1alpha2, Kind=CertificateRequest failed: Posthttps://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s: x509: certificate signed by unknown authority
Resolution
The resolution in this specific case is to either remove the Cert-Manager Deployment or fix the underlying issue causing the webhook to break.
Note that in this case, the broken CRD was the Cert-Manager deployment. However, any CRD which fails an excessive number of times can cause the Garbage Collection to fail.
Additional Information
- Related upstream bug.