etcd-backup addon fails to generate backups if ETCDCTL_CACERT, ETCDCTL_CERT and ETCDCTL_KEY is set i

Problem

• etcd-backup addon fails to generate backups if ETCDCTLCACERT, ETCDCTL CERT and ETCDCTL_KEY is set in the cronjob.

• Below environment variables are set in etcd-backup cronjob:

Javascript

# kubectl -n kube-system get cronjob etcd-backup-with-interval -o yaml |grep -A7 ETCDCTL_CACERT
            - name: ETCDCTL_CACERT
              value: /certs/apiserver/etcd/ca.crt
            - name: ETCDCTL_CERT
              value: /certs/apiserver/etcd/request.crt
            - name: ETCDCTL_KEY
              value: /certs/apiserver/etcd/request.key

• etcd-backup pods remain in NotReady state:

Javascript

# kubectl get pod -o wide | grep etcd
etcd-backup-with-interval-27801890-zjghf   1/2     NotReady   0                7m41s   10.164.209.47    10.164.209.47   <none>           <none>

• db.part files are created in the backup path:

Javascript

# kubectl logs etcd-backup-with-interval-27801890-zjghf etcd-backup
{"level":"info","ts":1668113401.1358364,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/backup/etcd-snapshot-2022-11-10_20:50:01_UTC.db.part"}%

Environment

• Platform9 Managed Kubernetes - v5.6.1(kubernetes version 1.23)

Resolution

• Edit the cronjob and remove below 6 environment variables:

Javascript

# kubectl -n kube-system edit cronjob etcd-backup-with-interval

>> Remove below 6 lines and save the config:
            - name: ETCDCTL_CACERT
              value: /certs/apiserver/etcd/ca.crt
            - name: ETCDCTL_CERT
              value: /certs/apiserver/etcd/request.crt
            - name: ETCDCTL_KEY
              value: /certs/apiserver/etcd/request.key

• Then check if the backups work.