Kubernetes CronJobs Failed with Error "too many missed start time (> 100)"

Problem

• The Kubernetes cronjobs failed to start with an error mentioned below

Error

Warning  FailedNeedsStart  3m21s (x93148 over 10d)  cronjob-controller  Cannot determine if job needs to be started: too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew

Environment

• Platform9 Managed Kubernetes - All Versions
• CronJobs

Cause

• For every cronjob, CronJob Controller checks how many schedules it missed in the duration from its last scheduled time until now. If there are more than 100 missed schedules, then it does not start the job and logs the error mentioned below.

CronJob Controller error

Cannot determine if job needs to be started. Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.

• The CronJob failures can occur if the CronJob's status.lastScheduleTime, metadata.creationTimestamp fields are unset.
• Kubernetes first looks at the lastScheduleTime field to calculate how many jobs may have been missed. If that field is unset (such as for a newly created CronJob), it uses the creationTimestamp field in the metadata.

Resolution

• The solution to this is recreating the failed cronjob as it will help to reset the null values for the spec.jobTemplate.metadata.creationTimestamp and spec.jobTemplate.spec.template.metadata.creationTimestamp fields.