Skip to main content

One post tagged with "cronjob"

View All Tags

Kubernetes CronJob failed to schedule: Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew

· 2 min read

Kubernetes v1.13.3, schedule 了一个 cronjob, 每 5 分钟运行一次, 但发现已经有 3 天没有新的 pod 被创建了:

# kubectl get cronjob/dingtalk-atndsyncer
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
dingtalk-atndsyncer   */5 * * * *   False     0        3d1h            4d21h

cronjob 中的.spec.concurrencyPolicy 为 Forbid, 不允许多任务并行, describe 该 cronjob 提示：FailedNeedsStart, 具体 message 是"Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew."

# kubectl describe cronjob/dingtalk-atndsyncer
Name:                       dingtalk-atndsyncer
Namespace:                  default
Labels:                     app=dingtalk-atndsyncer
Annotations:                <none>
Schedule:                   */5 * * * *
Concurrency Policy:         Forbid
Suspend:                    False
Starting Deadline Seconds:  <unset>
Selector:                   <unset>
Parallelism:                <unset>
Completions:                <unset>
Pod Template:
  Labels:  <none>
  Containers:
   dingtalk-atndsyncer:
    Image:      dingtalk-atndsyncer:v1.0
    Port:       <none>
    Host Port:  <none>
    Environment:
      ASPNETCORE_ENVIRONMENT:               Production
    Mounts:                                 <none>
  Volumes:                                  <none>
Last Schedule Time:                         Fri, 06 Sep 2019 08:15:00 +0800
Active Jobs:                                <none>
Events:
  Type     Reason            Age                   From                Message
  ----     ------            ----                  ----                -------
  Warning  FailedNeedsStart  43m (x790 over 178m)  cronjob-controller  Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.
  Warning  FailedNeedsStart  25m (x89 over 40m)    cronjob-controller  Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.
  Warning  FailedNeedsStart  119s (x117 over 22m)  cronjob-controller  Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.

Google 后仔细阅读官方文档(https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#starting-deadline), 说是如果没有配置.spec.startingDeadlineSeconds, 则从最后一次的 schedule 时间统计错过的 schedule 次数，如果超过 100 次就不再 schedule 尝试把.spec.startingDeadlineSeconds 配置为 300 秒, 意味着如果 5 分钟内错过 schedule 超过 100 次，才不会 schedule (因为 schedule 周期是 5 分钟, 所以是一个不太可能达到的条件), 配置后任务 schedule 正常了

ClustrMaps