Your First Kubernetes CronJob for Model Retraining

You've got a model in production. It was trained once, deployed, and it's serving predictions. Eventually someone asks: "shouldn't we retrain this on newer data?" The answer is yes, and the question is how.

The typical progression looks like this:

  1. Someone runs the training script manually on their laptop.
  2. This works until that person goes on vacation.
  3. The team sets up a shared VM where someone SSHes in and runs the script.
  4. This works until the VM gets restarted or reconfigured.
  5. The team automates it properly.

This article is about step 5 — specifically, using Kubernetes CronJobs to automate retraining. No fancy orchestration frameworks, no complex DAGs. Just a cron schedule and a container.

Prerequisites

You'll need:

  • A Kubernetes cluster (even a small one — retraining doesn't need a large cluster unless your model does)
  • A container registry (Docker Hub, GitHub Container Registry, or a private registry)
  • kubectl configured to talk to your cluster
  • A training script that works locally

Step 1: Make the training script self-contained

Before containerizing anything, the training script needs to work without human intervention. That means no interactive prompts, no hardcoded local paths, and no assumptions about the machine it runs on.

Here's a training script that pulls data, trains a model, and pushes the result to MLflow:

#!/usr/bin/env python3
"""train.py — automated retraining script."""

import os
import sys
import mlflow
import mlflow.sklearn
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

def main():
    tracking_uri = os.environ.get("MLFLOW_TRACKING_URI")
    if not tracking_uri:
        print("ERROR: MLFLOW_TRACKING_URI not set", file=sys.stderr)
        sys.exit(1)

    mlflow.set_tracking_uri(tracking_uri)
    mlflow.set_experiment("iris-classifier")

    with mlflow.start_run():
        # In a real system, this would pull fresh data from a database or object store
        X, y = load_iris(return_X_y=True)
        X_train, X_test, y_train, y_test = train_test_split(
            X, y, test_size=0.2, random_state=42
        )

        n_estimators = 100
        max_depth = 5

        mlflow.log_param("n_estimators", n_estimators)
        mlflow.log_param("max_depth", max_depth)

        model = RandomForestClassifier(
            n_estimators=n_estimators, max_depth=max_depth
        )
        model.fit(X_train, y_train)

        predictions = model.predict(X_test)
        acc = accuracy_score(y_test, predictions)
        mlflow.log_metric("accuracy", acc)
        print(f"Accuracy: {acc:.4f}")

        mlflow.sklearn.log_model(model, "model")
        print("Model logged to MLflow")

if __name__ == "__main__":
    main()

Key points:

  • Configuration comes from environment variables. The script doesn't know where MLflow lives — that's injected at runtime.
  • Errors produce nonzero exit codes. Kubernetes uses exit codes to determine if a Job succeeded or failed.
  • No interactive input. Everything is parameterized or has defaults.

Test this locally before containerizing it. If it doesn't work on your machine with just environment variables set, it won't work in a container.

Step 2: Build the container

The Dockerfile is straightforward:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY train.py .

CMD ["python", "train.py"]

And the requirements.txt:

scikit-learn==1.3.2
mlflow==2.9.2

Build and push:

docker build -t registry.example.com/ml-training/iris:v1 .
docker push registry.example.com/ml-training/iris:v1

Test the container locally before deploying to Kubernetes:

docker run --rm \
  -e MLFLOW_TRACKING_URI=http://host.docker.internal:5000 \
  registry.example.com/ml-training/iris:v1

If it runs and logs to MLflow, you're ready for Kubernetes.

Step 3: Create a Kubernetes Job

Before setting up the cron schedule, run the training as a one-off Job to verify it works in the cluster:

apiVersion: batch/v1
kind: Job
metadata:
  name: iris-retrain-test
  namespace: ml-jobs
spec:
  backoffLimit: 2
  template:
    spec:
      containers:
      - name: train
        image: registry.example.com/ml-training/iris:v1
        env:
        - name: MLFLOW_TRACKING_URI
          value: "http://mlflow.mlflow.svc.cluster.local:5000"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1"
      restartPolicy: Never

kubectl apply -f job.yaml
kubectl -n ml-jobs logs -f job/iris-retrain-test

A few things to note:

  • backoffLimit: 2 means Kubernetes will retry twice on failure, then mark the Job as failed. Without this, it retries indefinitely.
  • restartPolicy: Never is required for Jobs. Kubernetes manages retries at the Job level, not the Pod level.
  • Resource requests and limits prevent the training job from consuming the entire node. Set these based on your actual training requirements.

Step 4: Handle secrets

Your training script probably needs credentials — for the model registry, the data source, or both. Don't put these in the YAML file. Use Kubernetes Secrets.

Create the secret:

kubectl -n ml-jobs create secret generic mlflow-credentials \
  --from-literal=MLFLOW_TRACKING_URI=http://mlflow.mlflow.svc.cluster.local:5000 \
  --from-literal=AWS_ACCESS_KEY_ID=your-key \
  --from-literal=AWS_SECRET_ACCESS_KEY=your-secret

Reference it in the Job:

containers:
- name: train
  image: registry.example.com/ml-training/iris:v1
  envFrom:
  - secretRef:
      name: mlflow-credentials
  resources:
    requests:
      memory: "512Mi"
      cpu: "500m"
    limits:
      memory: "1Gi"
      cpu: "1"

envFrom injects every key in the Secret as an environment variable. The training script reads them via os.environ without knowing they came from a Secret.

Step 5: Set up the CronJob

Once the one-off Job works, wrap it in a CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: iris-retrain
  namespace: ml-jobs
spec:
  schedule: "0 3 * * 0"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 3600
      template:
        spec:
          containers:
          - name: train
            image: registry.example.com/ml-training/iris:v1
            envFrom:
            - secretRef:
                name: mlflow-credentials
            resources:
              requests:
                memory: "512Mi"
                cpu: "500m"
              limits:
                memory: "1Gi"
                cpu: "1"
          restartPolicy: Never

Breaking down the CronJob-specific fields:

  • schedule: "0 3 * * 0" — runs every Sunday at 3 AM. Standard cron syntax. Pick a time when the cluster is underutilized.
  • concurrencyPolicy: Forbid — if the previous run is still going when the next one is scheduled, skip it. This prevents pile-ups when training takes longer than expected.
  • successfulJobsHistoryLimit: 3 — keep the last 3 completed Job objects around for debugging. Older ones are garbage collected.
  • failedJobsHistoryLimit: 3 — same for failed Jobs.
  • activeDeadlineSeconds: 3600 — kill the Job if it runs longer than an hour. This is your safety net against hanging training runs.

Apply it:

kubectl apply -f cronjob.yaml

Verify it's scheduled:

kubectl -n ml-jobs get cronjob iris-retrain

You should see the schedule and the time of the next run.

Step 6: Know when it fails

A CronJob that fails silently is worse than no CronJob at all — you think retraining is happening when it isn't. You need alerting.

The simplest approach is a Kubernetes Event watcher. But for most teams, the practical answer is to check Job status from your existing monitoring stack.

If you're running Prometheus (and on Kubernetes, you probably are), the kube-state-metrics exporter already exposes Job metrics:

  • kube_job_status_failed — number of failed Jobs
  • kube_job_status_succeeded — number of succeeded Jobs

A basic Prometheus alert rule:

groups:
- name: ml-retraining
  rules:
  - alert: RetrainingJobFailed
    expr: |
      kube_job_status_failed{namespace="ml-jobs", job_name=~"iris-retrain.*"} > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "ML retraining job failed"
      description: "The iris-retrain CronJob has a failed run. Check logs with: kubectl -n ml-jobs logs job/{{ $labels.job_name }}"

This fires an alert if any retraining Job has been in a failed state for more than 5 minutes. Route it to Slack, PagerDuty, or email — wherever your team already gets alerts.

Quick check script

For teams without Prometheus, a simple script that runs on its own cron schedule works:

#!/bin/bash
FAILED=$(kubectl -n ml-jobs get jobs \
  -l app=iris-retrain \
  --field-selector=status.successful=0 \
  -o name 2>/dev/null | wc -l)

if [ "$FAILED" -gt 0 ]; then
  echo "WARNING: $FAILED failed retraining jobs" | \
    mail -s "Retraining failure" team@example.com
fi

Not elegant, but it works and it's better than silence.

What this looks like day-to-day

Once the CronJob is running, the day-to-day workflow changes:

  • No more manual retraining. The CronJob runs on schedule. New models appear in MLflow.
  • Debugging is straightforward. If a run fails, check the Job logs: kubectl -n ml-jobs logs job/<job-name>. The logs are the same output you'd see running locally.
  • Updating the training code means building a new container image and updating the CronJob's image reference. The schedule and infrastructure stay the same.
  • Adjusting the schedule is a one-line YAML change.

The main thing you'll want to add next is automated evaluation — comparing the new model against the current production model before promoting it. But that's a separate concern. The CronJob just ensures training happens reliably.

Common gotchas

Image pull failures. If your container registry requires authentication, set up an imagePullSecret on the namespace. Silent image pull failures are a common source of confusion.

Timezone. Kubernetes CronJobs use UTC by default. If you set schedule: "0 3 * * 0" expecting 3 AM local time, it might run at a different hour. Kubernetes 1.27+ supports timeZone on CronJobs if this matters.

Resource starvation. If the cluster doesn't have enough resources to schedule the training Pod, it'll sit in Pending state until the activeDeadlineSeconds kills it. Set resource requests realistically and monitor Pod scheduling.

Stale images. Using :latest tags means you're never sure which version of the training code is running. Use explicit version tags (:v1, :v2) or content-addressable tags (:sha-abc123).

Kubernetes CronJobs are the foundation of automated ML pipelines. If you need help setting up reliable retraining infrastructure, deploying.ai can help — reach out.