[{"data":1,"prerenderedAt":1509},["ShallowReactive",2],{"article-kubernetes-cronjob-model-retraining":3,"content-query-Odeje5bezg":851},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"body":11,"_type":845,"_id":846,"_source":847,"_file":848,"_stem":849,"_extension":850},"\u002Farticles\u002Fkubernetes-cronjob-model-retraining","articles",false,"","Your First Kubernetes CronJob for Model Retraining","A practical walkthrough of moving from manual model retraining on your laptop to an automated Kubernetes CronJob — covering container builds, secrets, and failure alerting.","2025-07-21",{"type":12,"children":13,"toc":830},"root",[14,22,28,33,63,68,75,80,111,117,122,127,139,144,178,183,189,194,205,218,226,231,242,247,256,261,267,272,283,292,297,338,344,349,354,363,368,377,396,402,407,416,421,494,499,508,513,522,527,533,538,543,556,581,586,595,600,607,612,621,626,632,637,688,693,699,717,742,760,802,806],{"type":15,"tag":16,"props":17,"children":19},"element","h1",{"id":18},"your-first-kubernetes-cronjob-for-model-retraining",[20],{"type":21,"value":8},"text",{"type":15,"tag":23,"props":24,"children":25},"p",{},[26],{"type":21,"value":27},"You've got a model in production. It was trained once, deployed, and it's serving predictions. Eventually someone asks: \"shouldn't we retrain this on newer data?\" The answer is yes, and the question is how.",{"type":15,"tag":23,"props":29,"children":30},{},[31],{"type":21,"value":32},"The typical progression looks like this:",{"type":15,"tag":34,"props":35,"children":36},"ol",{},[37,43,48,53,58],{"type":15,"tag":38,"props":39,"children":40},"li",{},[41],{"type":21,"value":42},"Someone runs the training script manually on their laptop.",{"type":15,"tag":38,"props":44,"children":45},{},[46],{"type":21,"value":47},"This works until that person goes on vacation.",{"type":15,"tag":38,"props":49,"children":50},{},[51],{"type":21,"value":52},"The team sets up a shared VM where someone SSHes in and runs the script.",{"type":15,"tag":38,"props":54,"children":55},{},[56],{"type":21,"value":57},"This works until the VM gets restarted or reconfigured.",{"type":15,"tag":38,"props":59,"children":60},{},[61],{"type":21,"value":62},"The team automates it properly.",{"type":15,"tag":23,"props":64,"children":65},{},[66],{"type":21,"value":67},"This article is about step 5 — specifically, using Kubernetes CronJobs to automate retraining. No fancy orchestration frameworks, no complex DAGs. Just a cron schedule and a container.",{"type":15,"tag":69,"props":70,"children":72},"h2",{"id":71},"prerequisites",[73],{"type":21,"value":74},"Prerequisites",{"type":15,"tag":23,"props":76,"children":77},{},[78],{"type":21,"value":79},"You'll need:",{"type":15,"tag":81,"props":82,"children":83},"ul",{},[84,89,94,106],{"type":15,"tag":38,"props":85,"children":86},{},[87],{"type":21,"value":88},"A Kubernetes cluster (even a small one — retraining doesn't need a large cluster unless your model does)",{"type":15,"tag":38,"props":90,"children":91},{},[92],{"type":21,"value":93},"A container registry (Docker Hub, GitHub Container Registry, or a private registry)",{"type":15,"tag":38,"props":95,"children":96},{},[97,104],{"type":15,"tag":98,"props":99,"children":101},"code",{"className":100},[],[102],{"type":21,"value":103},"kubectl",{"type":21,"value":105}," configured to talk to your cluster",{"type":15,"tag":38,"props":107,"children":108},{},[109],{"type":21,"value":110},"A training script that works locally",{"type":15,"tag":69,"props":112,"children":114},{"id":113},"step-1-make-the-training-script-self-contained",[115],{"type":21,"value":116},"Step 1: Make the training script self-contained",{"type":15,"tag":23,"props":118,"children":119},{},[120],{"type":21,"value":121},"Before containerizing anything, the training script needs to work without human intervention. That means no interactive prompts, no hardcoded local paths, and no assumptions about the machine it runs on.",{"type":15,"tag":23,"props":123,"children":124},{},[125],{"type":21,"value":126},"Here's a training script that pulls data, trains a model, and pushes the result to MLflow:",{"type":15,"tag":128,"props":129,"children":134},"pre",{"code":130,"language":131,"meta":7,"className":132},"#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"train.py — automated retraining script.\"\"\"\n\nimport os\nimport sys\nimport mlflow\nimport mlflow.sklearn\nfrom sklearn.datasets import load_iris\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\n\ndef main():\n    tracking_uri = os.environ.get(\"MLFLOW_TRACKING_URI\")\n    if not tracking_uri:\n        print(\"ERROR: MLFLOW_TRACKING_URI not set\", file=sys.stderr)\n        sys.exit(1)\n\n    mlflow.set_tracking_uri(tracking_uri)\n    mlflow.set_experiment(\"iris-classifier\")\n\n    with mlflow.start_run():\n        # In a real system, this would pull fresh data from a database or object store\n        X, y = load_iris(return_X_y=True)\n        X_train, X_test, y_train, y_test = train_test_split(\n            X, y, test_size=0.2, random_state=42\n        )\n\n        n_estimators = 100\n        max_depth = 5\n\n        mlflow.log_param(\"n_estimators\", n_estimators)\n        mlflow.log_param(\"max_depth\", max_depth)\n\n        model = RandomForestClassifier(\n            n_estimators=n_estimators, max_depth=max_depth\n        )\n        model.fit(X_train, y_train)\n\n        predictions = model.predict(X_test)\n        acc = accuracy_score(y_test, predictions)\n        mlflow.log_metric(\"accuracy\", acc)\n        print(f\"Accuracy: {acc:.4f}\")\n\n        mlflow.sklearn.log_model(model, \"model\")\n        print(\"Model logged to MLflow\")\n\nif __name__ == \"__main__\":\n    main()\n","python",[133],"language-python",[135],{"type":15,"tag":98,"props":136,"children":137},{"__ignoreMap":7},[138],{"type":21,"value":130},{"type":15,"tag":23,"props":140,"children":141},{},[142],{"type":21,"value":143},"Key points:",{"type":15,"tag":81,"props":145,"children":146},{},[147,158,168],{"type":15,"tag":38,"props":148,"children":149},{},[150,156],{"type":15,"tag":151,"props":152,"children":153},"strong",{},[154],{"type":21,"value":155},"Configuration comes from environment variables.",{"type":21,"value":157}," The script doesn't know where MLflow lives — that's injected at runtime.",{"type":15,"tag":38,"props":159,"children":160},{},[161,166],{"type":15,"tag":151,"props":162,"children":163},{},[164],{"type":21,"value":165},"Errors produce nonzero exit codes.",{"type":21,"value":167}," Kubernetes uses exit codes to determine if a Job succeeded or failed.",{"type":15,"tag":38,"props":169,"children":170},{},[171,176],{"type":15,"tag":151,"props":172,"children":173},{},[174],{"type":21,"value":175},"No interactive input.",{"type":21,"value":177}," Everything is parameterized or has defaults.",{"type":15,"tag":23,"props":179,"children":180},{},[181],{"type":21,"value":182},"Test this locally before containerizing it. If it doesn't work on your machine with just environment variables set, it won't work in a container.",{"type":15,"tag":69,"props":184,"children":186},{"id":185},"step-2-build-the-container",[187],{"type":21,"value":188},"Step 2: Build the container",{"type":15,"tag":23,"props":190,"children":191},{},[192],{"type":21,"value":193},"The Dockerfile is straightforward:",{"type":15,"tag":128,"props":195,"children":200},{"code":196,"language":197,"meta":7,"className":198},"FROM python:3.11-slim\n\nWORKDIR \u002Fapp\n\nCOPY requirements.txt .\nRUN pip install --no-cache-dir -r requirements.txt\n\nCOPY train.py .\n\nCMD [\"python\", \"train.py\"]\n","dockerfile",[199],"language-dockerfile",[201],{"type":15,"tag":98,"props":202,"children":203},{"__ignoreMap":7},[204],{"type":21,"value":196},{"type":15,"tag":23,"props":206,"children":207},{},[208,210,216],{"type":21,"value":209},"And the ",{"type":15,"tag":98,"props":211,"children":213},{"className":212},[],[214],{"type":21,"value":215},"requirements.txt",{"type":21,"value":217},":",{"type":15,"tag":128,"props":219,"children":221},{"code":220},"scikit-learn==1.3.2\nmlflow==2.9.2\n",[222],{"type":15,"tag":98,"props":223,"children":224},{"__ignoreMap":7},[225],{"type":21,"value":220},{"type":15,"tag":23,"props":227,"children":228},{},[229],{"type":21,"value":230},"Build and push:",{"type":15,"tag":128,"props":232,"children":237},{"code":233,"language":234,"meta":7,"className":235},"docker build -t registry.example.com\u002Fml-training\u002Firis:v1 .\ndocker push registry.example.com\u002Fml-training\u002Firis:v1\n","bash",[236],"language-bash",[238],{"type":15,"tag":98,"props":239,"children":240},{"__ignoreMap":7},[241],{"type":21,"value":233},{"type":15,"tag":23,"props":243,"children":244},{},[245],{"type":21,"value":246},"Test the container locally before deploying to Kubernetes:",{"type":15,"tag":128,"props":248,"children":251},{"code":249,"language":234,"meta":7,"className":250},"docker run --rm \\\n  -e MLFLOW_TRACKING_URI=http:\u002F\u002Fhost.docker.internal:5000 \\\n  registry.example.com\u002Fml-training\u002Firis:v1\n",[236],[252],{"type":15,"tag":98,"props":253,"children":254},{"__ignoreMap":7},[255],{"type":21,"value":249},{"type":15,"tag":23,"props":257,"children":258},{},[259],{"type":21,"value":260},"If it runs and logs to MLflow, you're ready for Kubernetes.",{"type":15,"tag":69,"props":262,"children":264},{"id":263},"step-3-create-a-kubernetes-job",[265],{"type":21,"value":266},"Step 3: Create a Kubernetes Job",{"type":15,"tag":23,"props":268,"children":269},{},[270],{"type":21,"value":271},"Before setting up the cron schedule, run the training as a one-off Job to verify it works in the cluster:",{"type":15,"tag":128,"props":273,"children":278},{"code":274,"language":275,"meta":7,"className":276},"apiVersion: batch\u002Fv1\nkind: Job\nmetadata:\n  name: iris-retrain-test\n  namespace: ml-jobs\nspec:\n  backoffLimit: 2\n  template:\n    spec:\n      containers:\n      - name: train\n        image: registry.example.com\u002Fml-training\u002Firis:v1\n        env:\n        - name: MLFLOW_TRACKING_URI\n          value: \"http:\u002F\u002Fmlflow.mlflow.svc.cluster.local:5000\"\n        resources:\n          requests:\n            memory: \"512Mi\"\n            cpu: \"500m\"\n          limits:\n            memory: \"1Gi\"\n            cpu: \"1\"\n      restartPolicy: Never\n","yaml",[277],"language-yaml",[279],{"type":15,"tag":98,"props":280,"children":281},{"__ignoreMap":7},[282],{"type":21,"value":274},{"type":15,"tag":128,"props":284,"children":287},{"code":285,"language":234,"meta":7,"className":286},"kubectl apply -f job.yaml\nkubectl -n ml-jobs logs -f job\u002Firis-retrain-test\n",[236],[288],{"type":15,"tag":98,"props":289,"children":290},{"__ignoreMap":7},[291],{"type":21,"value":285},{"type":15,"tag":23,"props":293,"children":294},{},[295],{"type":21,"value":296},"A few things to note:",{"type":15,"tag":81,"props":298,"children":299},{},[300,314,328],{"type":15,"tag":38,"props":301,"children":302},{},[303,312],{"type":15,"tag":151,"props":304,"children":305},{},[306],{"type":15,"tag":98,"props":307,"children":309},{"className":308},[],[310],{"type":21,"value":311},"backoffLimit: 2",{"type":21,"value":313}," means Kubernetes will retry twice on failure, then mark the Job as failed. Without this, it retries indefinitely.",{"type":15,"tag":38,"props":315,"children":316},{},[317,326],{"type":15,"tag":151,"props":318,"children":319},{},[320],{"type":15,"tag":98,"props":321,"children":323},{"className":322},[],[324],{"type":21,"value":325},"restartPolicy: Never",{"type":21,"value":327}," is required for Jobs. Kubernetes manages retries at the Job level, not the Pod level.",{"type":15,"tag":38,"props":329,"children":330},{},[331,336],{"type":15,"tag":151,"props":332,"children":333},{},[334],{"type":21,"value":335},"Resource requests and limits",{"type":21,"value":337}," prevent the training job from consuming the entire node. Set these based on your actual training requirements.",{"type":15,"tag":69,"props":339,"children":341},{"id":340},"step-4-handle-secrets",[342],{"type":21,"value":343},"Step 4: Handle secrets",{"type":15,"tag":23,"props":345,"children":346},{},[347],{"type":21,"value":348},"Your training script probably needs credentials — for the model registry, the data source, or both. Don't put these in the YAML file. Use Kubernetes Secrets.",{"type":15,"tag":23,"props":350,"children":351},{},[352],{"type":21,"value":353},"Create the secret:",{"type":15,"tag":128,"props":355,"children":358},{"code":356,"language":234,"meta":7,"className":357},"kubectl -n ml-jobs create secret generic mlflow-credentials \\\n  --from-literal=MLFLOW_TRACKING_URI=http:\u002F\u002Fmlflow.mlflow.svc.cluster.local:5000 \\\n  --from-literal=AWS_ACCESS_KEY_ID=your-key \\\n  --from-literal=AWS_SECRET_ACCESS_KEY=your-secret\n",[236],[359],{"type":15,"tag":98,"props":360,"children":361},{"__ignoreMap":7},[362],{"type":21,"value":356},{"type":15,"tag":23,"props":364,"children":365},{},[366],{"type":21,"value":367},"Reference it in the Job:",{"type":15,"tag":128,"props":369,"children":372},{"code":370,"language":275,"meta":7,"className":371},"containers:\n- name: train\n  image: registry.example.com\u002Fml-training\u002Firis:v1\n  envFrom:\n  - secretRef:\n      name: mlflow-credentials\n  resources:\n    requests:\n      memory: \"512Mi\"\n      cpu: \"500m\"\n    limits:\n      memory: \"1Gi\"\n      cpu: \"1\"\n",[277],[373],{"type":15,"tag":98,"props":374,"children":375},{"__ignoreMap":7},[376],{"type":21,"value":370},{"type":15,"tag":23,"props":378,"children":379},{},[380,386,388,394],{"type":15,"tag":98,"props":381,"children":383},{"className":382},[],[384],{"type":21,"value":385},"envFrom",{"type":21,"value":387}," injects every key in the Secret as an environment variable. The training script reads them via ",{"type":15,"tag":98,"props":389,"children":391},{"className":390},[],[392],{"type":21,"value":393},"os.environ",{"type":21,"value":395}," without knowing they came from a Secret.",{"type":15,"tag":69,"props":397,"children":399},{"id":398},"step-5-set-up-the-cronjob",[400],{"type":21,"value":401},"Step 5: Set up the CronJob",{"type":15,"tag":23,"props":403,"children":404},{},[405],{"type":21,"value":406},"Once the one-off Job works, wrap it in a CronJob:",{"type":15,"tag":128,"props":408,"children":411},{"code":409,"language":275,"meta":7,"className":410},"apiVersion: batch\u002Fv1\nkind: CronJob\nmetadata:\n  name: iris-retrain\n  namespace: ml-jobs\nspec:\n  schedule: \"0 3 * * 0\"\n  concurrencyPolicy: Forbid\n  successfulJobsHistoryLimit: 3\n  failedJobsHistoryLimit: 3\n  jobTemplate:\n    spec:\n      backoffLimit: 2\n      activeDeadlineSeconds: 3600\n      template:\n        spec:\n          containers:\n          - name: train\n            image: registry.example.com\u002Fml-training\u002Firis:v1\n            envFrom:\n            - secretRef:\n                name: mlflow-credentials\n            resources:\n              requests:\n                memory: \"512Mi\"\n                cpu: \"500m\"\n              limits:\n                memory: \"1Gi\"\n                cpu: \"1\"\n          restartPolicy: Never\n",[277],[412],{"type":15,"tag":98,"props":413,"children":414},{"__ignoreMap":7},[415],{"type":21,"value":409},{"type":15,"tag":23,"props":417,"children":418},{},[419],{"type":21,"value":420},"Breaking down the CronJob-specific fields:",{"type":15,"tag":81,"props":422,"children":423},{},[424,438,452,466,480],{"type":15,"tag":38,"props":425,"children":426},{},[427,436],{"type":15,"tag":151,"props":428,"children":429},{},[430],{"type":15,"tag":98,"props":431,"children":433},{"className":432},[],[434],{"type":21,"value":435},"schedule: \"0 3 * * 0\"",{"type":21,"value":437}," — runs every Sunday at 3 AM. Standard cron syntax. Pick a time when the cluster is underutilized.",{"type":15,"tag":38,"props":439,"children":440},{},[441,450],{"type":15,"tag":151,"props":442,"children":443},{},[444],{"type":15,"tag":98,"props":445,"children":447},{"className":446},[],[448],{"type":21,"value":449},"concurrencyPolicy: Forbid",{"type":21,"value":451}," — if the previous run is still going when the next one is scheduled, skip it. This prevents pile-ups when training takes longer than expected.",{"type":15,"tag":38,"props":453,"children":454},{},[455,464],{"type":15,"tag":151,"props":456,"children":457},{},[458],{"type":15,"tag":98,"props":459,"children":461},{"className":460},[],[462],{"type":21,"value":463},"successfulJobsHistoryLimit: 3",{"type":21,"value":465}," — keep the last 3 completed Job objects around for debugging. Older ones are garbage collected.",{"type":15,"tag":38,"props":467,"children":468},{},[469,478],{"type":15,"tag":151,"props":470,"children":471},{},[472],{"type":15,"tag":98,"props":473,"children":475},{"className":474},[],[476],{"type":21,"value":477},"failedJobsHistoryLimit: 3",{"type":21,"value":479}," — same for failed Jobs.",{"type":15,"tag":38,"props":481,"children":482},{},[483,492],{"type":15,"tag":151,"props":484,"children":485},{},[486],{"type":15,"tag":98,"props":487,"children":489},{"className":488},[],[490],{"type":21,"value":491},"activeDeadlineSeconds: 3600",{"type":21,"value":493}," — kill the Job if it runs longer than an hour. This is your safety net against hanging training runs.",{"type":15,"tag":23,"props":495,"children":496},{},[497],{"type":21,"value":498},"Apply it:",{"type":15,"tag":128,"props":500,"children":503},{"code":501,"language":234,"meta":7,"className":502},"kubectl apply -f cronjob.yaml\n",[236],[504],{"type":15,"tag":98,"props":505,"children":506},{"__ignoreMap":7},[507],{"type":21,"value":501},{"type":15,"tag":23,"props":509,"children":510},{},[511],{"type":21,"value":512},"Verify it's scheduled:",{"type":15,"tag":128,"props":514,"children":517},{"code":515,"language":234,"meta":7,"className":516},"kubectl -n ml-jobs get cronjob iris-retrain\n",[236],[518],{"type":15,"tag":98,"props":519,"children":520},{"__ignoreMap":7},[521],{"type":21,"value":515},{"type":15,"tag":23,"props":523,"children":524},{},[525],{"type":21,"value":526},"You should see the schedule and the time of the next run.",{"type":15,"tag":69,"props":528,"children":530},{"id":529},"step-6-know-when-it-fails",[531],{"type":21,"value":532},"Step 6: Know when it fails",{"type":15,"tag":23,"props":534,"children":535},{},[536],{"type":21,"value":537},"A CronJob that fails silently is worse than no CronJob at all — you think retraining is happening when it isn't. You need alerting.",{"type":15,"tag":23,"props":539,"children":540},{},[541],{"type":21,"value":542},"The simplest approach is a Kubernetes Event watcher. But for most teams, the practical answer is to check Job status from your existing monitoring stack.",{"type":15,"tag":23,"props":544,"children":545},{},[546,548,554],{"type":21,"value":547},"If you're running Prometheus (and on Kubernetes, you probably are), the ",{"type":15,"tag":98,"props":549,"children":551},{"className":550},[],[552],{"type":21,"value":553},"kube-state-metrics",{"type":21,"value":555}," exporter already exposes Job metrics:",{"type":15,"tag":81,"props":557,"children":558},{},[559,570],{"type":15,"tag":38,"props":560,"children":561},{},[562,568],{"type":15,"tag":98,"props":563,"children":565},{"className":564},[],[566],{"type":21,"value":567},"kube_job_status_failed",{"type":21,"value":569}," — number of failed Jobs",{"type":15,"tag":38,"props":571,"children":572},{},[573,579],{"type":15,"tag":98,"props":574,"children":576},{"className":575},[],[577],{"type":21,"value":578},"kube_job_status_succeeded",{"type":21,"value":580}," — number of succeeded Jobs",{"type":15,"tag":23,"props":582,"children":583},{},[584],{"type":21,"value":585},"A basic Prometheus alert rule:",{"type":15,"tag":128,"props":587,"children":590},{"code":588,"language":275,"meta":7,"className":589},"groups:\n- name: ml-retraining\n  rules:\n  - alert: RetrainingJobFailed\n    expr: |\n      kube_job_status_failed{namespace=\"ml-jobs\", job_name=~\"iris-retrain.*\"} > 0\n    for: 5m\n    labels:\n      severity: warning\n    annotations:\n      summary: \"ML retraining job failed\"\n      description: \"The iris-retrain CronJob has a failed run. Check logs with: kubectl -n ml-jobs logs job\u002F{{ $labels.job_name }}\"\n",[277],[591],{"type":15,"tag":98,"props":592,"children":593},{"__ignoreMap":7},[594],{"type":21,"value":588},{"type":15,"tag":23,"props":596,"children":597},{},[598],{"type":21,"value":599},"This fires an alert if any retraining Job has been in a failed state for more than 5 minutes. Route it to Slack, PagerDuty, or email — wherever your team already gets alerts.",{"type":15,"tag":601,"props":602,"children":604},"h3",{"id":603},"quick-check-script",[605],{"type":21,"value":606},"Quick check script",{"type":15,"tag":23,"props":608,"children":609},{},[610],{"type":21,"value":611},"For teams without Prometheus, a simple script that runs on its own cron schedule works:",{"type":15,"tag":128,"props":613,"children":616},{"code":614,"language":234,"meta":7,"className":615},"#!\u002Fbin\u002Fbash\nFAILED=$(kubectl -n ml-jobs get jobs \\\n  -l app=iris-retrain \\\n  --field-selector=status.successful=0 \\\n  -o name 2>\u002Fdev\u002Fnull | wc -l)\n\nif [ \"$FAILED\" -gt 0 ]; then\n  echo \"WARNING: $FAILED failed retraining jobs\" | \\\n    mail -s \"Retraining failure\" team@example.com\nfi\n",[236],[617],{"type":15,"tag":98,"props":618,"children":619},{"__ignoreMap":7},[620],{"type":21,"value":614},{"type":15,"tag":23,"props":622,"children":623},{},[624],{"type":21,"value":625},"Not elegant, but it works and it's better than silence.",{"type":15,"tag":69,"props":627,"children":629},{"id":628},"what-this-looks-like-day-to-day",[630],{"type":21,"value":631},"What this looks like day-to-day",{"type":15,"tag":23,"props":633,"children":634},{},[635],{"type":21,"value":636},"Once the CronJob is running, the day-to-day workflow changes:",{"type":15,"tag":81,"props":638,"children":639},{},[640,650,668,678],{"type":15,"tag":38,"props":641,"children":642},{},[643,648],{"type":15,"tag":151,"props":644,"children":645},{},[646],{"type":21,"value":647},"No more manual retraining.",{"type":21,"value":649}," The CronJob runs on schedule. New models appear in MLflow.",{"type":15,"tag":38,"props":651,"children":652},{},[653,658,660,666],{"type":15,"tag":151,"props":654,"children":655},{},[656],{"type":21,"value":657},"Debugging is straightforward.",{"type":21,"value":659}," If a run fails, check the Job logs: ",{"type":15,"tag":98,"props":661,"children":663},{"className":662},[],[664],{"type":21,"value":665},"kubectl -n ml-jobs logs job\u002F\u003Cjob-name>",{"type":21,"value":667},". The logs are the same output you'd see running locally.",{"type":15,"tag":38,"props":669,"children":670},{},[671,676],{"type":15,"tag":151,"props":672,"children":673},{},[674],{"type":21,"value":675},"Updating the training code",{"type":21,"value":677}," means building a new container image and updating the CronJob's image reference. The schedule and infrastructure stay the same.",{"type":15,"tag":38,"props":679,"children":680},{},[681,686],{"type":15,"tag":151,"props":682,"children":683},{},[684],{"type":21,"value":685},"Adjusting the schedule",{"type":21,"value":687}," is a one-line YAML change.",{"type":15,"tag":23,"props":689,"children":690},{},[691],{"type":21,"value":692},"The main thing you'll want to add next is automated evaluation — comparing the new model against the current production model before promoting it. But that's a separate concern. The CronJob just ensures training happens reliably.",{"type":15,"tag":69,"props":694,"children":696},{"id":695},"common-gotchas",[697],{"type":21,"value":698},"Common gotchas",{"type":15,"tag":23,"props":700,"children":701},{},[702,707,709,715],{"type":15,"tag":151,"props":703,"children":704},{},[705],{"type":21,"value":706},"Image pull failures.",{"type":21,"value":708}," If your container registry requires authentication, set up an ",{"type":15,"tag":98,"props":710,"children":712},{"className":711},[],[713],{"type":21,"value":714},"imagePullSecret",{"type":21,"value":716}," on the namespace. Silent image pull failures are a common source of confusion.",{"type":15,"tag":23,"props":718,"children":719},{},[720,725,727,732,734,740],{"type":15,"tag":151,"props":721,"children":722},{},[723],{"type":21,"value":724},"Timezone.",{"type":21,"value":726}," Kubernetes CronJobs use UTC by default. If you set ",{"type":15,"tag":98,"props":728,"children":730},{"className":729},[],[731],{"type":21,"value":435},{"type":21,"value":733}," expecting 3 AM local time, it might run at a different hour. Kubernetes 1.27+ supports ",{"type":15,"tag":98,"props":735,"children":737},{"className":736},[],[738],{"type":21,"value":739},"timeZone",{"type":21,"value":741}," on CronJobs if this matters.",{"type":15,"tag":23,"props":743,"children":744},{},[745,750,752,758],{"type":15,"tag":151,"props":746,"children":747},{},[748],{"type":21,"value":749},"Resource starvation.",{"type":21,"value":751}," If the cluster doesn't have enough resources to schedule the training Pod, it'll sit in Pending state until the ",{"type":15,"tag":98,"props":753,"children":755},{"className":754},[],[756],{"type":21,"value":757},"activeDeadlineSeconds",{"type":21,"value":759}," kills it. Set resource requests realistically and monitor Pod scheduling.",{"type":15,"tag":23,"props":761,"children":762},{},[763,768,770,776,778,784,786,792,794,800],{"type":15,"tag":151,"props":764,"children":765},{},[766],{"type":21,"value":767},"Stale images.",{"type":21,"value":769}," Using ",{"type":15,"tag":98,"props":771,"children":773},{"className":772},[],[774],{"type":21,"value":775},":latest",{"type":21,"value":777}," tags means you're never sure which version of the training code is running. Use explicit version tags (",{"type":15,"tag":98,"props":779,"children":781},{"className":780},[],[782],{"type":21,"value":783},":v1",{"type":21,"value":785},", ",{"type":15,"tag":98,"props":787,"children":789},{"className":788},[],[790],{"type":21,"value":791},":v2",{"type":21,"value":793},") or content-addressable tags (",{"type":15,"tag":98,"props":795,"children":797},{"className":796},[],[798],{"type":21,"value":799},":sha-abc123",{"type":21,"value":801},").",{"type":15,"tag":803,"props":804,"children":805},"hr",{},[],{"type":15,"tag":23,"props":807,"children":808},{},[809,811,820,822,828],{"type":21,"value":810},"Kubernetes CronJobs are the foundation of automated ML pipelines. If you need help setting up reliable retraining infrastructure, ",{"type":15,"tag":812,"props":813,"children":817},"a",{"href":814,"rel":815},"https:\u002F\u002Fdeploying.ai",[816],"nofollow",[818],{"type":21,"value":819},"deploying.ai",{"type":21,"value":821}," can help — ",{"type":15,"tag":812,"props":823,"children":825},{"href":824},"mailto:vlad@deploying.ai",[826],{"type":21,"value":827},"reach out",{"type":21,"value":829},".",{"title":7,"searchDepth":831,"depth":831,"links":832},2,[833,834,835,836,837,838,839,843,844],{"id":71,"depth":831,"text":74},{"id":113,"depth":831,"text":116},{"id":185,"depth":831,"text":188},{"id":263,"depth":831,"text":266},{"id":340,"depth":831,"text":343},{"id":398,"depth":831,"text":401},{"id":529,"depth":831,"text":532,"children":840},[841],{"id":603,"depth":842,"text":606},3,{"id":628,"depth":831,"text":631},{"id":695,"depth":831,"text":698},"markdown","content:articles:kubernetes-cronjob-model-retraining.md","content","articles\u002Fkubernetes-cronjob-model-retraining.md","articles\u002Fkubernetes-cronjob-model-retraining","md",{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"body":852,"_type":845,"_id":846,"_source":847,"_file":848,"_stem":849,"_extension":850},{"type":12,"children":853,"toc":1496},[854,858,862,866,889,893,897,901,925,929,933,937,945,949,976,980,984,988,996,1006,1013,1017,1025,1029,1037,1041,1045,1049,1057,1065,1069,1104,1108,1112,1116,1124,1128,1136,1151,1155,1159,1167,1171,1234,1238,1246,1250,1258,1262,1266,1270,1274,1284,1305,1309,1317,1321,1325,1329,1337,1341,1345,1349,1390,1394,1398,1412,1432,1446,1478,1481],{"type":15,"tag":16,"props":855,"children":856},{"id":18},[857],{"type":21,"value":8},{"type":15,"tag":23,"props":859,"children":860},{},[861],{"type":21,"value":27},{"type":15,"tag":23,"props":863,"children":864},{},[865],{"type":21,"value":32},{"type":15,"tag":34,"props":867,"children":868},{},[869,873,877,881,885],{"type":15,"tag":38,"props":870,"children":871},{},[872],{"type":21,"value":42},{"type":15,"tag":38,"props":874,"children":875},{},[876],{"type":21,"value":47},{"type":15,"tag":38,"props":878,"children":879},{},[880],{"type":21,"value":52},{"type":15,"tag":38,"props":882,"children":883},{},[884],{"type":21,"value":57},{"type":15,"tag":38,"props":886,"children":887},{},[888],{"type":21,"value":62},{"type":15,"tag":23,"props":890,"children":891},{},[892],{"type":21,"value":67},{"type":15,"tag":69,"props":894,"children":895},{"id":71},[896],{"type":21,"value":74},{"type":15,"tag":23,"props":898,"children":899},{},[900],{"type":21,"value":79},{"type":15,"tag":81,"props":902,"children":903},{},[904,908,912,921],{"type":15,"tag":38,"props":905,"children":906},{},[907],{"type":21,"value":88},{"type":15,"tag":38,"props":909,"children":910},{},[911],{"type":21,"value":93},{"type":15,"tag":38,"props":913,"children":914},{},[915,920],{"type":15,"tag":98,"props":916,"children":918},{"className":917},[],[919],{"type":21,"value":103},{"type":21,"value":105},{"type":15,"tag":38,"props":922,"children":923},{},[924],{"type":21,"value":110},{"type":15,"tag":69,"props":926,"children":927},{"id":113},[928],{"type":21,"value":116},{"type":15,"tag":23,"props":930,"children":931},{},[932],{"type":21,"value":121},{"type":15,"tag":23,"props":934,"children":935},{},[936],{"type":21,"value":126},{"type":15,"tag":128,"props":938,"children":940},{"code":130,"language":131,"meta":7,"className":939},[133],[941],{"type":15,"tag":98,"props":942,"children":943},{"__ignoreMap":7},[944],{"type":21,"value":130},{"type":15,"tag":23,"props":946,"children":947},{},[948],{"type":21,"value":143},{"type":15,"tag":81,"props":950,"children":951},{},[952,960,968],{"type":15,"tag":38,"props":953,"children":954},{},[955,959],{"type":15,"tag":151,"props":956,"children":957},{},[958],{"type":21,"value":155},{"type":21,"value":157},{"type":15,"tag":38,"props":961,"children":962},{},[963,967],{"type":15,"tag":151,"props":964,"children":965},{},[966],{"type":21,"value":165},{"type":21,"value":167},{"type":15,"tag":38,"props":969,"children":970},{},[971,975],{"type":15,"tag":151,"props":972,"children":973},{},[974],{"type":21,"value":175},{"type":21,"value":177},{"type":15,"tag":23,"props":977,"children":978},{},[979],{"type":21,"value":182},{"type":15,"tag":69,"props":981,"children":982},{"id":185},[983],{"type":21,"value":188},{"type":15,"tag":23,"props":985,"children":986},{},[987],{"type":21,"value":193},{"type":15,"tag":128,"props":989,"children":991},{"code":196,"language":197,"meta":7,"className":990},[199],[992],{"type":15,"tag":98,"props":993,"children":994},{"__ignoreMap":7},[995],{"type":21,"value":196},{"type":15,"tag":23,"props":997,"children":998},{},[999,1000,1005],{"type":21,"value":209},{"type":15,"tag":98,"props":1001,"children":1003},{"className":1002},[],[1004],{"type":21,"value":215},{"type":21,"value":217},{"type":15,"tag":128,"props":1007,"children":1008},{"code":220},[1009],{"type":15,"tag":98,"props":1010,"children":1011},{"__ignoreMap":7},[1012],{"type":21,"value":220},{"type":15,"tag":23,"props":1014,"children":1015},{},[1016],{"type":21,"value":230},{"type":15,"tag":128,"props":1018,"children":1020},{"code":233,"language":234,"meta":7,"className":1019},[236],[1021],{"type":15,"tag":98,"props":1022,"children":1023},{"__ignoreMap":7},[1024],{"type":21,"value":233},{"type":15,"tag":23,"props":1026,"children":1027},{},[1028],{"type":21,"value":246},{"type":15,"tag":128,"props":1030,"children":1032},{"code":249,"language":234,"meta":7,"className":1031},[236],[1033],{"type":15,"tag":98,"props":1034,"children":1035},{"__ignoreMap":7},[1036],{"type":21,"value":249},{"type":15,"tag":23,"props":1038,"children":1039},{},[1040],{"type":21,"value":260},{"type":15,"tag":69,"props":1042,"children":1043},{"id":263},[1044],{"type":21,"value":266},{"type":15,"tag":23,"props":1046,"children":1047},{},[1048],{"type":21,"value":271},{"type":15,"tag":128,"props":1050,"children":1052},{"code":274,"language":275,"meta":7,"className":1051},[277],[1053],{"type":15,"tag":98,"props":1054,"children":1055},{"__ignoreMap":7},[1056],{"type":21,"value":274},{"type":15,"tag":128,"props":1058,"children":1060},{"code":285,"language":234,"meta":7,"className":1059},[236],[1061],{"type":15,"tag":98,"props":1062,"children":1063},{"__ignoreMap":7},[1064],{"type":21,"value":285},{"type":15,"tag":23,"props":1066,"children":1067},{},[1068],{"type":21,"value":296},{"type":15,"tag":81,"props":1070,"children":1071},{},[1072,1084,1096],{"type":15,"tag":38,"props":1073,"children":1074},{},[1075,1083],{"type":15,"tag":151,"props":1076,"children":1077},{},[1078],{"type":15,"tag":98,"props":1079,"children":1081},{"className":1080},[],[1082],{"type":21,"value":311},{"type":21,"value":313},{"type":15,"tag":38,"props":1085,"children":1086},{},[1087,1095],{"type":15,"tag":151,"props":1088,"children":1089},{},[1090],{"type":15,"tag":98,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":21,"value":325},{"type":21,"value":327},{"type":15,"tag":38,"props":1097,"children":1098},{},[1099,1103],{"type":15,"tag":151,"props":1100,"children":1101},{},[1102],{"type":21,"value":335},{"type":21,"value":337},{"type":15,"tag":69,"props":1105,"children":1106},{"id":340},[1107],{"type":21,"value":343},{"type":15,"tag":23,"props":1109,"children":1110},{},[1111],{"type":21,"value":348},{"type":15,"tag":23,"props":1113,"children":1114},{},[1115],{"type":21,"value":353},{"type":15,"tag":128,"props":1117,"children":1119},{"code":356,"language":234,"meta":7,"className":1118},[236],[1120],{"type":15,"tag":98,"props":1121,"children":1122},{"__ignoreMap":7},[1123],{"type":21,"value":356},{"type":15,"tag":23,"props":1125,"children":1126},{},[1127],{"type":21,"value":367},{"type":15,"tag":128,"props":1129,"children":1131},{"code":370,"language":275,"meta":7,"className":1130},[277],[1132],{"type":15,"tag":98,"props":1133,"children":1134},{"__ignoreMap":7},[1135],{"type":21,"value":370},{"type":15,"tag":23,"props":1137,"children":1138},{},[1139,1144,1145,1150],{"type":15,"tag":98,"props":1140,"children":1142},{"className":1141},[],[1143],{"type":21,"value":385},{"type":21,"value":387},{"type":15,"tag":98,"props":1146,"children":1148},{"className":1147},[],[1149],{"type":21,"value":393},{"type":21,"value":395},{"type":15,"tag":69,"props":1152,"children":1153},{"id":398},[1154],{"type":21,"value":401},{"type":15,"tag":23,"props":1156,"children":1157},{},[1158],{"type":21,"value":406},{"type":15,"tag":128,"props":1160,"children":1162},{"code":409,"language":275,"meta":7,"className":1161},[277],[1163],{"type":15,"tag":98,"props":1164,"children":1165},{"__ignoreMap":7},[1166],{"type":21,"value":409},{"type":15,"tag":23,"props":1168,"children":1169},{},[1170],{"type":21,"value":420},{"type":15,"tag":81,"props":1172,"children":1173},{},[1174,1186,1198,1210,1222],{"type":15,"tag":38,"props":1175,"children":1176},{},[1177,1185],{"type":15,"tag":151,"props":1178,"children":1179},{},[1180],{"type":15,"tag":98,"props":1181,"children":1183},{"className":1182},[],[1184],{"type":21,"value":435},{"type":21,"value":437},{"type":15,"tag":38,"props":1187,"children":1188},{},[1189,1197],{"type":15,"tag":151,"props":1190,"children":1191},{},[1192],{"type":15,"tag":98,"props":1193,"children":1195},{"className":1194},[],[1196],{"type":21,"value":449},{"type":21,"value":451},{"type":15,"tag":38,"props":1199,"children":1200},{},[1201,1209],{"type":15,"tag":151,"props":1202,"children":1203},{},[1204],{"type":15,"tag":98,"props":1205,"children":1207},{"className":1206},[],[1208],{"type":21,"value":463},{"type":21,"value":465},{"type":15,"tag":38,"props":1211,"children":1212},{},[1213,1221],{"type":15,"tag":151,"props":1214,"children":1215},{},[1216],{"type":15,"tag":98,"props":1217,"children":1219},{"className":1218},[],[1220],{"type":21,"value":477},{"type":21,"value":479},{"type":15,"tag":38,"props":1223,"children":1224},{},[1225,1233],{"type":15,"tag":151,"props":1226,"children":1227},{},[1228],{"type":15,"tag":98,"props":1229,"children":1231},{"className":1230},[],[1232],{"type":21,"value":491},{"type":21,"value":493},{"type":15,"tag":23,"props":1235,"children":1236},{},[1237],{"type":21,"value":498},{"type":15,"tag":128,"props":1239,"children":1241},{"code":501,"language":234,"meta":7,"className":1240},[236],[1242],{"type":15,"tag":98,"props":1243,"children":1244},{"__ignoreMap":7},[1245],{"type":21,"value":501},{"type":15,"tag":23,"props":1247,"children":1248},{},[1249],{"type":21,"value":512},{"type":15,"tag":128,"props":1251,"children":1253},{"code":515,"language":234,"meta":7,"className":1252},[236],[1254],{"type":15,"tag":98,"props":1255,"children":1256},{"__ignoreMap":7},[1257],{"type":21,"value":515},{"type":15,"tag":23,"props":1259,"children":1260},{},[1261],{"type":21,"value":526},{"type":15,"tag":69,"props":1263,"children":1264},{"id":529},[1265],{"type":21,"value":532},{"type":15,"tag":23,"props":1267,"children":1268},{},[1269],{"type":21,"value":537},{"type":15,"tag":23,"props":1271,"children":1272},{},[1273],{"type":21,"value":542},{"type":15,"tag":23,"props":1275,"children":1276},{},[1277,1278,1283],{"type":21,"value":547},{"type":15,"tag":98,"props":1279,"children":1281},{"className":1280},[],[1282],{"type":21,"value":553},{"type":21,"value":555},{"type":15,"tag":81,"props":1285,"children":1286},{},[1287,1296],{"type":15,"tag":38,"props":1288,"children":1289},{},[1290,1295],{"type":15,"tag":98,"props":1291,"children":1293},{"className":1292},[],[1294],{"type":21,"value":567},{"type":21,"value":569},{"type":15,"tag":38,"props":1297,"children":1298},{},[1299,1304],{"type":15,"tag":98,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":21,"value":578},{"type":21,"value":580},{"type":15,"tag":23,"props":1306,"children":1307},{},[1308],{"type":21,"value":585},{"type":15,"tag":128,"props":1310,"children":1312},{"code":588,"language":275,"meta":7,"className":1311},[277],[1313],{"type":15,"tag":98,"props":1314,"children":1315},{"__ignoreMap":7},[1316],{"type":21,"value":588},{"type":15,"tag":23,"props":1318,"children":1319},{},[1320],{"type":21,"value":599},{"type":15,"tag":601,"props":1322,"children":1323},{"id":603},[1324],{"type":21,"value":606},{"type":15,"tag":23,"props":1326,"children":1327},{},[1328],{"type":21,"value":611},{"type":15,"tag":128,"props":1330,"children":1332},{"code":614,"language":234,"meta":7,"className":1331},[236],[1333],{"type":15,"tag":98,"props":1334,"children":1335},{"__ignoreMap":7},[1336],{"type":21,"value":614},{"type":15,"tag":23,"props":1338,"children":1339},{},[1340],{"type":21,"value":625},{"type":15,"tag":69,"props":1342,"children":1343},{"id":628},[1344],{"type":21,"value":631},{"type":15,"tag":23,"props":1346,"children":1347},{},[1348],{"type":21,"value":636},{"type":15,"tag":81,"props":1350,"children":1351},{},[1352,1360,1374,1382],{"type":15,"tag":38,"props":1353,"children":1354},{},[1355,1359],{"type":15,"tag":151,"props":1356,"children":1357},{},[1358],{"type":21,"value":647},{"type":21,"value":649},{"type":15,"tag":38,"props":1361,"children":1362},{},[1363,1367,1368,1373],{"type":15,"tag":151,"props":1364,"children":1365},{},[1366],{"type":21,"value":657},{"type":21,"value":659},{"type":15,"tag":98,"props":1369,"children":1371},{"className":1370},[],[1372],{"type":21,"value":665},{"type":21,"value":667},{"type":15,"tag":38,"props":1375,"children":1376},{},[1377,1381],{"type":15,"tag":151,"props":1378,"children":1379},{},[1380],{"type":21,"value":675},{"type":21,"value":677},{"type":15,"tag":38,"props":1383,"children":1384},{},[1385,1389],{"type":15,"tag":151,"props":1386,"children":1387},{},[1388],{"type":21,"value":685},{"type":21,"value":687},{"type":15,"tag":23,"props":1391,"children":1392},{},[1393],{"type":21,"value":692},{"type":15,"tag":69,"props":1395,"children":1396},{"id":695},[1397],{"type":21,"value":698},{"type":15,"tag":23,"props":1399,"children":1400},{},[1401,1405,1406,1411],{"type":15,"tag":151,"props":1402,"children":1403},{},[1404],{"type":21,"value":706},{"type":21,"value":708},{"type":15,"tag":98,"props":1407,"children":1409},{"className":1408},[],[1410],{"type":21,"value":714},{"type":21,"value":716},{"type":15,"tag":23,"props":1413,"children":1414},{},[1415,1419,1420,1425,1426,1431],{"type":15,"tag":151,"props":1416,"children":1417},{},[1418],{"type":21,"value":724},{"type":21,"value":726},{"type":15,"tag":98,"props":1421,"children":1423},{"className":1422},[],[1424],{"type":21,"value":435},{"type":21,"value":733},{"type":15,"tag":98,"props":1427,"children":1429},{"className":1428},[],[1430],{"type":21,"value":739},{"type":21,"value":741},{"type":15,"tag":23,"props":1433,"children":1434},{},[1435,1439,1440,1445],{"type":15,"tag":151,"props":1436,"children":1437},{},[1438],{"type":21,"value":749},{"type":21,"value":751},{"type":15,"tag":98,"props":1441,"children":1443},{"className":1442},[],[1444],{"type":21,"value":757},{"type":21,"value":759},{"type":15,"tag":23,"props":1447,"children":1448},{},[1449,1453,1454,1459,1460,1465,1466,1471,1472,1477],{"type":15,"tag":151,"props":1450,"children":1451},{},[1452],{"type":21,"value":767},{"type":21,"value":769},{"type":15,"tag":98,"props":1455,"children":1457},{"className":1456},[],[1458],{"type":21,"value":775},{"type":21,"value":777},{"type":15,"tag":98,"props":1461,"children":1463},{"className":1462},[],[1464],{"type":21,"value":783},{"type":21,"value":785},{"type":15,"tag":98,"props":1467,"children":1469},{"className":1468},[],[1470],{"type":21,"value":791},{"type":21,"value":793},{"type":15,"tag":98,"props":1473,"children":1475},{"className":1474},[],[1476],{"type":21,"value":799},{"type":21,"value":801},{"type":15,"tag":803,"props":1479,"children":1480},{},[],{"type":15,"tag":23,"props":1482,"children":1483},{},[1484,1485,1490,1491,1495],{"type":21,"value":810},{"type":15,"tag":812,"props":1486,"children":1488},{"href":814,"rel":1487},[816],[1489],{"type":21,"value":819},{"type":21,"value":821},{"type":15,"tag":812,"props":1492,"children":1493},{"href":824},[1494],{"type":21,"value":827},{"type":21,"value":829},{"title":7,"searchDepth":831,"depth":831,"links":1497},[1498,1499,1500,1501,1502,1503,1504,1507,1508],{"id":71,"depth":831,"text":74},{"id":113,"depth":831,"text":116},{"id":185,"depth":831,"text":188},{"id":263,"depth":831,"text":266},{"id":340,"depth":831,"text":343},{"id":398,"depth":831,"text":401},{"id":529,"depth":831,"text":532,"children":1505},[1506],{"id":603,"depth":842,"text":606},{"id":628,"depth":831,"text":631},{"id":695,"depth":831,"text":698},1785232227842]