End-to-End ML Model Walkthrough: From model.fit() to Production

Most machine learning tutorials end at model.fit(). You train a model, print the accuracy, and the tutorial is done. But that's maybe 10% of the real work.

What happens next? How does the model get into production? How do you retrain it when the data changes? How do you know if the new version is better than the old one? These are the questions that separate a notebook experiment from a working ML system.

This article walks through the full lifecycle of a deliberately simple model. The model itself is trivial — that's the point. The infrastructure around it is what matters.

The toy model

We'll use a scikit-learn classifier on the Iris dataset. Three classes, four features, 150 samples. You can't get simpler than this.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = RandomForestClassifier(n_estimators=100, max_depth=5)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")

This is where most tutorials stop. You have a model. It works. Now what?

The first training run

Before adding any infrastructure, let's think about what this training script actually produces and what questions come up the moment you try to use the result.

The output is a fitted model object sitting in memory. If you want to keep it, you pickle.dump() it to a file. And immediately you have problems:

  • Where does the file go? A local directory? A shared drive? An object store?
  • What do you name it? model.pkl? model_v2_final_FINAL.pkl?
  • What produced it? Which version of the code? What hyperparameters? What data?
  • Is it any good? What was the accuracy? On what test set?

You can solve each of these with discipline and naming conventions. Or you can use a tool that was built for exactly this.

MLflow enters the picture

MLflow is an experiment tracker. At its core, it answers: "what did I run, with what settings, and what happened?" Here's the same training script with MLflow:

import mlflow
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

mlflow.set_tracking_uri("http://mlflow.internal:5000")
mlflow.set_experiment("iris-classifier")

with mlflow.start_run():
    X, y = load_iris(return_X_y=True)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )

    n_estimators = 100
    max_depth = 5

    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_param("max_depth", max_depth)
    mlflow.log_param("test_size", 0.2)

    model = RandomForestClassifier(
        n_estimators=n_estimators, max_depth=max_depth
    )
    model.fit(X_train, y_train)

    predictions = model.predict(X_test)
    acc = accuracy_score(y_test, predictions)
    mlflow.log_metric("accuracy", acc)

    mlflow.sklearn.log_model(model, "model")

Not much more code, but now you get:

  • Experiment history. Every run is recorded with its parameters and metrics. You can compare ten different hyperparameter combinations side by side.
  • Artifact storage. The model is serialized and stored in a known location — not a random pickle file on someone's laptop.
  • Reproducibility metadata. MLflow logs the git commit, the Python environment, and the exact parameters. Six months from now, you can see exactly what produced this model.
  • A model registry. You can promote a run's model to "Staging" or "Production" in the registry, giving you a single source of truth for which model is currently live.

The difference between MLflow and a pickle file is the same as the difference between a version control system and a folder of zip files. You can survive without it, but you shouldn't.

Deploying the model

A model in a registry is still just a file. To be useful, it needs to serve predictions. There are many ways to do this — here's one straightforward approach using a Flask endpoint:

import mlflow
from flask import Flask, request, jsonify

app = Flask(__name__)

# Load the production model from the registry
model = mlflow.sklearn.load_model("models:/iris-classifier/Production")

@app.route("/predict", methods=["POST"])
def predict():
    data = request.json["features"]
    prediction = model.predict([data])
    return jsonify({"prediction": int(prediction[0])})

This is a minimal serving layer. In production, you'd want to think about:

  • Containerization. Package the serving code and its dependencies into a Docker image so it runs the same everywhere.
  • Health checks and readiness probes. Kubernetes needs to know when the service is ready to accept traffic.
  • Input validation. Don't trust the incoming data — validate shapes, types, and ranges before feeding them to the model.
  • Logging predictions. Store what the model predicted and what it received. You'll need this for monitoring and debugging.

The important architectural point: the serving layer pulls the model from the registry. It doesn't know or care how the model was trained. This decouples deployment from training — you can update the model without touching the serving code.

Why models need retraining

Deploy a model and it starts degrading. This isn't a bug — it's a fundamental property of ML systems. The world changes, and the model's training data becomes a less accurate picture of reality.

Common reasons for retraining:

  • Data drift. The distribution of incoming data shifts away from the training distribution. A fraud detection model trained on pre-pandemic transaction patterns will struggle with post-pandemic shopping behavior.
  • Concept drift. The relationship between inputs and outputs changes. What counted as spam email five years ago looks different from spam today.
  • New data availability. You've collected more labeled data since the last training run. The model should benefit from it.
  • Feature changes. Upstream data pipelines evolve — columns get added, formats change, sources get replaced.

The question isn't whether to retrain. It's how to retrain reliably, automatically, and without a human babysitting the process.

Scheduled GPU jobs with Kueue

Retraining needs compute. For deep learning models, that means GPUs. Even for our simple scikit-learn example, the pattern matters because it scales to real workloads.

On Kubernetes, the standard way to run a one-off job is a Job resource. But GPUs are expensive and scarce — you can't just let everyone submit jobs whenever they want. This is where Kueue comes in.

Kueue is a Kubernetes-native job queueing system. It manages:

  • Quotas. Team A gets 4 GPUs, Team B gets 8. No one can starve out another team.
  • Priorities. Production retraining jobs preempt experimental runs.
  • Fair sharing. Idle resources get redistributed rather than sitting unused.

A scheduled retraining job looks something like this:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: iris-retrain
  namespace: ml-jobs
spec:
  schedule: "0 2 * * 0"  # Every Sunday at 2am
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            kueue.x-k8s.io/queue-name: ml-training-queue
        spec:
          containers:
          - name: train
            image: ml-training:latest
            command: ["python", "train.py"]
            resources:
              limits:
                nvidia.com/gpu: 1
          restartPolicy: Never

The CronJob handles scheduling. Kueue handles resource allocation. The training script inside logs everything to MLflow. If the new model performs better than the current production model, it gets promoted in the registry. The serving layer picks it up.

No human intervention required.

The full loop

Zoom out and you can see the complete lifecycle:

  1. Data arrives — new samples, updated labels, corrected features.
  2. Training runs on a schedule (or is triggered by data changes). It uses GPU resources managed by Kueue.
  3. Experiment tracking via MLflow records every run's parameters, metrics, and artifacts.
  4. Evaluation compares the new model against the current production model on a held-out test set.
  5. Registration promotes the new model in the MLflow registry if it passes evaluation.
  6. Deployment picks up the new model version — either automatically or via a promotion step.
  7. Monitoring watches prediction quality, latency, and data drift in production.
  8. Retraining is triggered again when monitoring detects degradation, or simply on schedule.

Each of these steps involves infrastructure decisions. Where does the data live? How are GPUs allocated? Who can promote a model? How do you roll back? What counts as "better"?

The model itself — our little random forest — is almost an afterthought. The value is in the system that surrounds it.

What this looks like in practice

For a team getting started with MLOps, you don't need all of this on day one. A reasonable progression:

Week one: Get MLflow running. Start logging experiments instead of losing them. This alone is a major improvement — you can compare runs and reproduce results.

Next: Containerize your training script. Run it as a Kubernetes Job. You've now decoupled "where training runs" from "who runs it."

Then: Add Kueue for resource management. Set up a CronJob for scheduled retraining. Your model stays fresh without manual effort.

Finally: Build the promotion pipeline. Automated evaluation, registry promotion, serving layer updates. The full loop.

Each step is independently valuable. You don't need to build the whole thing before any of it is useful.

This is the kind of infrastructure work we help teams build at deploying.ai. If you're past the notebook stage and need your models running reliably in production, let's talk.