What MLflow Actually Does (and What It Doesn't)

MLflow comes up in every MLOps conversation. If you search for "how to manage ML experiments," it's the first tool you'll find. But the messaging is broad enough that it's hard to know what MLflow actually handles versus what you'll still need to build around it.

This article breaks it down concretely. What MLflow does well, what it does passably, and what it doesn't do at all.

The core: experiment tracking

At its heart, MLflow is a logging system for ML experiments. You call a few functions during training, and MLflow records what happened.

import mlflow

mlflow.set_experiment("fraud-detection")

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_param("epochs", 50)
    mlflow.log_param("batch_size", 128)

    # ... training happens ...

    mlflow.log_metric("auc", 0.934)
    mlflow.log_metric("precision", 0.891)
    mlflow.log_metric("recall", 0.867)

That's really it. You call log_param for inputs and log_metric for outputs. MLflow stores them in a database and gives you a UI to browse and compare runs.

This solves a very real problem. Without it, teams end up with spreadsheets, Slack messages ("hey, what learning rate did you use for that good run last Tuesday?"), or notebooks with output cells that may or may not match the code above them.

What you get from the tracking server

The MLflow UI lets you:

  • Compare runs side by side. Sort by any metric, filter by parameters, see which hyperparameter combinations actually moved the needle.
  • View run history. See every experiment chronologically. Useful when you need to answer "what changed between the model that worked and the one that didn't?"
  • Search runs programmatically. mlflow.search_runs() returns a DataFrame, so you can do your own analysis on experiment history.
import mlflow

# Find all runs with AUC > 0.9
runs = mlflow.search_runs(
    experiment_names=["fraud-detection"],
    filter_string="metrics.auc > 0.9",
    order_by=["metrics.auc DESC"],
)
print(runs[["params.learning_rate", "metrics.auc"]])

This is genuinely useful. The difference between "I think the model with learning rate 0.01 was better" and "here are the metrics for every run, sorted" is significant.

Artifact storage

Beyond parameters and metrics, MLflow stores files — called artifacts. The most common artifact is the trained model itself, but you can log anything: plots, data samples, configuration files.

# Log the model
mlflow.sklearn.log_model(model, "model")

# Log a confusion matrix plot
mlflow.log_artifact("confusion_matrix.png")

# Log a data sample
mlflow.log_artifact("sample_data.csv")

MLflow supports multiple artifact backends: local filesystem, S3, GCS, Azure Blob Storage, or HDFS. You configure this once on the tracking server and every client stores artifacts in the same place.

This matters because the alternative is ad-hoc. Models end up on local drives, in random S3 buckets, or attached to Slack messages. MLflow gives you a single, organized location with each artifact tied to the run that produced it.

The model registry

The model registry is where MLflow goes from "experiment tracking tool" to "part of your deployment pipeline." It's a central catalog of models with versioning and stage transitions.

The workflow:

  1. Train a model and log it to a run.
  2. Register it in the model registry under a name (e.g., "fraud-detector").
  3. Each registration creates a new version (v1, v2, v3...).
  4. Transition versions between stages: None → Staging → Production → Archived.
import mlflow

# Register a model from a run
result = mlflow.register_model(
    model_uri="runs:/abc123def/model",
    name="fraud-detector"
)

# Transition to staging
client = mlflow.tracking.MlflowClient()
client.transition_model_version_stage(
    name="fraud-detector",
    version=result.version,
    stage="Staging"
)

The registry answers questions that come up constantly in teams:

  • Which model is in production? Check the registry. The version in the "Production" stage is the one serving traffic.
  • What changed between versions? Each version links back to its run, which has all the parameters and metrics.
  • Who promoted this model? The registry logs stage transitions.
  • Can we roll back? The previous version is still in the registry. Transition it back to "Production."

The registry's limitations

The registry manages metadata and stage labels. It does not:

  • Actually deploy models. Transitioning to "Production" stage is just a label change. You need something else to notice that label and update your serving infrastructure.
  • Run validation tests. There's no built-in gate that says "only promote to Production if accuracy > X." You have to build that logic yourself.
  • Handle A/B testing or canary deployments. The registry has one "Production" slot per model name. Traffic splitting is your problem.

This is a common source of confusion. People expect the registry to be a deployment tool. It's a catalog.

What MLflow does passably

MLflow Projects

MLflow Projects is a packaging format for ML code. You define an MLproject file that specifies the entry point, parameters, and environment:

name: fraud-detection
conda_env: conda.yaml
entry_points:
  main:
    parameters:
      learning_rate: {type: float, default: 0.01}
      epochs: {type: int, default: 50}
    command: "python train.py --lr {learning_rate} --epochs {epochs}"

The idea is good — make training runs reproducible by pinning the environment and parameterizing the entry point. In practice, most teams outgrow this quickly. Docker containers give you more control, CI/CD systems give you better orchestration, and the MLproject format adds a layer of abstraction that doesn't carry its weight for production workloads.

It's fine for personal projects and small teams. For anything running on Kubernetes, you'll probably skip it.

MLflow Models (the serving component)

MLflow can serve models via mlflow models serve. It wraps your model in a REST endpoint:

mlflow models serve -m "models:/fraud-detector/Production" -p 5001

This gives you a working prediction endpoint. It's useful for testing and demos. For production, you'll want more control over the serving infrastructure — custom preprocessing, batching, GPU inference, health checks, autoscaling. Tools like Seldon, KServe, or even a plain Flask/FastAPI app typically replace this.

What MLflow doesn't do

This is where expectations often diverge from reality.

Data versioning

MLflow tracks which parameters and code produced a model, but it does not track which data was used. You can log a hash or a path as a parameter, but MLflow doesn't version datasets, detect data drift, or manage data lineage.

If you need data versioning, look at DVC, Delta Lake, or LakeFS. This is a separate problem from experiment tracking.

Feature stores

MLflow doesn't manage features. It doesn't compute, store, or serve feature values. If you need consistent feature computation between training and serving, you need a feature store (Feast, Tecton, or a custom solution).

Pipeline orchestration

MLflow doesn't schedule or orchestrate multi-step pipelines. It can log the results of each step, but it doesn't manage dependencies between steps, retry failed steps, or schedule runs.

For orchestration, teams use Airflow, Prefect, Argo Workflows, or Kubernetes CronJobs. MLflow is a logging destination, not an orchestrator.

Monitoring

MLflow doesn't monitor models in production. It doesn't track prediction latency, data drift, or model degradation over time. Monitoring is a separate concern — tools like Evidently, Whylogs, or custom Prometheus metrics handle this.

Access control (in the open-source version)

The open-source MLflow tracking server has no authentication or authorization. Anyone who can reach the server can see all experiments and modify all models. Databricks' managed MLflow adds access control, but the self-hosted version doesn't have it.

If you're running MLflow internally, you'll need to put it behind a reverse proxy with authentication, or accept that it's an open system within your network.

When to adopt MLflow

MLflow is worth adopting when:

  • You have more than one person training models, or you're training models often enough that you lose track of what you've tried.
  • You need a model registry to answer "what's in production and why."
  • You want a standard way to log experiments that doesn't depend on a specific cloud vendor.

MLflow is probably overkill when:

  • You're one person training one model occasionally. A notebook with notes is fine.
  • You're fully committed to a cloud ML platform (SageMaker, Vertex AI) that has its own tracking built in.

The practical architecture

In most setups we see, MLflow occupies a specific slot:

Training jobs (K8s / GPU nodes)
        │
        ▼
   MLflow Tracking Server  ──▶  Artifact Store (S3 / MinIO)
        │
        ▼
   MLflow Model Registry
        │
        ▼
   Deployment pipeline (CI/CD, Argo, custom)
        │
        ▼
   Serving infrastructure (KServe, Flask, etc.)

MLflow handles the middle layers — tracking and registry. Everything above (compute, scheduling) and below (deployment, serving, monitoring) is handled by other tools. Understanding this boundary is key to adopting MLflow without being disappointed by what it can't do.

Setting up MLflow as part of a production ML platform is something we do regularly at deploying.ai. If you're figuring out where MLflow fits in your stack, get in touch.