MLflow 3.0 is the latest major version, adding native GenAI tracing, agent evaluation primitives, and tighter Databricks integration. It keeps the four pillars of MLflow (tracking, projects, models, registry) but extends them with first-class support for LLM-backed systems.

Why use MLflow on Databricks specifically?

MLflow on Databricks is managed end-to-end. Experiments, models, deployments, and tracing all flow through Unity Catalog with proper governance. You don't run an MLflow tracking server, Databricks does. The same artifacts also work in any non-Databricks environment via open MLflow, so you're never locked in.

How does MLflow Tracing work for LLM applications?

MLflow Tracing captures every model call, retrieval, tool invocation, and intermediate step as a span. For agents and RAG systems, this gives you a complete execution trace per request, visible in the MLflow UI, queryable, and usable as input to evaluation.

What is the MLflow Model Registry used for?

The Model Registry is the staging area between training and production. Models are registered, versioned, transitioned through stages (Development, Staging, Production), and deployed from the registry. On Databricks, the registry lives in Unity Catalog and inherits its access controls.

Can I use MLflow with non-Databricks frameworks?

Yes. MLflow is open source and framework-agnostic. It tracks experiments and deploys models from PyTorch, TensorFlow, scikit-learn, XGBoost, LightGBM, and the major LLM providers. On Databricks, the integration is tighter, but the same code works elsewhere.

How does MLflow Evaluate work for RAG and agents?

MLflow Evaluate runs a dataset of inputs through your pipeline (RAG, agent, model) and applies a configurable set of judges to score the outputs. Built-in judges cover answer relevance, faithfulness, groundedness, and toxicity. Custom judges can be defined in Python or as LLM-as-judge prompts.

Topic guide

· Covered in Volume 4 · The AI Lakehouse

MLflow 3.0 on Databricks: The Complete Guide

MLflow is the connective tissue of every serious ML project on Databricks. The 3.0 release made it the connective tissue for GenAI too, experiments, UC Model Registry, aliases, traces, and evaluation, used together as a system. This guide covers what's new, what matters, and what production teams actually use.

10 min read Updated 2026-05-18 By Ritesh Modi

MLflow has done something rare: it stayed useful as the field around it shifted from classical ML to deep learning to LLMs. The 3.0 release is the version that makes this explicit. The four pillars are still there, tracking, projects, models, registry, but tracing and evaluation are now first-class primitives, and the GenAI workflow has its own shape.

On Databricks specifically, MLflow is the managed runtime for almost every ML and AI artifact in your workspace. Experiments, model versions, traces, evaluations, all governed through Unity Catalog, accessible through a single API, deployable to a single Model Serving endpoint.

The four-plus-two pillars

The classical four:

Tracking, log parameters, metrics, and artifacts from every experiment run.
Projects, package code and dependencies for reproducible execution.
Models, a standardized format for model artifacts, framework-agnostic.
Registry, versioned model storage with stage transitions and approval workflow.

The new two:

Tracing, span-based execution traces for LLM applications, RAG pipelines, and agents.
Evaluation, opinionated evaluation framework with built-in and custom judges.

On Databricks, all six are integrated. Unity Catalog governs which users see which experiments, which models, which traces. Model Serving deploys directly from the registry. The Lakehouse is the underlying storage.

Tracking, still the foundation

The tracking API is unchanged in spirit: mlflow.start_run, log params, log metrics, log artifacts, end. The discipline is what separates teams that ship from teams that don't.

Things that should always be logged for a production model:

The exact data version (Delta table snapshot ID or Unity Catalog volume version).
The full hyperparameter set, including defaults you didn't override.
The evaluation metrics on a held-out test set.
The model artifact itself.
The code revision (git SHA).

With Databricks autologging, much of this happens automatically. But the autolog defaults are a starting point, not a finish line. Be explicit about what matters for your workload.

Tracing, the unlock for LLM debugging

The single biggest practical win of MLflow 3.0 is tracing. A RAG system or agent now produces a structured trace for every request: the query, the retrieval, the chunks returned, the prompt sent to the model, the model's response, every tool call, every intermediate reasoning step.

This sounds obvious. It is not, in practice, what most teams had before. Without tracing, debugging "why did the agent say X" means re-running, hoping for reproducibility, hoping the same path is taken. With tracing, you open the trace in MLflow and read the actual path.

Tracing has a quiet second benefit: traces become evaluation inputs. You can replay production traces against a new prompt and measure regression directly. This collapses the dev/prod gap.

The Model Registry, properly used

The registry is the production interface to MLflow. Models in the registry have versions, stages, and aliases (in MLflow 2.x+, aliases replaced the older stage system). A typical flow:

Train and log a model in an experiment run.
Register the model to the registry. It gets version N.
Assign aliases: @candidate, @staging, @production.
Deploy by reference to the alias: serving always pulls the model tagged @production.
Promote a candidate to production by moving the alias. Old version stays accessible for rollback.

On Databricks, registry entries live in Unity Catalog, which means three-level naming (catalog.schema.model) and proper ACLs. Production model access is enforced at the catalog level. This eliminates an entire class of "who deployed what" investigations.

Evaluation: the discipline the field skipped for too long

For a decade, ML evaluation was straightforward, pick a test set, compute accuracy or AUC, ship if the number is high enough. LLM evaluation is harder. There is no single right answer for an open-ended question. Faithfulness, relevance, groundedness, safety, each is a separate judge.

MLflow Evaluate gives you a structured way to apply many judges in parallel. Built-in judges cover the common cases. Custom judges let you encode domain-specific rules, "did the legal-advice answer cite at least one statute?" or "did the medical answer include the dosage range?"

The pattern that works:

Build an evaluation dataset, input plus expected behavior, not necessarily expected exact output.
Define the judges that matter for your use case.
Run evaluation on every meaningful pipeline change.
Block deployment if any judge regresses past a threshold.

The same discipline you'd apply to unit tests, applied to LLM behavior.

What the book covers

The MLflow chapter in Databricks for Practitioners goes through:

A reference experiment tracking pattern that scales from one engineer to a team of 50.
Tracing setup for RAG and AgentBricks pipelines.
The full Unity Catalog model registry workflow.
Evaluation harness design with built-in and custom judges.
Promotion gating: how to wire MLflow into CI so a model can't reach production without passing evaluation.
Common antipatterns and the failure modes they create downstream.