Recap — MLflow, DVC, and Model Registry

# MLflow, DVC, and Model Registry Versioning is what lets you answer "which model is in production right now, and how do I reproduce or roll back from it?" Three layers: experiments, data, and registered models. ## MLflow: experiment tracking MLflow logs every training run — params, metrics, artifacts. ```python import mlflow import mlflow.sklearn mlflow.set_experiment("fraud-detection") with mlflow.start_run(run_name='xgb_v2_attempt_3'): # Log hyperparameters mlflow.log_params({ 'learning_rate': 0.05, 'max_depth': 6, 'n_estimators': 1000, }) # Train model = XGBClassifier(...) model.fit(X_train, y_train) # Log metrics val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1]) mlflow.log_metric('val_auc', val_auc) mlflow.log_metric('val_pr_auc', average_precision_score(y_val, model.predict_proba(X_val)[:, 1])) # Log the model mlflow.sklearn.log_model(model, 'model') # Log metadata mlflow.set_tag('data_version', 'v3.1') mlflow.set_tag('git_commit', current_git_commit()) ``` `mlflow ui` then provides a web UI showing all runs, sortable by metrics. Compare runs side-by-side. ## MLflow Model Registry The registry promotes a successful experiment run to a "registered model" — a stable name with versions. ```python # Register the model from a specific run mlflow.register_model( f'runs:/{run_id}/model', 'fraud_detector' ) ``` Now there's a named model `fraud_detector` with versions `1, 2, 3, ...`. Move them through stages: - `None` — newly registered, not promoted yet. - `Staging` — being tested. - `Production` — currently serving. - `Archived` — retired. ```python client = mlflow.tracking.MlflowClient() client.transition_model_version_stage( name='fraud_detector', version=4, stage='Production', ) ``` The serving code reads the current "Production" version: ```python model = mlflow.sklearn.load_model('models:/fraud_detector/Production') ``` This decouples model versions from the serving code. To deploy v5: register, mark as Production, restart the serving service. No code change. ## DVC: data versioning DVC tracks data files alongside git, with hashes stored in git and the actual files in S3/GCS/Azure. ```bash # Initialize dvc init # Add a data file dvc add data/training_set.parquet # git tracks the .dvc file (metadata) git add data/training_set.parquet.dvc data/.gitignore git commit -m "Training data v3" # Configure remote storage dvc remote add -d storage s3://my-bucket/dvc-storage dvc push # upload the actual data ``` Now `git checkout ` + `dvc pull` restores both code and data to that commit's state. DVC pipelines (dvc.yaml) link data preprocessing → training → evaluation. Re-runs only the steps whose inputs changed. For light-weight teams: DVC is solid but a learning curve. Alternative: dated snapshot folders in S3. `s3://data/snapshots/2026-05-26/training.parquet`. Date is the version. Simpler; less automated. ## Tagging the model with everything Every registered model should be traceable to: - The data version used. - The git commit of the training code. - The MLflow run ID. - The validation metrics it achieved. ```python client.set_model_version_tag( name='fraud_detector', version=4, key='data_version', value='v3.1' ) client.set_model_version_tag( name='fraud_detector', version=4, key='git_commit', value=current_git_commit() ) ``` Six months later, you can answer: "What data did the model in production v4 train on? What was its val AUC? Which git commit produced it?" All from MLflow metadata. ## What if I don't want MLflow? You can do this manually: - Train models → save to `s3://models/ / /model.pkl`. - Maintain a `models_meta.csv` with run info. - The "production" model is whichever file the serving service is configured to load. This works for tiny projects. As soon as you have 2+ models or 3+ engineers, the manual approach breaks down. MLflow is free; just adopt it. ## Promotion workflow Best practice for promoting a model to production: 1. **Train** — researcher runs experiments, logs to MLflow. 2. **Register** — best model registered, stage = None. 3. **Validate** — automated tests (predictions on a fixed test set; latency benchmarks; calibration check). 4. **Staging** — promote to Staging stage. Run against a sample of production traffic in shadow mode. 5. **Production** — once validated, promote to Production. Update serving config. 6. **Monitor** — track performance for the first week extra-carefully. A model goes None → Staging → Production over days or weeks, not in one commit. ## Rollback If the new model misbehaves: ```python client.transition_model_version_stage( name='fraud_detector', version=4, stage='Archived', ) client.transition_model_version_stage( name='fraud_detector', version=3, stage='Production', ) ``` Reload the serving service (or rely on the cache TTL