Recap — Building a Reproducible Pipeline

# Building a Reproducible Pipeline Reproducibility = given the same starting point, get the same result. Sounds trivial; in ML, it's surprisingly hard. Four things must be locked. ## Lock 1: Code Obvious: code in version control (git). Less obvious: every script that touched the training pipeline. - Feature engineering scripts. - Training code. - Hyperparameter tuning scripts. - Validation logic. If any of these live in a notebook that gets overwritten, or in a "private" branch, you've already lost reproducibility. Pattern: training is a runnable script with explicit inputs and outputs. Not a notebook for production. Notebooks are for exploration. ```bash python train.py --data-path data/v3 --output-dir models/v1.0.0 --config configs/xgb_v1.yaml ``` ## Lock 2: Data Code + same data = same model only if the data is the same. For real datasets that change over time, you need data versioning. Three options: ### DVC (Data Version Control) Tracks data files in S3/GCS with git-friendly hashes. ```bash dvc add data/training_data.parquet git add data/training_data.parquet.dvc data/.gitignore git commit -m "Training data v3" dvc push # uploads to remote storage ``` Now `git checkout ` + `dvc pull` reproduces the exact data state. ### Date-stamped snapshots Simpler: ```python # At training time training_data = load_data(snapshot_date='2026-05-26') ``` Snapshot path: `s3://data/snapshots/2026-05-26/`. Date in code; data immutable in storage. ### Commit the data Only if it's small (<100MB). Most ML datasets are too big for this. DVC is best practice; date-stamped snapshots are good-enough lightweight. ## Lock 3: Environment Same code + same data + same library versions = same model. ```python # Bad pip install scikit-learn xgboost pandas # Good pip install scikit-learn==1.4.0 xgboost==2.0.3 pandas==2.1.4 ``` Better: lock the whole environment. ### requirements.txt with pinned versions ``` scikit-learn==1.4.0 xgboost==2.0.3 pandas==2.1.4 numpy==1.26.0 ``` ### Poetry / pip-tools (lockfile) Captures transitive dependencies too. More robust. ### Docker (best) Lock the OS, Python version, library versions, all of it. ```dockerfile FROM python:3.11.5-slim COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . /app WORKDIR /app CMD ["python", "train.py"] ``` `docker build -t my-ml-trainer:v1 .` creates a reproducible environment that runs the same on anyone's machine. For training: Docker is overkill for solo iteration but worth it for shared/CI training. For serving: Docker is mandatory. Production environment must match training-time guarantees. ## Lock 4: Randomness ML algorithms often use randomness: - Train/test splits. - Bootstrap samples (random forest). - Initial weights (neural networks). - Mini-batch sampling. - Cross-validation fold assignment. Without seeding, two runs give slightly different models. Seemingly small differences cascade into different hyperparameter selections, different test scores, irreproducible results. ```python import random import numpy as np import torch random.seed(42) np.random.seed(42) torch.manual_seed(42) torch.cuda.manual_seed_all(42) torch.backends.cudnn.deterministic = True # for GPU determinism ``` For sklearn / XGBoost: pass `random_state=42` to every estimator and splitter. ```python X_train, X_test = train_test_split(X, y, random_state=42) model = XGBClassifier(random_state=42) ``` ## Tracking experiments Even with all four locks, you'll run many experiments. Track which model came from which: - Data version. - Code commit. - Config used. - Hyperparameters. - Resulting metrics. MLflow: ```python import mlflow with mlflow.start_run(): mlflow.log_param('learning_rate', 0.05) mlflow.log_param('max_depth', 6) mlflow.log_metric('val_auc', 0.85) mlflow.log_metric('test_auc', 0.83) mlflow.sklearn.log_model(model, 'model') mlflow.log_artifact('config.yaml') mlflow.set_tag('data_version', 'v3') mlflow.set_tag('git_commit', subprocess.check_output(['git', 'rev-parse', 'HEAD']).strip()) ``` Now `mlflow ui` shows every run. Re-find the best one. Compare params. Reproduce by checking out the git commit, restoring the data version, restoring the environment. ## What "reproducible" actually means Two flavors: ### Bit-for-bit reproducible Same code + same data + same env + same seed = exactly the same model. Possible for CPU-only training. Hard with GPU (non-deterministic kernels, atomic ops). ### Statistically reproducible Same general setup → models with very similar performance, even if exact weights differ. This is what you usually need. Match validation metrics within 0.5%; cluster centroids within 1%. For most ML systems, statistical reproducibility is the goal. Don't over-engineer for bit-for-bit unless you have a regulatory reason. ## Common reproducibility mistakes 1. **Notebooks as training code** — cell execution order matters; state leaks across cells. 2. **`pip i