Recap — Training/Serving Skew and How to Prevent It

# Training/Serving Skew and How to Prevent It Training-serving skew is the production ML bug with the worst signal-to-noise ratio. The model works in development, fails in production, and the cause is rarely the model itself. ## The mechanic In training: ```python def build_features(row): return [ row['amount'], row['merchant_id'].lower(), # lowercase np.log1p(row['user_history_count']), # log transform row['timestamp'].astype(int), # epoch ] ``` In serving: ```python def build_features_v2(req): return [ req.amount, req.merchant_id, # not lowercased — bug req.user_history_count, # not log-transformed — bug int(datetime.fromisoformat(req.timestamp).timestamp()), # different formula ] ``` Three subtle differences. The model receives different distributions than it was trained on. Predictions degrade silently. This happens in real teams constantly. The training pipeline is in a notebook; the serving pipeline is in a microservice. Both written by hand. They drift. ## The fix: one source of truth Option A: **Shared library** ```python # Single file imported by both training and serving # features/feature_engineering.py def build_features(row): return { 'amount': row['amount'], 'merchant_id_lower': row['merchant_id'].lower(), 'user_history_log': np.log1p(row['user_history_count']), 'timestamp_epoch': int(row['timestamp'].timestamp()), } ``` Both training scripts and serving service import this function. Same logic guaranteed. Lightweight, no infrastructure. Works for small projects. Option B: **Feature store** A dedicated system that: 1. Centralizes feature definitions. 2. Computes features in batch (for training). 3. Serves features in real-time (for serving). 4. Maintains consistency between the two. Tools: Feast (open-source), Tecton (managed), Hopsworks, AWS SageMaker Feature Store. ## What a feature store actually does For each feature you define: 1. **Batch processing**: compute the feature from historical data for training. 2. **Online serving**: compute (or look up) the feature at prediction time. 3. **Consistency guarantee**: both produce the same value for the same entity at the same point in time. Example with Feast: ```python # Define a feature view from feast import FeatureView, Field from datetime import timedelta user_features = FeatureView( name='user_features', entities=['user_id'], ttl=timedelta(days=7), schema=[ Field('total_orders', Int64), Field('avg_order_value', Float), ], source=BigQuerySource(table='user_features_table'), ) ``` Then: ```python # Training: get historical features historical_features = feature_store.get_historical_features( entity_df=user_entities, # who and when features=['user_features:total_orders', 'user_features:avg_order_value'], ).to_df() ``` ```python # Serving: get online features online_features = feature_store.get_online_features( entity_rows=[{'user_id': 'cust_123'}], features=['user_features:total_orders', 'user_features:avg_order_value'], ).to_dict() ``` Same feature view, same definition, same logic. ## Point-in-time correctness The hardest feature store guarantee: at training time, features must reflect what was known at the time of the prediction event. Example: predicting churn on 2025-04-15. Features at that point should include "number of logins in last 30 days" computed as of 2025-04-15 — not including events after 2025-04-15. This is called **point-in-time correctness** or **temporal consistency**. Skipping it = subtle training-serving skew. Feature stores handle this with timestamps: ```python historical_features = feature_store.get_historical_features( entity_df=pd.DataFrame({ 'user_id': ['cust_123', 'cust_456'], 'event_timestamp': [pd.Timestamp('2025-04-15'), pd.Timestamp('2025-04-15')], }), features=[...], ) ``` Feast computes each feature AS OF the event_timestamp. No future leak. ## When you don't need a feature store For most early-stage ML teams: - One or two models. - Features computed from a single source (warehouse). - Both training and serving query the same warehouse. A shared library is enough. Pay the feature-store cost (setup, ops) only when you have multiple models sharing features. ## When you do need one - 5+ models sharing features. - Real-time prediction with sub-100ms latency (warehouse queries too slow). - Features computed from multiple sources. - Strict point-in-time requirements. - Multiple teams collaborating. At this scale, a feature store amortizes its cost. ## The middle ground: SQL-defined features A pragmatic alternative: 1. Features defined in dbt models. 2. Training pulls from the dbt model. 3. Serving queries the same dbt model (or a materialized tab