Batch vs Online Inference

This lesson on Batch vs Online Inference is hands-on and example-driven. You will evaluate machine learning workloads to select between batch and online inference strategies based on latency requirements, compute costs, and user engagement rates. You will prevent high infrastructure waste by replacing low-conversion batch precomputations with reactive, on-demand prediction workflows.

What You'll Be Able To Do

• Categorize machine learning workloads into batch or online regimes based on latency sensitivity and business objectives.
• Map user session triggers and live market events to on-demand online inference workflows.
• Calculate compute waste metrics to identify when batch precomputations should transition to online inference.
• Architect low-latency serving routines for dynamic pricing, fraud detection, and session-aware recommendations.

Detailed Concept Walkthrough

1. Batch Prediction Fundamentals

Batch inference processes static datasets at scheduled recurring intervals for tasks that tolerate delayed outputs. It optimizes throughput over latency by predicting on stored data asynchronously.

• Execution Flow: A scheduled job (running daily, weekly, or hourly) pulls a snapshot of data, runs predictions across the entire batch, and stores the results in a database or object store for downstream systems to consume asynchronously.
• Under the Hood: High-throughput batch frameworks optimize GPU and CPU utilization through massive parallelization and vectorized operations across thousands or millions of records simultaneously without maintaining an always-on live endpoint.
• Best Practice: Reserve batch inference strictly for latency-insensitive tasks—such as user churn prediction, retention campaign targeting, or weekly risk reporting—where consumers do not require immediate responses to recent user actions.

# Scheduled batch job running daily at midnight
import pandas as pd
import joblib

model = joblib.load('churn_model.pkl')
user_data = pd.read_parquet('s3://lake/users_daily_snapshot.parquet')

# Vectorized batch prediction
user_data['churn_risk_score'] = model.predict_proba(user_data[['tenure', 'usage_hours', 'support_tickets']])[:, 1]
user_data[['user_id', 'churn_risk_score']].to_sql('user_retention_scores', con=engine, if_exists='replace')

Key Takeaway: Batch inference maximizes throughput for asynchronous, non-time-sensitive workloads by precomputing predictions on recurring schedules.

2. Real-Time Recommenders and Dynamic Pricing

Online inference generates predictions synchronously on demand in response to live user interactions or incoming event signals. It delivers instantaneous adaptation to transient states like in-session browsing or dynamic market shifts.

• Mechanism: Incoming HTTP or gRPC requests carry real-time features (such as current clickstream events, supply-demand balance, or driver location) directly into a hosted model endpoint that calculates a score in milliseconds.
• Under the Hood: The serving engine prioritizes low end-to-end latency (P95/P99 < 50ms) over maximum batch throughput, keeping model artifacts hot in memory to process individual requests or micro-batches instantly.
• Best Practice: Deploy online inference for session-aware recommenders, fraud detection, and dynamic pricing (e.g., ride-share surge pricing or marketplace buy-box competition) where predictions must reflect the immediate, present context.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class PricingRequest(BaseModel):
    driver_count: int
    rider_demand: int
    route_distance: float

@app.post("/predict-surge-price")
def calculate_price(req: PricingRequest):
    # Synchronous real-time inference using immediate contextual signals
    surge_multiplier = pricing_model.predict([[req.driver_count, req.rider_demand, req.route_distance]])[0]
    return {"surge_multiplier": float(surge_multiplier)}

Key Takeaway: Online inference is required whenever predictions must instantly adapt to live session context, market conditions, or fraud signals.

3. The Batch Precomputation Fallacy

Precomputing batch scores across an entire user base causes massive compute and financial waste when only a fraction of those users engage within the prediction window.

• Mechanism: Generating scores for 100% of registered accounts creates wasted compute proportional to the inactive user base; if 2% of users log in daily, 98% of precomputed predictions expire unused.
• Under the Hood: Batch precomputation incurs both wasted compute cycles during offline model runs and unnecessary database write operations, while also suffering from data staleness caused by multi-hour scheduling delays.
• Best Practice: Evaluate the ratio of Daily Active Users (DAU) to total registered users; switch low-conversion pipelines to on-demand online inference to calculate predictions only when a user initiates a session.

# Compute waste analysis function
def calculate_batch_waste(total_registered_users: int, daily_active_users: int, inference_cost_per_1k: float) -> dict:
    dau_rate = daily_active_users / total_registered_users
    wasted_compute_pct = (1.0 - dau_rate) * 100
    wasted_daily_cost = ((total_registered_users - daily_active_users) / 1000) * inference_cost_per_1k
    return {
        "wasted_pct": wasted_compute_pct,
        "wasted_daily_cost_usd": wasted_daily_cost
    }

# Example: 1M users, 2% DAU
metrics = calculate_batch_waste(1_000_000, 20_000, 0.05)
# {'wasted_pct': 98.0, 'wasted_daily_cost_usd': 49.0}

Key Takeaway: Avoid precomputing batch predictions for large, low-activity user bases to prevent extreme compute waste and stale scores.

Topics Covered in Batch vs Online Inference

1. Batch Prediction Fundamentals (0:04 - 0:43) — Scheduled batch jobs operate asynchronously for latency-insensitive workloads like customer churn.
2. Real-Time Recommender Systems (0:43 - 1:27) — Session-aware recommendations require immediate on-demand inference before users exit the platform.
3. Dynamic Pricing and Market Balancing (1:27 - 2:35) — Live market balancing and fraud detection require low-latency inference on real-time signals.
4. Inference Cost Optimization (2:35 - 3:30) — Precomputing predictions across full user bases causes massive compute waste when active engagement is low.

ML in Practice Cheat Sheet

• Scheduled Batch Job — Runs bulk inferences over static datasets on timed schedules

  cron = CronTrigger(hour=0, minute=0); scheduler.add_job(run_churn_batch, cron)
  

• Synchronous API Endpoint — Serves on-demand predictions in real time via HTTP/gRPC

  @app.post('/predict')
  def predict(item: Item): return {'score': model.predict([item.features])}
  

• Dynamic Pricing Inference — Calculates rates instantly based on live market conditions

  price = model.predict([[current_demand, driver_supply, weather_code]])[0]
  

• Batch Waste Formula — Calculates percentage of unused precomputed model predictions

  waste_pct = (1.0 - (daily_active_users / total_users)) * 100
  

Comparison Table

Common Pitfalls

• Mistake: Precomputing predictions for all users regardless of activity. Avoid: Calculate batch waste and switch to online inference when active user ratios are low.
• Mistake: Using batch inference for session-aware recommender systems. Avoid: Implement online prediction endpoints triggered directly by user session events.
• Mistake: Over-engineering real-time APIs for churn prediction tasks. Avoid: Schedule recurring batch pipelines when decisions operate on multi-day retention cycles.

FAQs

• When is batch inference preferable to online inference? Batch inference is ideal for high-volume, latency-insensitive workloads where predictions do not change based on real-time interactions, such as daily churn or weekly risk scoring.
• What is the Batch Precomputation Fallacy? It is the mistaken belief that precomputing predictions for all users is cost-effective, ignoring that computing scores for inactive users wastes infrastructure budget.
• Why is dynamic pricing unsuitable for batch inference? Dynamic pricing relies on highly volatile, transient market signals like immediate rider demand and driver availability that cannot be captured in delayed batch runs.