Building a Prediction Service — The FastAPI Pattern

This lesson on Building a Prediction Service — The FastAPI Pattern is hands-on and example-driven. You will refactor a pre-trained PyTorch/Tez text classification model into a lightweight, high-performance inference API using FastAPI and Uvicorn. You will isolate the inference graph, load weights outside the request cycle, calibrate raw logits to probabilities using sigmoid activation, and validate your endpoints interactively via Swagger UI.

What You'll Be Able To Do

• Prune training loops, optimizers, and loss metrics from PyTorch model code to isolate an inference-only architecture.
• Instantiate a FastAPI application and serve it asynchronously using Uvicorn with hot-reloading enabled.
• Persist model weights in memory outside request handlers to eliminate per-request loading latency.
• Construct parameterized GET endpoints with type-hinted query parameters for single-sample inference.
• Transform uncalibrated model logits into normalized binary class probabilities using torch.sigmoid.

Detailed Concept Walkthrough

1. Pruning Training Code for Inference

Serving deep learning models in production requires stripping away all training-specific dependencies and layers to minimize memory footprint and execution overhead. Only the dataset tokenization logic and the forward pass computation graph are needed for serving.

• Mechanism: Training code contains optimizers, learning rate schedulers, and loss computations (e.g., BCEWithLogitsLoss) that are irrelevant during inference. Pruning involves extracting only the TextModel architecture and the dataset class required to build tokenized input tensors (input_ids, attention_mask).
• Under the Hood: By eliminating backward-pass hooks and training dependencies, the Python runtime allocates less memory and avoids tracking gradient graphs, improving baseline response latency.
• Best Practice: Retain the exact tokenizer configuration and input dimensionality used during training to prevent silent data corruption or tensor shape mismatch exceptions during the forward pass.

import torch
import tez
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer

class TextModel(tez.Model):
    def __init__(self, num_classes):
        super().__init__()
        self.bert = AutoModel.from_pretrained("bert-base-uncased")
        self.out = nn.Linear(768, num_classes)

    def forward(self, ids, mask):
        _, out = self.bert(ids, attention_mask=mask, return_dict=False)
        return self.out(out)

Key Takeaway: Extract only the dataset tokenization and forward-pass model definitions to create a clean, memory-efficient inference pipeline.

2. FastAPI Lifecycle and Persistent Model Loading

FastAPI is an ASGI framework that provides automatic data serialization and type validation. Heavy operations like loading model weights must be performed once at application startup rather than inside endpoint handlers.

• Mechanism: Instantiate the FastAPI application at the top level of api.py and invoke model.load() globally so model weights load into GPU or host memory during the initial server spin-up.
• Execution Flow: When a client sends a request to /predict, the handler accesses the already-instantiated model in memory, eliminating redundant disk I/O and weight-parsing overhead.
• Syntax Rule: Launch the server using the Uvicorn CLI by supplying the module name and app object separated by a colon (uvicorn api:app --reload), allowing auto-generated OpenAPI documentation to be served at /docs.

from fastapi import FastAPI

app = FastAPI()

# Instantiate model and load weights into memory ONCE at module load time
model = TextModel(num_classes=1)
model.load("model.bin", weights_only=True)

@app.get("/")
def read_root():
    return {"status": "active", "service": "sentiment-analyzer"}

Key Takeaway: Load model weights globally at application startup so request endpoints perform pure in-memory tensor computation.

3. Single-Sample Inference and Dataset Placeholders

Standard PyTorch/Tez dataset classes often expect target labels for loss computation, requiring dummy values when processing unlabelled real-time API queries.

• Mechanism: Wrap the incoming query string into a single-item dataset instance, providing a placeholder target list such as [-1] to satisfy the dataset constructor's parameter signature.
• Execution Flow: Pass the single-sample dataset directly into model.predict(), explicitly setting batch_size=1 to prevent the engine from waiting for or assuming larger mini-batches.
• Under the Hood: Setting batch_size=1 ensures the inference engine processes the single input immediately without iterator overhead or unnecessary memory padding allocations.

class SentimentDataset:
    def __init__(self, text, tokenizer, max_len):
        self.text = text
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.text)

    def __getitem__(self, item):
        text = str(self.text[item])
        inputs = self.tokenizer.encode_plus(
            text, None, add_special_tokens=True, max_length=self.max_len, padding="max_length", truncation=True
        )
        return {
            "ids": torch.tensor(inputs["input_ids"], dtype=torch.long),
            "mask": torch.tensor(inputs["attention_mask"], dtype=torch.long),
            "targets": torch.tensor([-1], dtype=torch.float) # Dummy target placeholder
        }

Key Takeaway: Supply dummy target placeholders to training-derived dataset classes and enforce batch_size=1 for lowest real-time request latency.

4. Logit Calibration and Response Formatting

Binary models trained with BCEWithLogitsLoss output uncalibrated, unbounded raw logits that must be transformed into probabilities via the sigmoid function before serialization.

• Mechanism: Extract the raw prediction tensor or NumPy array from the prediction generator and apply torch.sigmoid() to project the logit value onto a standard [0, 1] probability range.
• Under the Hood: Models trained with combined logit loss functions omit internal activation layers for numerical stability during gradient updates; inference code must explicitly compute $\sigma(x) = \frac{1}{1 + e^{-x}}$.
• Syntax Rule: Convert the output tensor to a native Python float (e.g., using .item() or float()) before building the response dictionary, ensuring seamless JSON serialization by FastAPI.

@app.get("/predict")
def predict(text: str):
    dataset = SentimentDataset(text=[text], tokenizer=tokenizer, max_len=128)
    preds = model.predict(dataset, batch_size=1)
    
    # Iterate prediction batches and extract raw logit
    for prediction in preds:
        raw_logit = prediction[0][0]
        pos_score = float(torch.sigmoid(torch.tensor(raw_logit)).item())
        neg_score = 1.0 - pos_score
        return {"positive": pos_score, "negative": neg_score}

Key Takeaway: Always pass raw logits through torch.sigmoid and cast to standard Python floats before returning prediction responses.

Topics Covered in Building a Prediction Service — The FastAPI Pattern

1. Serving Prerequisites Review (0:00 - 3:05) — The instructor compares training pipelines with serving architectures and reviews the IMDB sentiment model architecture.
2. Pruning Code for Inference (3:05 - 4:39) — The training loop, optimizers, and loss definitions are removed to create a lean inference module.
3. FastAPI Initialization (4:39 - 5:53) — A FastAPI application instance is created alongside a basic health-check root route.
4. Uvicorn and Swagger Inspection (5:53 - 7:16) — The ASGI server is launched on port 12000 and tested via the interactive Swagger documentation UI.
5. Model Weight Loading (7:16 - 9:15) — The pre-trained model weights are loaded into memory globally and a typed prediction route is defined.
6. Inference Pipeline Integration (9:15 - 11:46) — Query strings are converted into dataset inputs with dummy targets and evaluated using a batch size of one.
7. Logit Calibration and Response (11:46 - 13:30) — Raw output logits are transformed via sigmoid activation into positive and negative confidence scores returned as JSON.

ML in Practice Cheat Sheet

• uvicorn <module>:<app_instance> --reload --port <port> — Starts ASGI server with hot reloading enabled

  uvicorn api:app --reload --port 12000
  

• FastAPI() — Instantiates the core FastAPI application object

  app = FastAPI()
  

• @app.get("/<route>") — Registers a GET endpoint handler function

  @app.get("/predict")
  def get_prediction(text: str):
      return {"input": text}
  

• model.load("<path>", weights_only=True) — Loads serialized model parameters into memory

  model.load("model.bin", weights_only=True)
  

• torch.sigmoid(<tensor>) — Converts raw logits to 0-1 probabilities

  prob = torch.sigmoid(torch.tensor(raw_logit)).item()
  

• model.predict(<dataset>, batch_size=1) — Executes single-sample inference over a dataset

  preds = model.predict(dataset, batch_size=1)
  

Comparison Table

Common Pitfalls

• Mistake: Loading model weights inside the endpoint function. Avoid: Load the model once globally at application startup.
• Mistake: Inverting the Uvicorn CLI target string syntax. Avoid: Pass filename followed by app instance as filename:app.
• Mistake: Returning unactivated logits directly from BCEWithLogitsLoss models. Avoid: Wrap raw logit outputs in torch.sigmoid before serialization.
• Mistake: Leaving dataset target requirements unhandled during inference. Avoid: Pass dummy values like negative one in the dataset constructor.
• Mistake: Omitting batch_size configuration during single-sample inference. Avoid: Explicitly set batch_size=1 in model.predict to minimize latency.

FAQs

• Why does my API return unbounded values like 3.42 instead of sentiment percentages? Models trained with BCEWithLogitsLoss output raw, unactivated logits. You must explicitly apply torch.sigmoid() to map output logits into a valid 0.0 to 1.0 probability range.
• Why should I use GET instead of POST for this sentiment endpoint? GET endpoints with query parameters allow quick manual debugging in browser address bars and Swagger UI; however, production payloads with long text should use POST with request bodies.
• What happens if I forget to specify batch_size=1 in model.predict()? The predict engine defaults to higher batch evaluations, incurring unnecessary memory buffering overhead and adding latency to real-time single-sample requests.
• How do I access the interactive API documentation? Navigate to the /docs path on your running server in any browser to open the auto-generated Swagger UI.