PyTorch Tensors + Autograd

This lesson on PyTorch Tensors + Autograd is hands-on and example-driven. You will master PyTorch tensor operations, automatic differentiation via Autograd, and verify your computational graph using numerical gradient checks. By the end of this session, you will build and overfit a minimal multi-layer perceptron (MLP) from scratch to validate your training loop pipeline.

What You'll Be Able To Do

• Construct and manipulate multidimensional PyTorch tensors with explicit data types and device placements.
• Track operations dynamically using Autograd and trigger backward graph traversals with backward().
• Implement a finite-difference numerical gradient check to validate analytical gradients.
• Zero out accumulated tensor gradients within an iterative parameter optimization loop.
• Overfit a minimal MLP on a synthetic toy batch to verify loss convergence to zero.

Detailed Concept Walkthrough

1. Tensor Fundamentals and Device Memory

PyTorch tensors are multidimensional arrays that wrap contiguous memory blocks and support hardware acceleration on GPUs/TPUs alongside automated gradient tracking.

• Mechanism: Tensors share storage across views and expose strided indexing over contiguous memory buffers without copying data unnecessarily.
• Under the Hood: Specifying requires_grad=True allocates metadata fields (grad, grad_fn, and graph history) directly inside the tensor struct.
• Best Practice: Always allocate tensors directly on the target execution device and specify dtype=torch.float32 to avoid unexpected type promotions or cross-device synchronizations.

import torch

# Allocate tensor directly on target device with float32
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x = torch.tensor([[1.0, 2.0], [3.0, 4.0]], dtype=torch.float32, device=device, requires_grad=True)
print(f"Tensor: {x}\nDevice: {x.device}\nRequires Grad: {x.requires_grad}")

Key Takeaway: Tensors are strided numerical arrays with built-in device targeting and gradient tracking metadata.

2. Autograd Engine and Dynamic Computational Graphs

PyTorch builds dynamic directed acyclic graphs (DAGs) on the fly during the forward pass, using reverse-mode autodiff to backpropagate gradients from scalar loss nodes.

• Mechanism: Each operation applied to a tracked tensor creates a Node (stored as grad_fn) containing the backward derivative formulation.
• Under the Hood: Calling loss.backward() traverses the DAG in reverse topological order, applying the chain rule to accumulate partial derivatives into .grad buffers.
• Best Practice: Disable the DAG engine via with torch.no_grad(): during inference and parameter updates to prevent memory leaks and graph tracking overhead.

import torch

x = torch.tensor([2.0, 3.0], requires_grad=True)
y = (x ** 2 + 3 * x).sum()
y.backward()

# dy/dx = 2x + 3 -> [7.0, 9.0]
print(f"Gradients: {x.grad}")
assert torch.allclose(x.grad, 2 * x.detach() + 3)

Key Takeaway: Dynamic DAGs are built during forward execution and evaluated backwards via the chain rule into tensor grad fields.

3. Finite-Difference Gradient Checking

Gradient checking verifies analytical backpropagation by calculating two-sided numerical approximations via tiny parameter perturbations.

• Mechanism: Central difference estimation computes (f(x + eps) - f(x - eps)) / (2 * eps) to approximate the true analytical derivative with second-order error.
• Execution Flow: Compare the analytical gradient x.grad against the numerical gradient using relative error tolerances: |g_num - g_ana| / (max(|g_num|, |g_ana|) + eps).
• Best Practice: Use double precision (torch.float64) and a small epsilon (e.g., 1e-5 to 1e-7) during gradient checks to avoid floating-point truncation issues.

import torch

def func(t):
    return (t ** 3).sum()

x = torch.tensor([1.5, -2.0], dtype=torch.float64, requires_grad=True)
y = func(x)
y.backward()

eps = 1e-6
num_grad = torch.zeros_like(x)
for i in range(x.numel()):
    x_pos = x.detach().clone()
    x_neg = x.detach().clone()
    x_pos[i] += eps
    x_neg[i] -= eps
    num_grad[i] = (func(x_pos) - func(x_neg)) / (2 * eps)

rel_err = torch.norm(x.grad - num_grad) / (torch.norm(x.grad) + torch.norm(num_grad))
print(f"Relative Error: {rel_err.item():.2e}")

Key Takeaway: Central finite differences provide a mathematically rigorous ground truth to validate custom Autograd derivatives.

4. Overfitting a Tiny MLP from Scratch

Overfitting a tiny batch of 2-8 samples down to near-zero loss confirms that the forward pass, loss calculation, backward pass, and parameter updates are correctly wired.

• Mechanism: Initialize explicit weight matrices and bias vectors with requires_grad=True, running iterative optimization over a miniature synthetic dataset.
• Execution Flow: Execute forward matrix multiplications, compute MSE or Cross-Entropy loss, invoke loss.backward(), update parameters in-place, and zero out gradients.
• Best Practice: Always execute weight updates inside a torch.no_grad() block and mutate parameters in-place (e.g., w.sub_(lr * w.grad)) while zeroing .grad tensors manually.

import torch

# Toy dataset: 4 samples, 2 input features, 1 target
X = torch.randn(4, 2)
Y = torch.tensor([[1.0], [0.0], [0.0], [1.0]])

# Initialize weights manually
W1 = torch.randn(2, 4, requires_grad=True)
b1 = torch.zeros(4, requires_grad=True)
W2 = torch.randn(4, 1, requires_grad=True)
b2 = torch.zeros(1, requires_grad=True)

lr = 0.05
for step in range(200):
    # Forward pass: Linear -> ReLU -> Linear
    h = torch.relu(X @ W1 + b1)
    pred = h @ W2 + b2
    loss = ((pred - Y) ** 2).mean()
    
    loss.backward()
    
    with torch.no_grad():
        for p in [W1, b1, W2, b2]:
            p.sub_(lr * p.grad)
            p.grad.zero_()

print(f"Final Overfit Loss: {loss.item():.6f}")

Key Takeaway: Driving loss to near-zero on a micro-batch guarantees graph connectivity and bug-free gradient propagation.

Topics Covered in PyTorch Tensors + Autograd

1. Tensors and Computational Graph (0:00 - 2:15) — Learners explore tensor allocation, data types, device memory placement, and the requires_grad parameter flag.
2. Autograd Mechanics and Backward Pass (2:15 - 4:45) — The instructor traces dynamic DAG construction, grad_fn attachments, and scalar backward propagation.
3. Numerical Gradient Checking (4:45 - 7:30) — The lesson demonstrates writing finite-difference tests to mathematically verify analytical backward derivatives.
4. Overfitting a Tiny MLP Drill (7:30 - 10:00) — Learners implement a complete manual optimization loop from scratch to drive training loss to zero on a micro-batch.

DL Basics Cheat Sheet

• torch.tensor(data, requires_grad=True) — Creates tensor and tracks operations for automatic differentiation

  w = torch.tensor([1.0, 2.0], requires_grad=True)
  

• tensor.backward(gradient=None) — Computes graph gradients via reverse-mode autodiff into .grad

  loss.backward()
  

• with torch.no_grad(): — Context manager disabling graph construction and gradient computation

  with torch.no_grad():
      w -= lr * w.grad
  

• tensor.grad.zero_() — Zeros accumulated gradient buffer in-place to prevent accumulation

  w.grad.zero_()
  

• tensor.detach() — Returns a new tensor detached from current computational graph

  y_pred = model_output.detach().cpu().numpy()
  

• torch.autograd.gradcheck(func, inputs) — Evaluates analytical gradients against finite-difference approximations automatically

  torch.autograd.gradcheck(torch.sin, torch.randn(2, dtype=torch.float64, requires_grad=True))
  

Comparison Table

Common Pitfalls

• Mistake: Forgetting to zero gradients before calling backward. Avoid: Always call grad.zero_() or optimizer.zero_grad() before each backward pass.
• Mistake: Mutating tracked model weights without a no_grad block. Avoid: Wrap parameter updates in with torch.no_grad() and use in-place operators.
• Mistake: Performing gradient checking with standard single-precision floats. Avoid: Cast inputs to torch.float64 to prevent numerical truncation and roundoff errors.

FAQs

• Why does PyTorch accumulate gradients across backward() calls instead of overwriting them? Accumulation natively supports distributed batch splitting and gradient accumulation across multiple mini-batches when GPU memory is constrained.
• What is the difference between tensor.detach() and with torch.no_grad():? detach() separates a specific tensor from the DAG, whereas torch.no_grad() disables DAG construction for all operations inside its block.
• Why is overfitting a tiny batch of 4 samples the first diagnostic step? If a network cannot overfit a toy batch to zero loss, the model architecture, loss function, or update step has a structural bug.