k-Means / DBSCAN / Hierarchical Lab

This lesson on k-Means / DBSCAN / Hierarchical Lab is hands-on and example-driven. You will segment unlabeled customer data using k-Means, DBSCAN, and Hierarchical Clustering in Python. You will evaluate clustering quality and mathematically justify your choice of cluster count using the elbow method and silhouette analysis.

What You'll Be Able To Do

• Preprocess and scale unlabeled feature matrices for distance-based clustering algorithms
• Implement k-Means clustering and select optimal k using inertia elbow curves and silhouette scores
• Configure DBSCAN with tuned epsilon and minimum samples to extract arbitrary shapes and isolate noise
• Construct agglomerative hierarchical dendrograms to identify natural structural cutoffs in segmentation data
• Compare clustering performance across geometric, density-based, and linkage-based evaluation metrics

Detailed Concept Walkthrough

1. k-Means and Cluster Validation

k-Means partitions feature space into k Voronoi cells by iteratively minimizing within-cluster sum-of-squares (inertia). It assumes convex, isotropic clusters of roughly equal variance.

• Mechanism: Initializes k centroids (using k-means++ to maximize spread), assigns each sample to its nearest centroid via Euclidean distance, and updates centroids to the mean of assigned points until convergence.
• Under the Hood: Convergence occurs when centroid reassignment shift falls below a tolerance threshold (tol) or max iterations are reached, achieving a local optimum in O(t * k * n * d) runtime.
• Best Practice: Determine the optimal k by locating the 'elbow' inflection point in inertia plots alongside peak mean silhouette coefficients, which quantify cluster separation versus cohesion.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import numpy as np

# Fit k-means with k-means++ initialization
kmeans = KMeans(n_clusters=4, init='k-means++', n_init=10, random_state=42)
labels = kmeans.fit_predict(X_scaled)

# Evaluate clustering separation and compactness
sil_score = silhouette_score(X_scaled, labels)
print(f"Inertia: {kmeans.inertia_:.2f}, Silhouette: {sil_score:.3f}")

Key Takeaway: Always validate k-Means cluster counts using both inertia elbow inflection and silhouette score maximization.

2. Density-Based Spatial Clustering (DBSCAN)

DBSCAN discovers clusters of arbitrary geometric shape based on local spatial density thresholds without requiring a predefined cluster count. Points residing in low-density regions are isolated as noise anomalies.

• Mechanism: Classifies each point as a Core point (having at least min_samples within distance eps), Border point (reachable from a Core point), or Noise point (labeled as -1).
• Under the Hood: Traverses the density-connected graph using spatial indexing structures (e.g., k-d trees or Ball trees) to expand clusters across connected core neighborhoods in O(n log n) expected time.
• Best Practice: Tune eps by plotting sorted k-nearest neighbor distances and locating the maximum curvature knee; set min_samples to at least 2 * dimensions.

from sklearn.cluster import DBSCAN
from sklearn.neighbors import NearestNeighbors

# Estimate eps using 4-NN distance curve
neighbors = NearestNeighbors(n_neighbors=4).fit(X_scaled)
distances, _ = neighbors.kneighbors(X_scaled)

# Fit DBSCAN with selected eps and min_samples
db = DBSCAN(eps=0.35, min_samples=5).fit(X_scaled)
noise_ratio = (db.labels_ == -1).mean()
print(f"Clusters: {len(set(db.labels_)) - (1 if -1 in db.labels_ else 0)}, Noise: {noise_ratio:.1%}")

Key Takeaway: Use DBSCAN when cluster shapes are non-spherical or when your pipeline requires explicit noise and outlier rejection.

3. Agglomerative Hierarchical Clustering

Hierarchical clustering builds a bottom-up nested tree of partitions by iteratively merging the closest pair of clusters according to a linkage criterion. It eliminates the need to commit to a fixed k upfront.

• Mechanism: Starts with each observation as a singleton cluster and merges the closest pair sequentially based on distance metrics (Euclidean, Manhattan) and linkage functions (Ward, Complete, Average).
• Under the Hood: Ward linkage minimizes the total within-cluster variance increase at each step, yielding balanced clusters similar to k-Means but structured hierarchically with O(n^3) naive or O(n^2 log n) optimized time complexity.
• Best Practice: Visualize cluster hierarchy using dendrograms from SciPy to identify the largest vertical distance cutoffs before instantiating scikit-learn AgglomerativeClustering.

from scipy.cluster.hierarchy import dendrogram, linkage
from sklearn.cluster import AgglomerativeClustering
import matplotlib.pyplot as plt

# Generate linkage matrix using Ward's method
Z = linkage(X_scaled, method='ward')

# Cut tree at target cluster count
agg = AgglomerativeClustering(n_clusters=3, linkage='ward')
cluster_labels = agg.fit_predict(X_scaled)

Key Takeaway: Hierarchical clustering provides full visibility into multi-scale data groupings and informs discrete cluster cutoff decisions.

4. Feature Standardization and Distance Metrics

Distance-based clustering algorithms calculate geometric proximity across feature dimensions, making variance standardization essential to avoid feature dominance.

• Mechanism: Unscaled features with large dynamic ranges artificially dominate Euclidean distance calculations, causing clustering boundaries to reflect only the highest-magnitude column.
• Under the Hood: StandardScaler centers data to zero mean and unit variance (z = (x - mu) / sigma), transforming feature hyperspace into an isotropic coordinate system where distances carry equal dimensional weighting.
• Best Practice: Always apply StandardScaler or RobustScaler (if heavy outliers are present) inside a scikit-learn Pipeline before feeding feature matrices into distance-dependent clusterers.

from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Build modular segmentation pipeline
seg_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clusterer', KMeans(n_clusters=3, random_state=42))
])
seg_pipe.fit(X_raw)
assigned_clusters = seg_pipe.named_steps['clusterer'].labels_

Key Takeaway: Never run distance-based clustering on raw unscaled features; always standardize inputs first.

Topics Covered in k-Means / DBSCAN / Hierarchical Lab

1. Lab Overview and Setup (0:00 - 1:15) — Outlines the unlabeled customer segmentation objectives and feature preprocessing steps.
2. k-Means and Elbow Analysis (1:15 - 3:00) — Demonstrates fitting k-Means models across k values and plotting inertia elbows.
3. Silhouette Validation (3:00 - 4:45) — Calculates silhouette coefficients to mathematically justify the final cluster count.
4. DBSCAN Density Clustering (4:45 - 6:30) — Applies DBSCAN to detect non-spherical groupings and isolate anomalous noise points.
5. Hierarchical Dendrogram Cuts (6:30 - 7:45) — Constructs linkage matrices and cuts dendrograms to validate structural cluster groupings.
6. Model Comparison and Summary (7:45 - 9:00) — Summarizes trade-offs between geometric, density, and hierarchical clustering pipelines.

ML Case Labs Cheat Sheet

• KMeans(n_clusters=k, init='k-means++', n_init=10) — Partitions data into k spherical clusters minimizing inertia

  km = KMeans(n_clusters=3, init='k-means++', random_state=42).fit(X)
  

• DBSCAN(eps=0.5, min_samples=5) — Groups dense regions and tags outliers as noise (-1)

  db = DBSCAN(eps=0.4, min_samples=5).fit(X)
  

• AgglomerativeClustering(n_clusters=k, linkage='ward') — Recursively merges clusters to build a bottom-up hierarchy

  agg = AgglomerativeClustering(n_clusters=4, linkage='ward').fit(X)
  

• silhouette_score(X, labels) — Measures cluster cohesion versus separation from -1 to 1

  score = silhouette_score(X, labels)
  

• silhouette_samples(X, labels) — Calculates individual silhouette coefficient for each sample

  sample_scores = silhouette_samples(X, labels)
  

• scipy.cluster.hierarchy.linkage(y, method='ward') — Computes hierarchical clustering linkage matrix for dendrogram visualization

  Z = linkage(X, method='ward')
  

Comparison Table

Common Pitfalls

• Mistake: Clustering raw features without standardizing scales across dimensions. Avoid: Preprocess all features with StandardScaler before computing distance metrics.
• Mistake: Choosing k solely by minimizing inertia without silhouette evaluation. Avoid: Select k where silhouette scores peak and inertia displays clear elbow leveling.
• Mistake: Running k-Means on non-linear or elongated density manifolds. Avoid: Deploy DBSCAN or non-linear spectral clustering for complex non-convex topologies.
• Mistake: Setting DBSCAN epsilon arbitrarily without inspecting k-NN distance curvature. Avoid: Plot sorted k-nearest neighbor distances to locate the elbow transition.

FAQs

• What should I do if the inertia elbow plot has no distinct sharp bend? Evaluate the silhouette score across candidates and inspect cluster sizes to select the k with optimal separation and interpretability.
• Why does DBSCAN assign label -1 to some data points? Label -1 denotes noise points that lack sufficient neighboring points within the epsilon radius and are not reachable from core points.
• Which linkage metric should I default to for Agglomerative Clustering? Ward linkage is the standard default because it minimizes within-cluster variance and tends to produce balanced, cohesive clusters.