Market Basket Analysis in Python: Support, Confidence, Lift & Apriori Explained

When you browse Amazon and see "Frequently bought together", or walk through a supermarket and find chips placed directly next to fresh salsa, you are experiencing the direct output of Market Basket Analysis.

For data analysts and machine learning engineers, association rule mining is one of the highest-ROI analytical techniques. It requires no labelled training data, produces easily interpretable business rules, and directly increases Average Order Value (AOV) and gross margins.

In this comprehensive guide, we will break down the underlying mathematics, build an item co-occurrence matrix from scratch in Pandas, and implement production-grade rule mining using mlxtend.

1. Transforming Transaction Logs into Baskets

Transactional databases store orders in a normalized vertical format: one row per product line-item. To perform market basket analysis, we must transform these event rows into a horizontal Basket Matrix, where each row represents a distinct order and each column represents a boolean flag ($1$ if purchased, $0$ otherwise).

Let's simulate a retail dataset of 1,000 grocery orders with intentionally injected co-occurrence patterns (e.g., Bread & Butter, Chips & Salsa, Pizza & Coke):

import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
import seaborn as sns
 
# Set random seed for reproducibility
random.seed(42)
np.random.seed(42)
 
products = [
    'Milk', 'Bread', 'Butter', 'Eggs', 'Cheese',
    'Chips', 'Salsa', 'Beer', 'Diapers', 'Coke', 'Pizza', 'Apples'
]
 
transactions = []
order_id = 1
 
for _ in range(1000):
    # Sample random basket size between 1 and 5 items
    basket_size = random.randint(1, 5)
    basket = random.sample(products, basket_size)
    
    # Inject real-world cross-sell behavior
    # 1. If Bread is in basket, 70% chance to add Butter
    if 'Bread' in basket and 'Butter' not in basket and random.random() < 0.70:
        basket.append('Butter')
        
    # 2. If Chips is in basket, 60% chance to add Salsa
    if 'Chips' in basket and 'Salsa' not in basket and random.random() < 0.60:
        basket.append('Salsa')
        
    # 3. If Pizza is in basket, 80% chance to add Coke
    if 'Pizza' in basket and 'Coke' not in basket and random.random() < 0.80:
        basket.append('Coke')
        
    for item in basket:
        transactions.append({'OrderID': order_id, 'Product': item})
        
    order_id += 1
 
df_orders = pd.DataFrame(transactions)
print(f"Total Line Items: {len(df_orders):,}")
print(f"Unique Orders: {df_orders['OrderID'].nunique():,}")
df_orders.head(8)

Raw Transaction Sample

One-Hot Encoding the Baskets

We can pivot this transactional table into a binary basket matrix using Pandas pd.crosstab() or mlxtend.preprocessing.TransactionEncoder:

# Approach A: Pure Pandas Crosstab
basket_matrix = pd.crosstab(df_orders['OrderID'], df_orders['Product'])
 
# Clip counts to 1 (in case a customer bought multiple quantities of the same item)
basket_matrix = basket_matrix.clip(upper=1)
basket_matrix.head(6)

Encoded Basket Matrix

2. Product Co-Occurrence Matrix & Heatmap

Before running algorithmic rule mining, we can calculate how many times every pair of products appeared in the same order using vectorized matrix multiplication:

Co-occurrence Matrix = M^T · M

Where $M$ is the $(N \times P)$ binary basket matrix, and $M^T$ is its transpose.

# Matrix dot product gives co-occurrence counts
co_occurrence = basket_matrix.T.dot(basket_matrix)
 
# Set diagonal to 0 (we are interested in distinct item affinities, not Milk with Milk)
np.fill_diagonal(co_occurrence.values, 0)
 
# Visualize with Seaborn Heatmap
plt.figure(figsize=(10, 8))
sns.heatmap(
    co_occurrence,
    annot=True,
    fmt='d',
    cmap='Purples',
    linewidths=0.5,
    cbar_kws={'label': 'Co-occurrence Frequency'}
)
plt.title('Product Co-Occurrence Matrix (1,000 Transactions)', fontsize=14, pad=12)
plt.xlabel('Product B', fontsize=11)
plt.ylabel('Product A', fontsize=11)
plt.tight_layout()
plt.show()

Interpreting the Co-occurrence Counts

• Bread & Butter: Co-occurred 241 times
• Chips & Salsa: Co-occurred 218 times
• Pizza & Coke: Co-occurred 234 times
• Beer & Apples: Co-occurred only 29 times (pure baseline chance)

While co-occurrence counts give a general view, raw numbers alone are misleading. If an item is purchased in 90% of all orders, it will co-occur with everything by chance. We need Association Rule Metrics.

3. Mathematical Foundations: Support, Confidence & Lift

Association rules are expressed in the form:

Antecedent (A) => Consequent (B)

Meaning: "If a customer purchases item A, they are likely to also purchase item B."

                 [ All Transactions: N = 1,000 ]
  +-------------------------------------------------------------+
  |                                                             |
  |     +--------------------+       +--------------------+     |
  |     |   Transactions     |       |   Transactions     |     |
  |     |   with Item A      |       |   with Item B      |     |
  |     |                    |       |                    |     |
  |     |        +-----------+-------+-----------+        |     |
  |     |        |   Transactions with BOTH      |        |     |
  |     |        |        ( A ∩ B )              |        |     |
  |     |        |   SUPPORT = Count(A∩B) / N    |        |     |
  |     |        +-----------+-------+-----------+        |     |
  |     +--------------------+       +--------------------+     |
  |                                                             |
  +-------------------------------------------------------------+

1. Support

Support measures the popularity of an itemset. It represents the proportion of total transactions that contain both items A and B:

Support(A => B) = P(A ∩ B) = Count(A and B) / Total Transactions (N)

• High Support: The rule applies to a large percentage of total customers.
• Low Support: The combination is rare or niche.

2. Confidence

Confidence measures reliability. Given that a customer has purchased item A, what is the conditional probability that they will also purchase item B?

Confidence(A => B) = P(B|A) = Support(A ∩ B) / Support(A) = Count(A and B) / Count(A)

[!WARNING] Confidence is asymmetric: Confidence(A => B) ≠ Confidence(B => A). For instance, 80% of customers who buy Caviar might also buy Champagne, but only 15% of Champagne buyers purchase Caviar.

3. Lift

Lift measures the strength of association while controlling for how popular item B is overall. It is the ratio of observed co-occurrence to the expected co-occurrence if A and B were completely independent:

Lift(A => B) = P(B|A) / P(B) = Support(A ∩ B) / (Support(A) * Support(B))

4. Leverage & Conviction

• Leverage: Support(A ∩ B) - (Support(A) * Support(B)). Measures the difference in probability above random independence (value of 0 means independence).
• Conviction: (1 - Support(B)) / (1 - Confidence(A => B)). Measures the degree of implication; infinite value means the rule holds 100% of the time with zero exceptions.

4. Mining Rules with mlxtend (Apriori & FP-Growth)

The Python library mlxtend is the industry standard for frequent pattern mining. It implements both the Apriori algorithm and FP-Growth.

pip install mlxtend

from mlxtend.frequent_patterns import apriori, association_rules
 
# Step 1: Mine frequent itemsets with minimum support threshold of 4%
frequent_itemsets = apriori(
    basket_matrix,
    min_support=0.04,
    use_colnames=True
)
 
# Step 2: Generate association rules evaluated on Lift >= 1.2
rules = association_rules(
    frequent_itemsets,
    metric="lift",
    min_threshold=1.2
)
 
# Step 3: Clean, sort, and display top actionable cross-sell rules
rules['antecedents_str'] = rules['antecedents'].apply(lambda x: ', '.join(list(x)))
rules['consequents_str'] = rules['consequents'].apply(lambda x: ', '.join(list(x)))
 
display_cols = [
    'antecedents_str', 'consequents_str',
    'support', 'confidence', 'lift', 'leverage', 'conviction'
]
 
sorted_rules = rules[display_cols].sort_values(by='lift', ascending=False).reset_index(drop=True)
sorted_rules.head(6)

Association Rules Output Table

5. Building an Automated Cross-Sell Recommendation Engine

We can encapsulate our mined association rules into a reusable Python class that provides instantaneous product recommendations given any shopping cart contents:

class CrossSellRecommender:
    def __init__(self, rules_df):
        self.rules = rules_df
        
    def recommend(self, current_cart: list, min_confidence=0.5, top_n=3):
        """
        Given a list of items in the customer's cart, find top recommended cross-sells.
        """
        recommendations = []
        cart_set = set(current_cart)
        
        for _, row in self.rules.iterrows():
            antecedent_set = set(row['antecedents'])
            consequent_set = set(row['consequents'])
            
            # If the antecedent is fully contained within the cart
            # and the consequent is not already in the cart:
            if antecedent_set.issubset(cart_set) and not consequent_set.issubset(cart_set):
                if row['confidence'] >= min_confidence and row['lift'] > 1.0:
                    for item in consequent_set:
                        recommendations.append({
                            'recommended_product': item,
                            'triggered_by': ', '.join(antecedent_set),
                            'confidence': round(row['confidence'], 3),
                            'lift': round(row['lift'], 2)
                        })
                        
        if not recommendations:
            return pd.DataFrame(columns=['recommended_product', 'triggered_by', 'confidence', 'lift'])
            
        rec_df = pd.DataFrame(recommendations)
        # Deduplicate recommendations taking the highest lift rule
        rec_df = rec_df.sort_values(by=['lift', 'confidence'], ascending=False).drop_duplicates(subset=['recommended_product'])
        return rec_df.head(top_n).reset_index(drop=True)
 
# Initialize engine
engine = CrossSellRecommender(rules)
 
# Test Scenario 1: Customer adds Pizza to cart
print("Cart: ['Pizza']")
display(engine.recommend(['Pizza']))
 
# Test Scenario 2: Customer adds Bread and Chips to cart
print("\nCart: ['Bread', 'Chips']")
display(engine.recommend(['Bread', 'Chips']))

Recommendation Engine Output

Cart: ['Pizza']

Cart: ['Bread', 'Chips']

6. Strategic Business Applications & Pitfalls

Merchandising Strategies

1. Dynamic Bundling: Offer automated discounts when complementary pairs are added together (e.g., "Save 15% when you buy Pizza + Coke").
2. Checkout Modal Optimization: When a customer clicks Proceed to Checkout with item A, trigger a one-click add-on modal offering item B if Confidence >= 0.70 and Lift >= 2.0.
3. Physical Store Layout (Aisle Separation): In supermarkets, high-lift pairs are often placed at opposite ends of the store (e.g., Bread in Aisle 1, Butter in Aisle 8) to force shoppers to walk through intermediate aisles, driving impulse purchases.

Critical Pitfalls to Avoid

• The Milk Trap (High Confidence, Meaningless Lift): Common staples appear in 50%+ of baskets. Every item will show high confidence pointing to Milk. Always filter by Lift > 1.2, never confidence alone.
• Cannibalization / Substitutes (Lift < 1.0): Never recommend Coke Zero when a user adds Diet Coke to their cart. These are substitute products with negative correlation.
• Sparse Catalogs: If your store has 50,000 SKUs with low transaction volume per item, aggregate products into hierarchical categories (e.g., Beverages → Carbonated Soft Drinks) before mining association rules.

Summary & Next Steps

Market Basket Analysis bridges the gap between descriptive statistics and actionable machine learning:

1. Transform transaction records into binary basket matrices with pd.crosstab().
2. Evaluate rule strength using Support (P(A ∩ B)), Confidence (P(B|A)), and Lift (P(B|A) / P(B)).
3. Mine high-velocity patterns efficiently using the mlxtend Apriori and FP-Growth engines.
4. Deploy production recommendation engines to maximize average order value.

Continue building your practical data toolkit with our related guides:

• Market Basket Analysis Playbook — End-to-end implementation playbook for retail cross-selling and merchandising optimization.
• Data Analyst Roadmap 2026 — Step-by-step career path and essential project portfolio skills.
• Python Pandas for Data Analysis Guide — Mastering vectorization, merging, and grouping.
• Pandas vs SQL Performance & Syntax — Choosing the right tool for large-scale data wrangling.