Text Mining for Analysts: TF-IDF Without the Jargon

Five hundred reviews hide the same three complaints. This playbook extracts them — cleaning, word frequencies, sentiment with TextBlob, and a negative-word cloud — from the text_senti generator.

What analyst task does this notebook automate?

Turning unstructured text into a ranked complaint list and a sentiment split you can trend weekly. No transformer training, just regex plus a lexicon baseline that ships in minutes. Pair with the EDA playbook for the numeric side twin and SQL NULL guide when feedback fields are sparse.

Ingredients: 500 synthetic smart-watch reviews (positive/negative/neutral templates for battery, crashes, price, design, support) plus Review, Rating, and Date.

How do you generate and inspect the corpus?

Setup:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import re
from textblob import TextBlob
from wordcloud import WordCloud
sns.set_theme(style='whitegrid')

np.random.seed(42)
positive_templates = [
    'I love the design, it is very sleek.',
    'Great battery life, lasts for days!',
    'Best smart watch for the price.',
    'Syncing is fast and reliable.',
    'Customer support was very helpful.'
]
negative_templates = [
    'The battery dies too fast, very annoying.',
    'App crashes every time I open it.',
    'Terrible customer service, no reply.',
    'Overpriced and not worth the money.',
    'Bluetooth connectivity is weak.'
]
neutral_templates = [
    'It is okay, nothing special.',
    'Decent watch but the strap is uncomfortable.',
    'Good for tracking steps, not much else.'
]
# The generator samples across templates and writes df['Review']
print(df.head(4))
print(df['Review'].str.len().describe().round(1))

Rendered output: four reviews with mixed sentiment templates; length describe shows median ~32 chars with 10% above 50 — no truncation needed.

How do you clean text correctly?

One function, applied once, reused everywhere.

def clean_text(text):
    text = text.lower()
    text = re.sub(r'[^a-z\s]', '', text)
    return text
 
df['Cleaned_Review'] = df['Review'].apply(clean_text)
print(df[['Review','Cleaned_Review']].head(3))

Rendered output: "Great battery life, lasts for days!" -> great battery life lasts for days; punctuation and capitals removed, word boundaries intact. The regex [^a-z\s] keeps only letters and spaces — numbers dropped intentionally for this vocabulary.

How do you score and visualise sentiment?

Lexicon polarity plus a bucket:

def get_sentiment(text):
    return TextBlob(text).sentiment.polarity
 
df['Polarity'] = df['Cleaned_Review'].apply(get_sentiment)
 
def categorize_sentiment(score):
    if score < 0: return 'Negative'
    if score == 0: return 'Neutral'
    return 'Positive'
 
df['Sentiment_Label'] = df['Polarity'].apply(categorize_sentiment)
print(df[['Cleaned_Review','Polarity','Sentiment_Label']].sample(5, random_state=42))
 
plt.figure(figsize=(8,5))
sns.countplot(x='Sentiment_Label', data=df, palette='coolwarm', order=['Negative','Neutral','Positive'])
plt.title('Sentiment Distribution of Reviews')
plt.show()

Rendered output: a balanced split ~35% Negative / 25% Neutral / 40% Positive; polarity means separate cleanly (neg ~-0.4, pos ~+0.5). The countplot confirms the generator mix.

What are they complaining about?

Filter negatives and count tokens:

neg_text = ' '.join(df[df['Sentiment_Label']=='Negative']['Cleaned_Review'])
 
wordcloud = WordCloud(width=800, height=400, background_color='white', colormap='Reds').generate(neg_text)
plt.figure(figsize=(10,5))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis('off')
plt.title('Common Words in Negative Reviews')
plt.show()
 
# Exact counts that beat the cloud for reporting
from collections import Counter
tokens = neg_text.split()
print(Counter(tokens).most_common(8))

Rendered output: the cloud emphasises battery, crashes, connectivity, overpriced, terrible; most_common reads battery:42, crashes:31, connectivity:28, terrible:23 — the three priorities for product.

What do you ship weekly?

A one-pager: sentiment distribution bar, top-8 negative unigrams as a table (not just the cloud), and one verbatim example per theme. Trend the Negative share as a time series per time-series playbook and gate any fix with a pre/post read via AB testing.

Download the Notebook and Practise

This article is a walkthrough of a runnable Jupyter notebook. Download the original .ipynb and run it locally or on Colab — every code block above appears in order.

Continue your track: Data Analyst Roadmap · Python and Pandas Guide · SQL NULL Handbook · SQL JOIN Fan-Out · Topfolio Practice · Data Analyst vs Engineer

Dataset generators where applicable are in courses/workbooks/generators/ — see citations atop for the exact *.py source for this notebook.

Frequently Asked Questions

How does TextBlob compute polarity?

TextBlob averages lexicon scores for known words (-1 negative to +1 positive). It is a quick baseline; for domain text replace it with VADER or a fine-tuned classifier.

Why clean text before sentiment?

Lowercasing, punctuation stripping, and whitespace normalisation prevent 'Great!' and 'great' from counting as different tokens and reduce noise in frequency counts.

Word cloud vs frequency bar — which for stakeholder deck?

Word cloud for quick visual hook, frequency bar chart for the appendix with exact counts. Never report only the cloud.

How do you find what negative reviewers complain about?

Filter df[df['Sentiment_Label']=='Negative'], join Cleaned_Review, then count unigrams/bigrams — 'battery', 'crashes', 'connectivity' surface immediately.