This lesson on Feature Engineering and Categorical Encoding is hands-on and example-driven. You will be able to classify categorical data into nominal and ordinal types, which determines the appropriate encoding strategy for machine learning models. You will differentiate between Label Encoding and One-Hot Encoding, recognizing the limitations and use cases for each technique. This knowledge is crucial for preparing categorical features effectively.
What You'll Be Able To Do
- Classify categorical data into nominal and ordinal types.
- Explain why machine learning algorithms require feature encoding.
- Compare the mechanics of Label Encoding versus One-Hot Encoding.
- Identify the limitations associated with Label Encoding (priority issue).
- Describe the Dummy Variable Trap and its relation to multicollinearity.
- Determine when to use One-Hot Encoding to avoid unintended ranking.
Topics Covered in Feature Engineering and Categorical Encoding
- Data Classification (1:00 - 2:30) — Data is classified into categorical (qualitative) and numerical (quantitative) types.
- Nominal vs Ordinal Data (2:30 - 4:30) — Nominal data lacks order while ordinal data has ordered categories with unknown distances between them.
- Need for Encoding (4:30 - 5:45) — Categorical variables must be converted to numerical values because most machine learning algorithms cannot handle them directly.
- Label Encoding Process (5:45 - 7:45) — Label Encoding converts categories into unique numeric forms starting from zero, making data machine readable.
- Label Encoding Limitation (7:45 - 9:30) — The limitation is the priority issue, where the model incorrectly assigns rank based on the assigned numerical value.
- One-Hot Encoding Process (9:30 - 11:30) — One-Hot Encoding avoids the ranking issue by creating dummy variables, representing each category as a binary vector.
- One-Hot Encoding Limitation (11:30 - 13:30) — The limitation is the Dummy Variable Trap, which causes multicollinearity due to high correlation between the resulting features.
- Fixing Multicollinearity (13:30 - 14:00) — To overcome multicollinearity, one of the dummy variables must be dropped, often checked using the Variance Inflation Factor (VIF).
SQL Cheat Sheet
-
Categorical Data— Data divided into groups, like sex or educational levelSELECT DISTINCT sex FROM users; -
Nominal Data— Categories without inherent order or quantitative valueSELECT DISTINCT eye_color FROM users; -
Ordinal Data— Categories with a natural order, but unknown distanceSELECT * FROM surveys ORDER BY income_level; -
Label Encoding— Converts categories to unique numerical integers starting at zeroSELECT country, ENCODE(country) AS country_id FROM data; -
One-Hot Encoding— Creates dummy variables, representing categories as binary vectorsSELECT fruit, CASE WHEN fruit='Apple' THEN 1 ELSE 0 END AS is_apple FROM inventory; -
Multicollinearity— High correlation between independent features in a modelSELECT VIF(feature1, feature2) FROM model_stats;
Comparison Table
| Feature | Label Encoding | One-Hot Encoding |
|---|---|---|
| Best Use Case | Ordinal data (ordered categories) | Nominal data (unordered categories) |
| Output Format | Single integer column | Multiple binary columns (dummy variables) |
| Primary Limitation | Creates artificial priority/rank issue | Dummy Variable Trap/Multicollinearity |
| Feature Count | Low (avoids dimensionality) | High (increases dimensionality) |
Common Pitfalls
- Mistake: Using Label Encoding on nominal data like country names. Avoid: This creates an artificial priority issue where higher numbers are ranked higher.
- Mistake: Using all dummy variables created by One-Hot Encoding in a model. Avoid: Drop one dummy variable to prevent the Dummy Variable Trap and multicollinearity.
- Mistake: Assuming all ML algorithms require categorical encoding. Avoid: Some algorithms inherently handle categorical variables without explicit conversion.
- Mistake: Ignoring multicollinearity in linear or logistic regression models. Avoid: Check for multicollinearity using the Variance Inflation Factor (VIF).
FAQs
- Why must categorical variables be encoded? Machine learning algorithms generally require numerical input to process data and calculate relationships effectively. Encoding converts categories into a machine-readable numerical format.
- What is the Dummy Variable Trap? It occurs when dummy variables are highly correlated, meaning one variable can be predicted from the others. This causes multicollinearity, which destabilizes regression models.
- How do you fix multicollinearity caused by One-Hot Encoding? Drop one of the dummy variables created for a category. This ensures independence among the remaining features and resolves the trap.
- Is Label Encoding always bad? No, it is appropriate for ordinal data where the numerical order reflects the inherent category order, such as low, medium, and high income levels.
🔧 Leakage-Safe Preprocessing with Scikit-Learn Pipelines
Ad-hoc fit_transform on the full dataset is the most common leakage vector
in production code. The fix is structural:
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
pre = ColumnTransformer([
("num", Pipeline([("imp", SimpleImputer(strategy="median")),
("sc", StandardScaler())]), num_cols),
("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
("oh", OneHotEncoder(handle_unknown="ignore"))]), cat_cols),
])
model = Pipeline([("pre", pre), ("clf", YourEstimator())])
# Cross-validate `model` — preprocessing refits inside every fold. No leakage.
handle_unknown="ignore"keeps unseen production categories from crashing inference. High-cardinality text/city columns: target-encode inside folds or hash — never one-hot 5,000 levels.- Tree models skip the scaler; linear/distance models require it.