Back to Inference: Hypothesis Testing and Beyond

Chi-Square and Categorical Data

The right test when both variables are categorical. Two forms — independence and goodness-of-fit. FIND_VIDEO: search 'chi-square test independence goodness of fit' — recommended channel: StatQuest. Aim for 10 min or under.

21 minutesVideo LessonPDF notes

Watch Chi-Square and Categorical Data on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Chi-Squared Introduction — The Chi-Squared test is introduced as a statistical method for feature selection, often used in univariate or bivariate analysis.
Test Application Criteria — Chi-Squared computes statistics between non-negative features and the target class, specifically for categorical variables in classification.
Data Setup and Selection — The Titanic dataset is loaded, and relevant categorical features are selected alongside the target variable, 'survived'.
Label Encoding Requirement — It is stated that label encoding must be performed on all categorical features before applying the Chi-Squared test.
Encoding Binary Features — Label encoding is demonstrated for the binary 'sex' column using `np.where` to convert 'male' and 'female' to 1 and 0.
Encoding Multi-Category Features — A dictionary comprehension approach is used to assign ordinal labels to the multi-category 'embarked' column.
Encoding Boolean Features — The 'alone' boolean column is label encoded into 1s and 0s using `np.where`.
Train Test Split — Train-test split is implemented using `sklearn` to separate the data into training sets, which is necessary to avoid overfitting.
Performing Chi-Squared — The `sklearn.feature_selection.chi2` function is imported and prepared to run on the training data.
Interpreting Chi-Squared Output — It is explained that the `chi2` function returns two values: the F-score, which is the statistic, and the P-value.

PDF notes

Bottom line: You will learn how to apply the Chi-Squared statistical test to select the most relevant categorical features for a classification task. You will be able to prepare raw categorical data using label encoding and perform train-test splits before calculating feature importance scores using `sklearn`. This process helps evaluate the relationship between non-negative features and the target class.

This lesson on Chi-Square and Categorical Data is hands-on and example-driven. You will learn how to apply the Chi-Squared statistical test to select the most relevant categorical features for a classification task. You will be able to prepare raw categorical data using label encoding and perform train-test splits before calculating feature importance scores using sklearn. This process helps evaluate the relationship between non-negative features and the target class.

What You'll Be Able To Do

Determine when the Chi-Squared test is appropriate for feature selection.
Prepare categorical features for Chi-Squared testing using label encoding.
Apply the sklearn.feature_selection.chi2 function to calculate feature scores.
Interpret the resulting F-score and P-value to evaluate feature relevance.
Implement train-test split to prevent model overfitting during feature evaluation.

Topics Covered in Chi-Square and Categorical Data

Chi-Squared Introduction (0:00 - 1:15) — The Chi-Squared test is introduced as a statistical method for feature selection, often used in univariate or bivariate analysis.
Test Application Criteria (1:15 - 2:40) — Chi-Squared computes statistics between non-negative features and the target class, specifically for categorical variables in classification.
Data Setup and Selection (2:40 - 4:30) — The Titanic dataset is loaded, and relevant categorical features are selected alongside the target variable, 'survived'.
Label Encoding Requirement (4:30 - 6:50) — It is stated that label encoding must be performed on all categorical features before applying the Chi-Squared test.
Encoding Binary Features (6:50 - 9:50) — Label encoding is demonstrated for the binary 'sex' column using np.where to convert 'male' and 'female' to 1 and 0.
Encoding Multi-Category Features (9:50 - 13:00) — A dictionary comprehension approach is used to assign ordinal labels to the multi-category 'embarked' column.
Encoding Boolean Features (13:00 - 14:30) — The 'alone' boolean column is label encoded into 1s and 0s using np.where.
Train Test Split (14:30 - 16:30) — Train-test split is implemented using sklearn to separate the data into training sets, which is necessary to avoid overfitting.
Performing Chi-Squared (16:30 - 18:00) — The sklearn.feature_selection.chi2 function is imported and prepared to run on the training data.
Interpreting Chi-Squared Output (18:00 - 19:30) — It is explained that the chi2 function returns two values: the F-score, which is the statistic, and the P-value.

SQL Cheat Sheet

Chi-Squared Test — Evaluates relationship between non-negative features and target class
```
from sklearn.feature_selection import chi2
```
Non-Negative Feature — Features containing only zero or positive values (e.g., counts, boolean)
```
df['sex'] = np.where(df['sex'] == 'male', 1, 0)
```
Label Encoding — Converts categorical text labels into numerical representations (0, 1, 2...)
```
df['sex'] = np.where(df['sex'] == 'male', 1, 0)
```
P-Value — Probability of observing data if the null hypothesis were true
- In practice: if p_value < 0.05: reject_null_hypothesis
sklearn.feature_selection.chi2 — Returns F-score (statistic) and P-value for feature selection
- In practice: f_p_value = chi2(X_train, Y_train)
Train Test Split — Separates data into training and testing sets to avoid overfitting
```
from sklearn.model_selection import train_test_split
```

Comparison Table

Statistical Test	Data Type	Primary Purpose
Chi-Squared	Categorical vs. Categorical	Feature selection in classification.
T-Test	Numerical vs. Categorical (2 groups)	Compare means of two groups.
ANOVA Test	Numerical vs. Categorical (>2 groups)	Compare means across multiple groups.

Common Pitfalls

Mistake: Applying Chi-Squared directly to string or object type categorical data. Avoid: Always perform label encoding to convert categories to non-negative integers first.
Mistake: Using Chi-Squared on features that contain negative values. Avoid: Ensure all input features (X) are non-negative, such as boolean or frequency counts.
Mistake: Forgetting to check for null values before applying the test. Avoid: Handle or impute NaN values, as they can cause the chi2 function to return NaN results.
Mistake: Skipping the train-test split before feature evaluation. Avoid: Split the data first to ensure feature selection generalizes and avoids overfitting.

FAQs

Why is label encoding mandatory before applying the Chi-Squared test? Chi-Squared requires numerical, non-negative input (like counts or discrete labels) to calculate the expected frequencies accurately.
What does it mean if the P-value is less than 0.05? A low P-value suggests the observed relationship between the feature and the target is statistically significant, meaning the feature is likely important.
Can I use Chi-Squared for regression tasks? No, the score is specifically designed to evaluate categorical variables in a classification task relative to the output class.

Common pitfalls

Mistake: Applying Chi-Squared directly to string or object type categorical data. Avoid: Always perform label encoding to convert categories to non-negative integers first.
Mistake: Using Chi-Squared on features that contain negative values. Avoid: Ensure all input features (X) are non-negative, such as boolean or frequency counts.
Mistake: Forgetting to check for null values before applying the test. Avoid: Handle or impute NaN values, as they can cause the chi2 function to return NaN results.
Mistake: Skipping the train-test split before feature evaluation. Avoid: Split the data first to ensure feature selection generalizes and avoids overfitting.

Hello all, my name is Krushna Naik, and welcome to my YouTube channel. So guys, we are going to continue the feature selection playlist and now we are going to see the tutorial five. And in tutorial five, we are going to see how to perform chi-squared test for feature selection. If you don't know about chi-squared test guys, it is basically a kind of statistical test itself. And if you really want to see a video regarding chi-squared and why it is actually used, I've already made a video about it with respect to different types of tests like t-test, chi-squared test and ANOVA test. You can consider like all this test are like univariate and bivariate test analysis. So please make sure that you watch this video. All the links will be given over here. I've already given the statistical test. This is our practical thing and this is our thing. And many people are actually confused with respect to t-test, all the different kind of tests and why it is used. Specifically, if I take as an example for feature selection, definitely it can be used, okay. So this is the sklearn page of chi-squared and we'll try to discuss this. This same thing I've copied and pasted over here to make it easily understandable for you. Now, why is chi-squared test used? Okay. So we compute chi-squared stats between each non-negative feature and class. Class is basically your target feature. Okay. Non-negative feature basically means features that are not having any negative values, okay. This score should be used to evaluate categorical variables in a classification task. Always remember, if you really need to evaluate or try to understand the relationship with respect to a category variable with respect to the output, at that time we use chi-squared test. And if you remember about chi-squared test, we'll also be understanding about p-value, you know, p-value. When the value is less than 0.05, then we basically say to reject or accept the null hypothesis, based on that it will be coming up over here, okay. And apart from that, you can see, this core can be used to select the n-underscore features with the highest value for the test chi-squared statistics from X, which must contain only non-negative features such as boolean or frequencies. You know, this kind of definitely, frequencies will also there, you know, it is just like a discrete count, you know, and term counts in document classification relative to the classes. Always remember, based on your output feature, we are trying to find out this particular relationship and we are trying to evaluate this, okay. And again, you can see all this particular description. But let's me go ahead and try to show you a practical example through which we can perform a chi-squared test. And here, I'm just going to quickly, you know, quickly show you a very good technique how to actually find it out and what all things we should actually take care of, you know. And based on that, we will be doing it. Now, first of all, where is the dataset actually coming from? So to begin with, guys, we will be taking the normal Titanic dataset because see, in Titanic dataset, you have a lot of category variables, right. So for that, what I'm going to do quickly is that I'm going to import seaborn seaborn as sns and after this, I'll be using sns.load_dataset and here I'm actually going to consider Titanic dataset, okay. And if you don't know about seaborn, it is a visualization library. It has a lot of default dataset that is present inside this. So I'm just going to use df and probably let's go and check out my df.head. I'm just going to close this. It has got executed successfully. So let me write df.head. Once I execute this, here you can see a lot of features. Now, let's see which all are category features in this particular case. And in order to see it, what I can do is that I can just select because I've already worked with this particular dataset. So let's consider one sex column as a category feature, p-class as a category feature, right, because here you have one, two, three classes, right. If you have already worked in the EDA part of exploratory data analysis in Titanic dataset, you basically have three variables. Then I may also take embarked as my column. I may also take probably there is something called as alone. Alone also is probably a category feature. So I'll be considering all these category features and then I'll take the survived column also and then we'll try to find out how we can use feature selection. Like which are the most important features from those categories. So let's go ahead. So before that, if you really want to see some more analysis, you can write df.info. And here you are actually getting your df.info. Now, what I'll do is that remember, whenever you have your category features. So first of all, here the category features that I'm going to select, I'll just list it down in a comment section. So I'm going to take the category features like sex, probably embarked. I'm just writing it down so that it makes you, it gives you a clear understanding, alone, probably alone, and then this is your passenger class, p-class. And after this, your final class, which is called as survived, which is your output feature because we need to compare these all categories. Which all are the important categories when compared to the output category that is survived, right. So we are going to consider all these things. Now, what I'm going to do quickly is that a very, very simple thing that I'm going to do is that I'll just write df is equal to df of all these columns, right. So, I'll just copy it over here. So, I'll just copy it and paste it over here. Okay. So, I've just copied and pasted it over here. And now, if I write df.head, probably I think survived column will be fine because if I have survived, okay, it is in small letters. So, I'll make it like this. So, this is my df.head now. Now, remember, this is my output feature. This is my all the category features, right. And I am trying to basically evaluate this category features based on this output feature, right, which is my classes, like survived, zero or one. Now, always remember, guys, before we apply chi-squared test, we have to perform label encoding on each and every feature. Okay. This is the first condition before applying a chi-squared test. We need to apply label encoding in each and every feature. So, first of all, let me write it as label encoding. How do we perform label encoding? So, I'll just write down a comment. Let's perform label encoding. I think label encoding is very, very simple. I'll just use numpy, if possible. Let's perform label encoding on sex column. And these all are very, very important, guys. You will also be able to do it because many people are asking how to evaluate category features. Here it is, right. So, here you have this and here I'm just going to write it as sex. Now, before this, what I'm going to do, I'm also going to import numpy library because I'll be using numpy. So, df of x is equal to np.where. I'll just write np.where condition and I'll write df of sex, wherever the sex is equal to, I think sex has male and female, right. So, I'll write male, something like this, then whenever this is true, I'll make it as one, else I'll make it as zero. So, this is how, if I execute it, probably this particular entire column where male is one, where where the value is male, it will become one, else it will become zero. So, I'm just doing some kind of label encoding. In label encoding, what we do, we assign some labels. If you have three categories, it will become zero, one, two. So, something like that. The next category, sorry, next category is that let's let's perform label encoding on which column? Embark column. Now, in embark column, it is like you have multiple categories. It may be more than two categories. Now, in order to make this much more generic, I'm just going to write a very simple code which looks something like this. First of all, I will do something like this. See this, what I'm doing is that I'm using this dictionary comprehension where I'm saying that, uh, just assign the labels in this embark column based on the number of values, right. First of all, I'll create ordinal labels. Ordinal label basically make sure that how many number of unique labels are here. And based on that, if I just try to execute this, let's see. Okay. I'll just try to execute this and if I try to write this ordinal label, just a second, ordinal label. If I try to see the output, here you have a key value pairs with respect to dictionary, like k is equal to i, k this i is nothing but it is iterating through all the unique values inside this embark column. So, total number of categories or labels I can see is zero, one, two and three and there are some nan values also. So, nan will be considered as my third category. So, what I'll do is that now I'll assign this ordinal values with the embark column. So, wherever there will be s, it will be replaced by zero. Wherever there will be c, it will be replaced by one. Wherever the value is q, it will be replaced by two. And like this, we are actually creating some you can consider that we are actually creating this kind of values, you know, with respect to label encoding. So, in my embark column, now, if I go and see my df.head, okay, here you'll be able to see in embark column, you'll be having zero, one, two, three and whenever there are value as q or nan, it will be two or three, right. Now, this was done. Now, what we have? We also have alone column. So, let me also do it for alone column. So, what I'm going to do, let's perform label encoding on alone, on alone column. Okay. After doing this, what I'm going to do is that I'm probably going to use the same code. df of alone and I'll write np.where wherever df of alone is probably false. I think it is a boolean value. Let's see. boolean value. How do we check whether it is a boolean value or not? What I'll do is that I will just write, probably I've seen this, okay. In df.info, if we see alone, it is a nothing but a boolean value, right. So, I'm just going to write it as false, okay. So, I'll say, okay, wherever this is true, let it, let's make it as one, else let's make it as zero. Okay. Now, if I try to execute and if I try to see my df.head, so, here now we have done all the label encoding perfectly. The passenger class is already label encoded as zero, one, two, three or one, two, three, right. So, we don't have to worry about this. Now, comes an opportunity for us to perform chi-square. But before that, what we are going to do is that we are going to do train test split. Train test split. And remember, in train test split is usually done to avoid overfitting. Usually done to avoid overfitting. So, in order to avoid overfitting, we are actually doing train test split. And for that, obviously everybody knows how to do it. From sklearn.model_selection, I'm going to import train test split. Now, see this, guys, it's like I've been using this for many days now. Probably if you also try to do it, you'll also be able to write so quickly and that is all from practice, okay. So, I'm just going to use this all things, uh, over here. Now, basically, I'll write my df of of this all values, my independent features as my x-axis, right. So, here I will write df of, okay, one more bracket, and here instead of taking all this particular features, see, here we had taken all these features. Now, I'll just take it till passenger class, okay. So, I'll copy it till passenger class till here, right, perfect. Now, if I go to my y-axis, which I'll write in the next line, it will be nothing but df of survived, right. Survived. And after this, I'll use my test size. test_size is equal to 0.3 and my random state. It can be anything. random_state is equal to zero. Once I execute it, it has got executed successfully. Nothing. I've just done a train test split. Now, if I go and see my X_train.head, you'll be seeing other than survived column, you have everything. Now, the next statement comes is that how do we perform chi-squared test. Now, in order to perform perform chi-squared test, anyhow, I had, please make sure that you watch the theoretical video, guys. chi-squared, let me write it as chi-squared test. So, performing chi-squared test is very, very simple. You import from sklearn.feature_selection. I'm going to import the chi-squared values, right, chi-squared library. Now, inside the chi-squared, what I'm going to do quickly is that I am going to just check. But before that, let's see whether there is any null values inside my X_train because I've not handled any null values. So, in order to test it, I'll write isnull. isnull.sum and once I execute it, here I can see, okay, no null values, fine. So, here I'll write X_train, comma Y_train, okay. So, I'm just going to apply chi-square on X_train and Y_train. And remember, this will return two values. Chi-square, I'll just write down a comment. So, chi-square returns two values. One is, one, the first one is basically your f-score and the p-value, right. So, it returns this two value. Just try to see what is p-value. I've already made a video about it, okay, what is p-value and all. So, it is basically saying with respect to the probability, mostly, right. And here, suppose I'll say f_p_value. I'm just going to assign over here. Remember, once I execute this, guys, if you go and see your f_p_values, here you'll be getting two things. Okay. Why you are getting nan? Because let's see. Okay. Sex column, okay, sex column is all having only one value. Let me just do again a train test split and let me see. See, that nan had usually come because here if I go and see my X_train.unique, sorry, X_train of sex.unique, then probably I have received all the unique values over here. Now, see, only one value is there. Let me just execute it once again. Okay. Male, male and female, okay. Let me just execute it once again. Probably I'll do multiple times train test split. Okay. Here also, let me just use some random state. And let me just remove random state as 100 because any time I do same thing will come, right. Okay, still no values. No values. Okay. The problem is that in the sex column, right, initially, let me see what mistake we have done. Where df of sex is equal to 1, 0. So, if I'm checking df of sex, why it is not coming as 1? Okay. The problem is I understood what is the problem. Let me just execute it once again from here. Let me just quickly do it, guys. I think we made some mistake and probably because of that, okay. Let me just see this code, okay. So, I'm just going to remove this and let's see what are what is the output that I'll get. 1, 0, 0, okay, fine. This works fine. Now, if I write df of sex and just remove it, and if I go and see df.head, probably it should get updated. I don't know, like some mistakes has happened. This is working fine. So, I'm just going to remove this now. Let's perform this ordinal mapping. Now, if I go and see df.head, probably it works fine. Yes, yes, yes. df of alone is there. Probably alone, I'll just execute it. Now, if I go and see my df.head, now I'm getting it right, right? 1, 0. I don't know, some some mistakes happened. So, because of that, we were getting all zeros over here. So, let's execute it once again. Now, if I go and see my X_train.head, and see 1, 0 has come. Okay, perfect. So, if I go and check my null values, no null values. Now, I will perform my chi-squared test. And finally, this this this chi-squared will give two values. One is the f-score. The first value that you see in this array are your f-scores, okay. This f-scores needs to be higher. The more the higher the values of the f-score, the more important the feature is. Then the second array parameters or array values that you are seeing is basically p-value. And we know that p-value, the lesser the p-value, the more important that particular feature is, okay. So, here you'll be able to see 10 raised to minus 6. Here you'll be able to see 10 raised to minus 4, 10 raised to minus 3, 10 raised to minus 16. So, out of this, which is the most important feature? Definitely this one, right? So, in order to check which is that particular feature, I'm going to quickly convert this into. So, pandas is already there. We'll convert this into series. So, here I will be taking my f_p_value of one, right? If I if I use like this of one, we are basically considering the p-value. One point you have to note that, guys, the lower the p-value, the more important that particular feature is, okay. And suppose if you're doing it for a huge dataset, at that time it is very, very difficult to see this particular values because you won't be able to plot it. So, what we do is that we try to sort this particular values. And in order to sort it, I will just use it. See, this is my p-values, okay. This is my p-values. I'm just trying to make it as a series, okay. And then what I'll do is that I'll I'll also make sure that p_values.index is equal to X_train columns. I'm just going to also keep my index over here. And if I go and see my p-values now and execute it, oh, okay, sorry. It's p_values, sorry. p_values, okay. Okay, pandas, we have not yet used, right? So, let me just import pandas also. import pandas as pd. So, once I execute it, here you'll be able to see that, okay, we are getting this particular value. But what I want is that, probably I want to also sort this, okay, let's sort it. So, I'll say p_values.sort_index, okay. And I'll I'll just say that ascending is equal to true. So, these all values. Now, you can see that out of this, when you're sorting as ascending is equal to true. If I make ascending is equal to false, that basically means from smaller number to the larger number, we will be getting displayed over here. Now, here you can see that the smaller p-value is nothing but 10 raised to minus 16. Now, this indicates, here I'm just going to write some observation. The observation says that sex column sex column is the most is the most important important column when compared to when compared to the output the output feature feature. You should always write this observation, okay, survived, survived column. Now, just understand, guys. Let's let's let's see that why these results are very, very much valid. I hope everybody has seen Titanic movie and you all know that, if you do EDA also, you definitely know that men died a lot, right, when compared to females because female were actually sent in the boat, right? Now, here you are able to find out sex is really, really playing a major role. Since this p-value is very less, it is really, really playing a major role with respect to your output feature. So, this is the most important column in able to predict the survived column. You can understand in this particular way and this is obviously true because we have probably, you know, we have actually made this observation also, right? We we have made this observation. Let me just make this as a markdown quickly and let me make this as a code, right? We have seen this and it's it's awesome, right? Because we are able to understand this. Then, obviously, passenger class. See, we know that in passenger class one and two, people died less when compared to passenger class three because they were stuck inside the boat. They were staying somewhere beneath the boat, right? In the last floor, something like that, right? Then you had embarked, then you had alone, right? So, these all features are basically saying whatever we did in the EDA, right, of Titanic dataset. If you have not seen, check out my complete machine learning playlist. There you will be able to find it out. And this observation looks amazing for the feature selection. So, chi-squared definitely is giving us a very good result, right? So, I would suggest just try to do it by yourself and yes, I'll see you all in the next video. Have a great day ahead. And remember, guys, this is my feature selection playlist. Again, the link will be given in the description and even in the comment. This is the tutorial fifth that we have seen and chi-squared is also called as fisher score. Okay. It is also named as fisher score. So, I hope you like this particular video. I'll see you all in the next video. Have a great day. Thank you, all. Bye-bye.

Frequently asked questions

Why is label encoding mandatory before applying the Chi-Squared test?

Chi-Squared requires numerical, non-negative input (like counts or discrete labels) to calculate the expected frequencies accurately.

What does it mean if the P-value is less than 0.05?

A low P-value suggests the observed relationship between the feature and the target is statistically significant, meaning the feature is likely important.

Can I use Chi-Squared for regression tasks?

No, the score is specifically designed to evaluate categorical variables in a classification task relative to the output class.