Class Imbalance — SMOTE, Class Weights, Threshold Adjustment

Let's explore another popular data balancing technique that is Smote. Smote or Synthetic Minority Oversampling Technique differs from the other data balancing methods because it creates new synthetic data points for the minority class instead of just simply duplicating the existing ones. This approach helps to maintain the original data diversity, reduces overfitting, and improve the model performance across various algorithms and data sets. Let's have a theoretical as well as practical understanding of Smote. What is Smote? Smote is a synthetic minority oversampling technique and is a popular algorithm used in a machine learning to address the problem of imbalanced data sets. But why Smote? Smote can help to reduce overfitting of the model to the majority class by creating synthetic samples from the minority class that are similar but not identical to the original samples belonging to the minority class. This can prevent the model from memorizing the training data and generalize in a better way to the new data. But how these synthetic samples or synthetic examples getting generated with the help of minority class examples? Consider this is the minority class example, and this is the second minority class example. We are going to generate new synthetic example by interpolating these two minority class examples. Here we have new synthetic minority class example getting generated from the original minority class examples. Smote does not result in any loss of information from original data set as it generates synthetic samples based on the existing samples in the minority class. And also Smote can be used with various types of data including continuous and categorical variables. Consider an example of imbalance data set. Here we have class-wise data distribution is not equally distributed. The number of samples present in class 0 is comparatively higher than the number of samples present in class 1. But what is the miracle happening after applying Smote? Let's apply Smote. Once we apply Smote, we have balanced data. The number of samples present in the class 0 and the number of samples present in the class 1 are now same. But how Smote works? And what are the mathematical operation that Smote performs to balance the data set? Now we are going to see the step-by-step detailed information of how Smote works. In the step 1, Smote identify the minority class from the data set. Consider an example. Red circles belonging to the majority class and the blue circles belonging to the minority class. Now we are randomly select an instance from the minority class and for the selected instance the k-nearest neighbors are getting identified. The value of k is specified by the user and it represents the number of closest instances to be considered. Here we have these two neighbors of our original minority class example. Now let's move on to the step 2. Let's zoom in into this ellipse. X1 is the first neighbor of our original minority class example. And X2 is the second neighbor of the minority class example X. Now we are going to generate a synthetic example S1 from by interpolating X and X1. This is the newly generated synthetic example which is S1. S1 is generated with the help of this formula. S1 = X + R * (X - X1) where X and X1 are the N-dimension feature vector and R is the random number lies between 0 and 1. Shortly X - X1 is the difference between the features belonging to the data sample X and the features belonging to the data sample X1. Similarly, we generate S2 synthetic example for X and X2. That is by interpolating X and X2. But there are some issues with Smote. Smote can generate noisy synthetic example when the minority class examples located near to the majority class examples. Let's zoom into this ellipse. Here X and X1 belonging to the minority class and X2 belonging to the majority class. S1 is a synthetic data example generated by interpolating two minority class example X and X1 and S2 is a synthetic data example generated by interpolating X and X2. Here X2 is a majority class example. By interpolating minority class example which is X and the majority class example X2 S2 is a not good representative of our minority class examples. Here we have two synthetic examples. This also happens when there is a overlapping between minority class instance and the majority class instance and it result in a synthetic examples which are not good representative of minority class. Here we can see S2 is a not good representative of minority class because it is generated by interpolating minority class example X and the majority class example X2. To address these issues with Smote we have extension of Smote which is Borderline-Smote. Borderline-Smote divided into two parts. It work with Borderline minority examples as well as non-Borderline minority examples. Consider a data set. Here red samples belonging to the majority class and blue samples belonging to the minority class. These are the borderline examples and these are the non-borderline examples. We are applying operations performed in traditional Smote on the non-borderline minority examples and we apply Borderline-Smote on the borderline minority examples. Now let's see what are the steps that we perform in Borderline-Smote. In step 1, we first we identify minority class instances that are near to the majority class instances. It is also known as Borderline examples. Consider a data set where blue samples belonging to the minority class and the red samples belonging to the majority class and this is the Borderline examples. In the Borderline examples neighbors belong to both minority class as well as majority class. And this is done by applying k-nearest neighbor algorithm to the minority class instances and identifying those instances that have at least one majority class neighbor within their k-nearest neighbor. Now let's move on to the step 2. In step 2 for each Borderline example we randomly select one of its k-nearest neighbor from the minority class only and generate synthetic example by interpolating these two minority class examples only. Now let's zoom into the ellipse. Here we can see X and X1 belonging to the minority class and X2 belong to the majority class. We generate S1 with the help of X and X1. That is by interpolating X and X1 with this formula S1 = X + R * (X - X1). Here the red sample which is X2 belonging to the majority class and the blue sample X belonging to the minority class. We are not going to generate any synthetic example on this line because it is not good representative of minority class. So this is how Borderline-Smote tackle the problem of Borderline examples. Now let's move on to the practical implementation of Smote and Borderline-Smote in Python. Now let's move on to the part of practical implementation of Smote and Borderline-Smote. We have already conducted a modelling on imbalance data and also explored some techniques like random under-sampling and random oversampling that precisely used to address the problem of data imbalance. I highly recommend you to visiting that particular video that is on handling data imbalance using random oversampling and random under-sampling. The video will provide you the theoretical as well as practical knowledge of what is data imbalance, what are the consequences of data imbalance and how to handle data imbalance using random under-sampling and random oversampling. You have to visit that particular video because I am going to use the same Jupiter Notebook and the same data set. This is our data set. The data set is about vehicle insurance data. It is having two classes, class 0 and class 1. Class 0 contains this number of samples and class 1 contain this number of samples. From this you can able to understand our data set is imbalance data. I will share the link of this data set and my Jupiter code notebook in the video description so you'll be able to access my code as well as this data set. Now let's move on to the main part. Firstly we import our all the libraries and load our data set. This is our data. The ID column does not add any value in our data as well as in model learning process. So firstly we drop it and after dropping the ID column this is our data. And these are the number of samples belonging to the specific class. This is the class distribution and from bar graph you can able to understand our data set is an imbalance data. And again our data set contains some categorical variables so firstly we convert it into a number form with the help of label encoder. I have already explained this part in the video on handling data imbalance using random under-sampling and random oversampling. So please visit that particular video. We are going to use this particular architecture that is artificial neural network architecture and this is the function that we use for evaluating our model. So let's run this and again we load our model architecture with the help of this. Now let's move on to the Smote part. Before that I will show you some results of previous models. These are the results that we obtain from the models that we train on the original imbalance data. You can see these are the quite worst result. And again we perform some under-sampling on the majority class and after that we have these results. These are quite good results. Again we perform random oversampling and we found these results. Here you can see the F1 score after performing random under-sampling and random oversampling. These are quite good in number. Now let's move on to the Smote. Firstly we see our data. We have to run this part because we haven't encoded our categorical variable. Then our data is look like this. So now let's load our data. This is our data. Now the gender column and vehicle age and vehicle damage column is now in number form. Let's segregate the explanatory variables and the target variable. The target variable is response variable and the remaining variables are the explanatory variables. Firstly we segregate these two variables and after it we split it into a training and testing part. 80% is the training part and 20% is the testing part. We use stratify is equals to target variable because we want equal distribution of each classes in training data as well as testing data. So let's run this cell. Now we import Smote from imblearn package. It is a part of Python packages. Here you can see the Smote. The imblearn you have to just search imblearn on Google. You can able to access all about its documentation. Here you can see in the user guide you can able to access about Smote, everything about Smote. Now let's move on to the main part. Here we import Smote and again we load Smote like this. You have to pass two arguments that is sampling strategy and k-neighbors. I have put it here 10 and the sampling strategy is minority because I want to perform oversampling on the minority class. And again we resample the X train and Y train because we are going to perform oversampling on the training data itself, not on the testing data. So let's run this cell. Now let's see the class-wise data sample counts. Here you can see the class 1 and class 0 contains same number of samples. Now again we are going to normalize our data that is X train and X test data with the help of Min Max scaler. So let's run this. Now we are going to train our model with the help of this X train and Y train. So let's run this cell. Here we are going to use batch size is about 32 and number of epochs is 5. The model training process will take some time. So we'll meet after fifth epoch. Now our model training process is done. Let's take an inference on the testing data with the help of this nn.predict. nn is our model. After that we round it up. We are going to round up our probabilities. Let's run this. And now we are going to see our model accuracy with the help of this measure_accuracy function that we defined earlier. Let's run this. Here you can see the F1 score is quite good for both the classes. And also number of misclassified example for the minority class which is class 1 is very low in number. So after performing Smote we have good results. Now let's move on to the another part which is Borderline-Smote. So let's load our data. This is our data. Again we are going to split it into two parts that is training and testing. The same process that we have done in the Smote. Now import Borderline-Smote from the same library which is imblearn. And again resample our training data because we are not going to resample our testing data. Now let's see the number of counts label wise counts in our training data set. These are the number of counts that we have in our training data. Now again we are going to normalize our training and testing data. We are going to normalize only explanatory variables, not the target variable because it is a wrong. So we use Min Max scaler for normalizing our data. So let's run this. After that we are going to train our model on X train and Y train. We have used batch size 32 and number of epoch 5. So let's run this. This model training process will take some time. So we'll meet after fifth epoch. Here our model training process is done. Now let's perform prediction on the testing data with the help of this nn.predict. So let's run this. Now we are going to evaluate our model. Here we can see we have better accuracy. The precision, recall and F1 score are quite good for both the classes and number of misclassified examples for the minority class are also in lower in the number. So here let's firstly compare the results of all the techniques that we perform to the actual imbalance data. So let's move into back. So this is the result that we obtain after training a model on the imbalance data. Here you can see the precision, recall and F1 score and you can compare it with the models that we train on the balance data. So this is the result after training a model on the imbalance data. It is quite worst result. And after performing under-sampling, the results are quite good than that of model strain on the imbalance data. Again, after oversampling, the result are similar to that of results regarding the random under-sampling. And again, for Smote, we have same number of score precision, recall and F1 score. So we can use any out of this method to perform balancing on this data because these four methods are fairly suitable to our data set and we can use these methods to balance our data. This is how we can use Smote to handle data imbalance. Thank you for watching. If you enjoyed the video, please subscribe my YouTube channel. Your support means a lot for me. See you in the next exciting video. Bye.