Dear fellow coders, welcome to One Little Coder. If you have ever published a machine learning model in production, it's very important for you to monitor the machine learning model. Why? Because model drift. If you have never heard about model drift, this video is going to give you a very quick summary of what is model drift and what should you do about it. So, first of all, what is model drift? Model drift is nothing but, let's say you have got a machine learning model, a very simple one, let's take MX + C, a very old one, like a linear regression. Over a period of time, your predictions actually degrade a lot. So, the accuracy of your prediction, the number of false positives, a lot of these metrics go for a toss, and that is when you think that there is a model drift.
Now, what is model drift? Model drift can be of two big types, like if you want to categorize it, two large categories. The first one is called a concept drift and the second one is called a data drift. Okay, so the first one is called a concept drift. Let me write drift here. Yeah. And then the second one is called data drift. Now, what is concept drift and data drift? It's quite simple. So, whenever you have got a model like this, like we just saw Y = MX + C, when Y changes, when the statistical nature and properties of Y changes, Y, which is the actual variable that you are trying to predict. So, whenever the nature or statistical properties of the variable that you are predicting changes, that is called concept drift. So, whatever, like when this changes, it's called concept drift. Now, what is model, uh, data drift? Whenever the nature of the variable that you are, like the predictors, the explanatory variables change, then that is called data drift. So, if Y changes, that's called concept drift. If X changes, or, sorry, the nature of X or characteristics of X changes, then it's called data drift. Now, you might ask me, why would it even change? Like, what is the possibility of changing? Let me give you an example.
So, right now, I think most of you might aware of COVID situation. In a lot of countries, COVID is better. In a lot of countries, COVID is still worse. So, before COVID, let's say there is an air, um, air flying agency, and that agency, let's say they had a model to predict a certain outcome. Let's say in this case, the outcome is to predict the number of people that they would be flying. So, if you assume that the number of people would be flying is Y, and then X is like they are going to use a lot of variables, like holidays, macroeconomic factors, all those things. Okay. So, now, if you take a scenario like this thing, since last March, probably you might have noticed that this entire model would have probably gone for a toss. Because why? Now, there are borders closed. In every country, most probably you would have seen the borders are closed, so there was no chance that a flight could even operate.
So, but the model, if the model was still existing, the model would be still predicting Y amount of passengers flying. And this is a very classical example of model drifting. So, what has happened in this situation is you can see that both Y and X has changed in this situation. Why Y has changed? Because the nature of people's mentality to fly has changed. People cannot fly because there is a country restriction, a legal restriction. And what is changed in data, X, if you see, the underlying data, like, for example, macroeconomic factors, people lost their jobs, people do not have money, and attributes like this have made that they cannot change. So, in this case, both Y and X has changed, resulting in a model drift. That means that the original prediction of whatever was happening would be terribly wrong in this case because that may not be entirely valid at this point. So, this is a very classical example of what is happening with model drift.
Again, if you want a very simpler example, again, you can, you know, you can you can look at this. Let's say, like, somebody has built a face recognition model. Okay. And it's a very bad face. Okay. Let's say somebody had built a face recognition model. Now, this face is Y. Now, after a point, like, you like there's a global trend, let's assume there's a global trend where everybody starts wearing a mask. Okay. Now, none of the face recognition model that you have built would work fine because the training data, the entire set of training data that you used to build this model, never had, you know, mask. And now everybody in the world started wearing mask, which means your Y is not Y anymore. Because what you have trained your model to predict or detect is actually a face. But ultimately, what has happened is there's a new component added to everybody's face on the planet, so your model is not going to function as much as you expected in the past. So, which means your model has gone for a toss. Which means your model is drifting. So, it's like, you know, how in car races, people drift. So, it started drifting, which means your model is not going to function as much as you wanted it to function. For example, if you had told somebody like, my, let's say AUC is like 0.8, it is not going to be the same case anymore, because you cannot flaunt on your presentation that your model has still got a 0.8 AUC.
So, what do you do about it? Like, so now the first step is we know our predictions are very bad. Model has started drifting. And, like, now, what is next? What do you do about it? So, one of the things that people usually do is model retraining. So, you retrain the model. Retraining. What is model retraining? Model retraining is nothing but a process of training the model again, but with new set of data that you have got that has indeed resulted in your model drift. So, rather than training your face recognition model with the same existing data that you had in the past, you would start using the model with masks on it, like faces with masks. So, now your latest data of model training would include masks as well. So, this is a very simple option. There are a lot of other options, but the simplest option is you actually retrain the model with the new set of data that has, you know, the new set of statistical characteristics that you have got.
Now, the next question is, can we ever avoid model drifting? Like, that's a very good question, right? Can we actually build a robust model that can always avoid model drifting? While I do not have a very good answer for this thing, I would recommend you to read two books. One is Black Swan, and the second is Fooled by Randomness. Both these books are by Taleb and randomness. So, the reason I am asking you to read these books, these books explore the concept of building models that are not good in predicting things, and when even when people still think that the models do a good job in predicting. So, for us as a data scientist or machine learning practitioner, a machine learning engineer, it's very important for us to realize the fact that the world changes. The world would indeed change. The properties of the variables that we are trying to predict, the properties of the variables that we use to predict these variables would also change. So, it is important for us to realize and acknowledge that this change is imminent, like it's like you cannot stop it, and so you have to, you know, prepare yourself like what happens in such a situation.
And that is where a very important concept comes into picture, which is called model monitoring. What is model monitoring? Model monitoring is nothing but monitoring your production model, monitoring. So, why do you want to monitor your model? You need to monitor your model in order to find out whether there is a drift in the first place, are properties changing, your accuracy is dropping. So, all these things may not come naturally to you. Like, nobody's going to come and tell you like, "Boss, I went to your website, it's not recognizing my face." Like, it it happens. Like, for example, if you have seen in the news, once the pandemic kicked in, everybody started wearing masks. A lot of iOS devices work only on, let's say, face recognition, face ID. And then faces start, like, they couldn't recognize the faces. So, Apple had to, you know, send a patch. All these things happen when you have a consumer-facing product where consumers are quite outspoken. But if not, then the best way is to do model monitoring. Model monitoring, you can do a lot of different ways, but there are some tools that help you, you know, monitor the model and understand how much is your model drifting and all these things.
If you see this, if you see this life cycle, ML life cycle and challenges by Databricks, you can actually see that there is a component at the end, which is primarily to monitor the model, and then one of the things that they monitor is data drift and model drift. So, it is very important, like if you have got models in production, if you are a company where you have got models. It could be what, like, it could be, you know, any type of production. Like, for example, somebody might serve a model as an API. Somebody might have a cron job that, you know, regularly runs and, you know, stores the prediction in a SQL database. Whatever model you have got, if that model is going to be somewhere where nobody is seeing the result every day, you need to have a robust model monitoring system, and then you need to monitor the drift, data drift and model drift because it's very important for you to make sure that your model does what it is supposed to do. Because imagine like now you have got a credit card, predict default rate predicting model. So, imagine like you have predicted somebody would default while that person may not be defaulting, and it is primarily because of model drifting, maybe like false positive, and now your collection team is going to that person and asking for, you know, like kind of details. So, this would result in a very bad customer experience and also it would result in probably like a lawsuit or something. So, it is very important for you to model your, monitor your model not just for model drifting, for a lot of purposes, but if you see this life cycle, you can see that model drifting and data drifting is to be monitored and that is a very important like mostly like the final situation of your ML life cycle, and it is also, you know, something that everybody should start doing it.
This, the objective of this video was to help you understand what is model drift as a concept, what is concept drift, what is data drift, and how do you monitor it and what is about it. I hope this video was helpful to you. If you have liked this video, please give a thumbs up and subscribe to the channel and also share it with your friends because this is not just about the person who builds the model, but it's about, you know, whoever is consuming the model. If you have any other question, please let me know in the comment section. Otherwise, stay safe, take care.
