Back to Evaluation, Leakage, and Unsupervised

Hyperparameter Tuning — Grid, Random, Bayesian

Three strategies: brute force, random, smart. When each is right and where Optuna fits. FIND_VIDEO: search 'hyperparameter tuning bayesian optimization' — recommended channel: StatQuest / sklearn docs. Aim for 10 min or under.

20 minutesVideo LessonPDF notes

Watch Hyperparameter Tuning — Grid, Random, Bayesian on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Objective Function Definition — Minimizing an objective function is the fundamental goal of hyperparameter tuning.
Grid Search Intuition — Grid search checks every parameter combination exhaustively to find the minimum objective value.
ML Search Space — Hyperparameters define a combinatorial search space, illustrated by 36 combinations for a decision tree.
Grid Search Implementation — Code demonstrates iterating through all 998 combinations to find the minimum value of y.
Random Search Concept — Random search samples a fixed, smaller number of combinations to quickly find a near-optimal result.
Bayesian Search Model — Bayesian search uses a cheap surrogate function to approximate the expensive, unknown true objective function.
Utility Function Role — The utility function guides the search by balancing exploration of new regions and exploitation of known good areas.
Bayesian Search Results — Bayesian search finds a near-optimal result with significantly fewer iterations than exhaustive Grid Search.

PDF notes

Bottom line: You will learn the core mechanics and trade-offs of three major hyperparameter tuning strategies: Grid Search, Random Search, and Bayesian Optimization. You will be able to select the appropriate tuning method based on the complexity of your model and the available computational resources.

This lesson on Hyperparameter Tuning — Grid, Random, Bayesian is hands-on and example-driven. You will learn the core mechanics and trade-offs of three major hyperparameter tuning strategies: Grid Search, Random Search, and Bayesian Optimization. You will be able to select the appropriate tuning method based on the complexity of your model and the available computational resources.

What You'll Be Able To Do

Define the objective function in hyperparameter tuning.
Compare the search space exploration strategy of Grid Search versus Random Search.
Implement a basic Grid Search iteration process (conceptually).
Explain the role of the surrogate function in Bayesian Optimization.
Differentiate between exploration and exploitation in optimization.
Select an appropriate hyperparameter tuning method for a given ML problem.

Topics Covered in Hyperparameter Tuning — Grid, Random, Bayesian

Objective Function Definition (0:00 - 1:00) — Minimizing an objective function is the fundamental goal of hyperparameter tuning.
Grid Search Intuition (1:00 - 2:40) — Grid search checks every parameter combination exhaustively to find the minimum objective value.
ML Search Space (2:40 - 4:40) — Hyperparameters define a combinatorial search space, illustrated by 36 combinations for a decision tree.
Grid Search Implementation (4:40 - 10:00) — Code demonstrates iterating through all 998 combinations to find the minimum value of y.
Random Search Concept (10:00 - 11:40) — Random search samples a fixed, smaller number of combinations to quickly find a near-optimal result.
Bayesian Search Model (11:40 - 14:30) — Bayesian search uses a cheap surrogate function to approximate the expensive, unknown true objective function.
Utility Function Role (14:30 - 15:30) — The utility function guides the search by balancing exploration of new regions and exploitation of known good areas.
Bayesian Search Results (15:30) — Bayesian search finds a near-optimal result with significantly fewer iterations than exhaustive Grid Search.

SQL Cheat Sheet

Grid Search — Exhaustively checks every possible combination of defined hyperparameters
- In practice: x in {-1, 0, 1, 2}, find x minimizing y=x^2.
Random Search — Samples a fixed number of random combinations from the search space
- In practice: Select 200 random indices from 998 possible combinations.
Hyperparameters — Configuration settings external to the model learned from data
- In practice: Decision tree depth in {2, 5}, minimum samples in {50, 100}.
Objective Function — The metric (loss or score) that the tuning process seeks to minimize
- In practice: Minimize negative AUC (equivalent to maximizing AUC).
Surrogate Function — An approximate function used to model the expensive objective function
- In practice: The red line approximating the unknown black box function.
Utility Function — Guides the next sampling point by balancing exploration and exploitation
- In practice: Function deciding whether to explore new regions or exploit current knowledge.

Comparison Table

Method	Strategy	Efficiency
Grid Search	Exhaustive search of all combinations	Low (high computation cost)
Random Search	Random sampling of the search space	Medium (faster than Grid)
Bayesian Optimization	Model the objective function using a surrogate	High (fewer evaluations needed)

Common Pitfalls

Mistake: Using Grid Search when the search space is very large or high-dimensional. Avoid: Switch to Random Search or Bayesian methods to save computation time.
Mistake: Assuming Random Search will always find the global optimum. Avoid: Recognize Random Search finds a near-optimal point quickly, not necessarily the best.
Mistake: Ignoring the utility function's role in Bayesian Optimization. Avoid: Ensure the utility function effectively balances exploring unknown regions and exploiting known good areas.
Mistake: Defining hyperparameter ranges too narrowly for Grid Search. Avoid: Use wide, sensible ranges to ensure the optimal value is included in the search space.

FAQs

Why is minimizing negative AUC the same as maximizing AUC? Optimization algorithms typically seek to minimize a function (loss). By minimizing the negative of a score (like AUC), you achieve the goal of maximizing the original score.
What is the total search space for the decision tree example? The total search space is the product of the number of options for each hyperparameter: 3 x 2 x 2 x 3 = 36 possible combinations.
What does "exploit the current knowledge" mean in Bayesian search? It means sampling points near the current best-observed result, assuming that nearby points are likely to also be good.
Why do we need a surrogate function in Bayesian Optimization? Evaluating the true objective function (training and testing the ML model) is computationally expensive. The surrogate function provides a cheap approximation to guide the search efficiently.

Common pitfalls

Mistake: Using Grid Search when the search space is very large or high-dimensional. Avoid: Switch to Random Search or Bayesian methods to save computation time.
Mistake: Assuming Random Search will always find the global optimum. Avoid: Recognize Random Search finds a near-optimal point quickly, not necessarily the best.
Mistake: Ignoring the utility function's role in Bayesian Optimization. Avoid: Ensure the utility function effectively balances exploring unknown regions and exploiting known good areas.
Mistake: Defining hyperparameter ranges too narrowly for Grid Search. Avoid: Use wide, sensible ranges to ensure the optimal value is included in the search space.

Hello hi everyone. So this is our tutorial for hyperparameter tuning. We will see the intuition and small examples behind hyperparameter tuning approaches which we take into machine learning. Right? So basically, let's say I have this function which is y = x² and I want to find out the value of x which minimizes this y, okay? So that's the objective that we want to minimize the value of y given x, right? So the first way which is a naive and the way which you can think as the first approach is basically, I can go through all the values of x, right? So let's say for x = -1, I calculate y this equals 1, okay? This is step 1, right? Let's say this is step 1. Next what I will do is I'll go to next value of x which is 0 in this case and get the value of y, right? Then I will go to the next. Right? So till now this is my minimum value of y. So this is the best candidate till now, right? The next iteration if y reduces than this 0, then we'll update the x. Okay? Let's write down one more thing. Let's say I'm writing best x till now. So after this iteration, this is -1. This is 0 because y reduced. But here y didn't reduce so it's still 0, right? So let's say I go to next iteration I put x = 2 which gives me y as 4. Right? So in this case still my best x is 0, right? So you can see once I iterate through this table, I'll get my minimum value of y which is 0 at x = 0, right? So this is the idea behind grid search that you go through all the parameters one by one or of the search space one by one and try to find out the value of y and figure out the values which minimizes this objective function in our case, okay? So the same thing the same intuition we can follow in any machine learning algorithm, okay? So what happens in let's say in ML I'm taking example of let's say decision trees which is easy to understand. Okay? So in decision tree the parameters which I'm interested in are let's say So in decision tree there are certain hyper parameters which we can tune. For example, how many nodes I want, right? The depth of the tree which I'm interested in, minimum sample weight, right? Minimum samples I want in each leaf, then sample weight or class weight we can put, right? So let's say my nodes is in the range of I want to try 5, 10, 20. Let's keep it this way. Depth I want 2, 5, minimum sample I want let's say 50 or 100 in this case. Then sample class weight I tried with 1, I want to try with 5. I want to try with balanced, let's say. Okay? So the total search space in this case becomes 3 x 2 x 2 x 3 which is 36 possible combination in this case, right? Uh like we had around 7 possible candidates in this example. Here we have 36, right? So when SKlearn or any other package which enables grid search it will go through all the possible combination within this 36, right? So it will take let's say 5 nodes, 2 depth, 50 minimum sample and sample class weight as 1 and build a model, right? And give me the loss which I'm interested in, okay? So let's say the loss which I'm interested in is negative of AUC, right? So if I want to increase the AUC, it's same as I'm decreasing the negative of AUC, right? So this could be one of my optimization parameter, right? So I want to maximize AUC, for that I want to select the best value of this parameter set, okay? So let's go to a simple example and see how does it work, okay? So the basic libraries which we require, NumPy and math. So this is the NumPy array x which I'm creating which takes values from 0.1 to 50, incremented by 0.05 and I'm just randomly shuffling the same. Similarly, I have created one more array, z which ranges from 2.1 to 52, incremented by 0.05 and uh this is how it looks like. So I have minimum value of x 0.1, max value of x 49, and so on, right? So using x and z, I'm creating a new variable y which is x² + z * sinusoidal value of x, okay? This is a random example I took. We can take anything, right? So my y gets minimized at 876th index, and the value at that index is 4.4 for x and 49.84 for z, okay? So I have x values 998, y 998. So basically, I have to try across this 998 combination which minimizes my value for y. So here I'm initializing an empty list which is my search space where I want to search. Basically, I want to store the value of x and z at each index so that I can search through that and find out what works best for me, okay? So I should see 998 values in my search space, okay? Now here what we are trying is so I'm initializing my best x value at 0, best z value at 0, and minimum value of y I'm initializing with a large quantity, okay? Now coming here, we are searching through each and every value within my search space. So as it is a list, so the first value is x and the second value is z. So I'm storing them here, then creating y value for both of them, right? Now if my y value which I got is less than this minimum y, I'm updating my best x, best z, and minimum y, okay? So in the first iteration this will always be true because we have taken this number too large, okay? But in the next iteration this will only get updated when we get something minimal than the current minimum value of y, okay? So I've got we have done the value. So as expected it find out that my best x is at 4.4, best z at 49.84 which kind of minimizes my y to this particular quantity, right? This is expected because it is searching through each and every possible combination, right? Now coming to random search, it's pretty simple. So instead of trying out all possible 998 combination out of this, can I try a smaller number, right? Let's say I want to try out of 200 samples. So idea is that I should be nearby the optimal point by trying random values, okay? I might not get the optimal result, but I will be close to this -28 which is my minimum value of y in this case, okay? So here I am selecting just 200 random indices from the x, okay? These are the indices. Now I'm doing the same thing, but across only the selected indices, right? So, you can see we couldn't find -23 which was the best value of y, but we came pretty close to that, right? So that is the idea behind random search. You might not get the exact value of y, but you'll get somewhere near to the optimal point, okay? Now, let's Okay? So, let's see the intuition behind Bayesian search, okay? So, let's say this is x and y axis I have, right? So, this is -1, 1. So, this is how my original function look like, let me write it down. So, this is nothing but my original function, right? Which I don't know. I'm not given this function. If I know it then I can directly differentiate and get the values. So, for me what I can do is, let's say there is one black box or this is some function, okay? I give it the value of x and it returns me y, right? This is what I have, I don't know the actual function, okay? So, what I can do is, essentially, let's say I randomly tried the value of -1, right? So, let's say I tried this value of -1. I gave -1 to this function and it returned me 1 as an output, okay? So, I will kind of have one point over here, right? Let's say I gave 2 in the second example and it will come somewhere here, right? So, for now, if you see my function which I've which I can estimate right now is this red line, right? So, the idea is that I don't know this actual function, but by experimenting a few values, can I get a approximate function which can give me similar result as this original function, okay? So, right now this is the function, right? Let's say I kind of tried this value 1 over here as well, okay? So, now this is 1. So, so my function will kind of so approximate function which I have right now will look something like this, right? So, right now I have two option. Either, so this area below this function and above this red point are unexplored, right? So, either I can go into that region or I can try something in between, right? So, there is So, there is one function called utility function which kind of helps me to figure out whether I should explore in the unexplored regions or I should be in the same region, okay? So, let's say now I gave this function to my utility function and I asked it where should I go next, okay? Now let's say it resulted this 0 in this case, right? And so, this is the observation I will get from this function over here, right? Now my approximate function becomes this, right? So, now you can see though I don't know the actual function, but I have kind of figured out this approximate function and now from this function I can get what are the minimum values of x which will minimize this y, okay? So, that is the idea behind Bayesian search. So, we create this surrogate function, right? And it Okay? So, we create this surrogate function and And there is this utility function which helps me to decide whether I should explore the new region or exploit the current knowledge, okay? Let's go and see an example, okay? So, I'm going to use hyperopt package over here. So, these are some of the requirements which we need. So, don't worry about them too much, okay? So, I'm creating a sample dictionary here over here to remind how can we extract values out of a dictionary, right? So, that's just it. Now, parameter search space or feature space which we can define, right? For my function, the only value which I'm interested to change is the index in my search space. So, there were 998 values in my search space. So, I'm just interested in single quantity at what index my function takes minimum value, right? So, this is my parameter space. Now, this function at the below line it returns me the value of y for a given index, okay? So, hp.choice will select a random value from search space range which is 0 to 997 in this case, and for that index what is the value of x and z and the corresponding value of y that's we are getting from this function, okay? So, again we are minimizing the y's, right? So, we are starting with some large number 9999 for this case. So, this small function this calls this minimize function in turn and checks whether the value returned by the function is less than the current best. If yes, then it assigns the new value of y to the best and updates the parameter, okay? So, this is where it happens. So, what this fmin function is doing? It is finding out the minimum value of parameter search space or the finding out the values from parameter search space which minimizes f, okay? And we are trying some 750 iterations here. So, let's see how does it work. So, currently my y is a too high of a number then it reduced to 20. So, it has completed the runs and we can see that the best value it found out at index number 81. Let me put it here and see what are the values of x, y and z over here, okay? So, it came pretty close to the best value of y. So, our actual minimum value was at 28, but it came pretty closer to that, right? So, they are the intuitions behind the three of the most popular hyperparameter tuning algorithms. Thank you.

Frequently asked questions

Why is minimizing negative AUC the same as maximizing AUC?

Optimization algorithms typically seek to minimize a function (loss). By minimizing the negative of a score (like AUC), you achieve the goal of maximizing the original score.

What is the total search space for the decision tree example?

The total search space is the product of the number of options for each hyperparameter: 3 x 2 x 2 x 3 = 36 possible combinations.

What does "exploit the current knowledge" mean in Bayesian search?

It means sampling points near the current best-observed result, assuming that nearby points are likely to also be good.

Why do we need a surrogate function in Bayesian Optimization?

Evaluating the true objective function (training and testing the ML model) is computationally expensive. The surrogate function provides a cheap approximation to guide the search efficiently.