This is an AB testing question. Let's say that you design an experiment to measure the impact of financial rewards have on user response rates. So, let's say it's a survey and the result shows that the treatment group with $10 in rewards has a 30% response rate, while the control group without rewards has a 50% response rate. This is, you know, obviously odd. And so, could you explain what might have happened and how you could improve this experimental design? Yeah, for sure. So, this definitely seems odd because, I mean, general intuition says that if you give someone a financial incentive, so then they tend to reciprocate or respond more. Ah, if the control group itself is having 50%, then the treatment group should at least have 50%. So, some things I would definitely think is, I mean, one would be like I would probably trust the experimentation process and then see that okay, maybe offering financial incentive is discouraging them because they might feel that we are kind of like buying their response and they don't want to really do it. Right. So that could be a hypothesis. That seems very unlikely but that could be the reason everything went well and that is what's happening and the company is better off just by not giving the incentives. Another thing could be something in terms of sample ratio mismatch, you know like for whatever reason the randomization is not really happening. So, let's say if we plan to do like 50/50, a treatment and control group, so that randomization or that division is not really happening. So, we can go and check on that. For example, let's say that the link to the survey is systematically getting broken for people in the reward group. Maybe we are attaching the reward link where they can go and claim and because of that, the load time is increasing and they're getting frustrated. So, basically the experience of filling this survey might be systematically bad for people in the reward group. So, that is a plausible reason to happen. And this generally happens more often than not in companies. So, when they're trying to render like a new new feature and it is taking systematically longer than expected and people get frustrated and they don't continue with the process flow. Something like that could be happening. So, I would definitely go and test if everything is same in terms of time taken to complete among the people who converted. What is the time taken to complete that survey form for control and treatment and see if they are significantly different. So, that is something I would definitely test. So, let's say if even that checks out. There is no problem in terms of like how the survey is loading, in terms of you know, even the survey completion times, it's pretty much the same. Not significant difference. Then in order to, again, I would probably go back to my initial hypothesis saying that for whatever reason people are feeling that giving them reward is kind of like buying their loyalty and not really liking it and they're less likely to complete the survey. So, in order to again reinforce this hypothesis, I would test something, like I create like one more group with something in between like a $5 reward and $6 reward and see like how the conversion rate is. So, based on this hypothesis, let's say if I give like a $5 reward, the conversion rate should be around like, you know, between 30 and 50%. If that is again happening, then, you know, that reinforces my hypothesis and then I conclude that it is probably better not to give reward. So, I will create like an interim group, you know, something in between or something more than $10 and see if it is around still 30% or even reduced. So, I will do something like that. Like at least from my research work, I've seen multiple papers that use financial rewards to improve survey rates. So, I don't believe that that is the case and I would probably go back and check my experiment, how my experiment is getting conducted again and see if I can fix anything there. Gotcha. What could be an alternative I guess explanation that Is there any worth towards increasing the size of the reward in this case? Like maybe people think $10 is too small, you know, would it be then worth testing? Or is it the fact that maybe the control group without any kind of reward gives them, you know, some sort of other benefit towards like the $10. So, if this experiment was really testing the impact of reward on survey completion rates, then even the call to action or the message should be exactly identical except the fact or, you know, like the email saying that hey, if you complete this survey then you will get this, you can claim this $10 reward. If for whatever reason control group had some other incentive, so maybe what I'm testing is not the effectiveness of $10 reward compared to the control. I'm testing the effectiveness of $10 reward compared to the incentive in control. I will definitely go back and look at the exact call to action that I'm sending my customers or target population and then see whether there is this, I mean, what I'm testing is what I am trying to test. Gotcha. Yeah. So, I mean, for an example, right, what if you advertise it in the subject line like $10 reward for the treatment group, but then for the control group, you can't really say $10 reward at all, then you have to completely change the subject line in that case, right? So, is that testing multiple variants then, or is that testing multiple effects? Or is that still like a valid test? So, for the same reward, you can always give the call to action in multiple ways. For example, so there is some research that says that so for the same $10 reward, if you give store credit versus like a free cash or, you know, like Amazon gift card. So, the effects are slightly different. Reward might be the same, but the way you operationalize it will have different effects. In the same way, like having an email with a subject line that says $10 reward seems kind of like a like spam because you get all these spammy emails that look like too good to be true and people are now used to them and used to just archiving them or deleting them or reporting as spam. So, maybe you're not operationalizing it in a good manner. So, you could test like a different call to action where subject line is similar to the control group, but it's just the reward that when they open the email that is prompting them to complete it. So, yeah. So, if for the same $10 reward, having it in subject versus just having it in body, definitely will have different effects. Again, so the reason why we would be testing them is to see what's working and use it for the entire population of users. So, you test it on a sample and then use it for the entire population. There is a downstream process that will use the results of this experiment. So, you would want to design in such a way that can be used in the downstream for the population of users. So, I would test both the cases and then see what's exactly happening. Yeah, I agree with that too. Specifically, I was thinking that if one email is more catered towards, let's say, you know, helping out because you should help out on our survey versus another one is more catered towards, here's $10 and it's a pretty easy way to make $10, then you're kind of biasing for people that have a financial incentive versus people that just have a regular kind of helping out incentive towards your research study. So, yeah. That makes sense. So, let's say that this is a corrupt result. You saw changes in a variety of places, right, whether it was the subject line or something else. And let's say you want to run like a great experiment next time. You still have some financial budget. You mentioned that you would just test $5 next to see what would happen. Is that the perfect experiment that you could run next time? Like what is the perfect experiment that you would run instead of this like $10 kind of financial reward system? Could you just like describe it from start to end? Yeah, sure. So, the goal that you are trying to optimize here is conversion rates. I mean, we want as many people to fill the survey as possible. So, it's not just the completion rate, you would want them to put effort into filling the survey. So, we would look at the completeness of the survey or, you know, the length of, if there are text questions, we'll look at the length of the text, all that. So, even those should also be considered as the winning, I mean, the important metrics, not just the conversion rate. So, I will first decide on what all is important to me. So, are those like the completeness? Is that important as well? So, if it is, what is the weight to it? So, I will create like a hybrid metric which is representative of what success looks like for me. So, that will be my step one is to have like a good overall metric. So, once that is done, I've decided that I want to maximize this new metric that I created. I want that to be higher for my winning group and which I would subsequently implement for everyone else. So, the way like I would decide on sample size for this experiment or then the amount of time that this should run, so that will be based on practical significance because like given infinite sample size, you can detect any smallest of the effects. So, it doesn't make sense to just put tens of thousands of people because just to detect an effect. I will see what the practical significance for me is. So, if you take this example of conversion rate, so does 5% increase in conversion rate make sense for me to implement this reward? If yes, anything less probably doesn't make sense. 5% or more is the only one that I would be interested in. So, then I will get my sample size based on that effect size. 5% improvement given distributions of that metric, I will get the sample size from. So, you have like, let's say, 80% power and 0.05 practical significance and the effect size, any sample size calculator will give me like what the sample size for each group should be. And so, I will see if that makes sense for me. So, if I have more users, that means I can test more groups. So, that's how I will decide my sample size. So, I will always decide my sample size based on practical significance, not saying that, okay, like these are the people that that we used for something else, I will just test it. So, I will always take that as the starting point. The next step after I decide on sample size would be to see again, you know, to test different variants like this case. So, okay, we've already seen this. So, there is a drop because of the reward. So, then now I can test, I have my control group with no incentive, then have my other treatment group with $10 reward. If I know that everything else checks out, which is, you know, there is no sample ratio mismatch, everything was working well and I just want to test this hypothesis again, you know, like subject line versus some other reward. So, I will test some reward, something in between, which is like $5 reward, if that is enough. So, this is kind of like a pricing question again, you know, like what is the optimal price or incentive for you to like finally set. So, I will test something like a $5 reward and I will also test without subject line having that as just a push inside the email to send. So, I will not really test the same exact $10 reward in the subject line for 30% conversion rate because I already know that. So, if if my $5 reward falls somewhere in between, then I know that people are getting discouraged by being shown this financial incentive in the subject line itself. So, if I don't see that, then yeah, so then I have like your problems. But otherwise, like I don't have to test this group again, because now I am assuming that everything worked as is. So, I will only test the other one and definitely I will test the other one without having this $10 reward in the subject line. Gotcha. Cool. Yeah. Last question. This one is kind of tangentially related, but let's say that we ran this, you know, new experiment again with like the money, $10 reward. We made everything correct. Now we see that the financial incentive has increased the response rate, let's say to 60% versus the regular one is 50%. But we have a feeling that after looking through a few of them that the responses are, you know, like too fast, like the sentences are like, they're like very short, you know, they're actually not doing a lot of feedback. What would you think is happening there? You know, kind of obviously, maybe they're just based on the financial success. But like what would you do going forward after like seeing this? Yeah. So, so I think I kind of touched upon this in the previous one where I said I'll create like a hybrid metric which takes care of not just the conversion rate, but also the number of complete responses and all the other things, which is like the length of the text, all this which matters to me. So, that's why if you see like a lot of companies so they don't just have like conversion rate as the single metric, they have like multiple metrics as a combination of these metrics with weights as their metric that they're optimizing for. I mean, even like tech companies, you know, like conversion rate itself is not the main thing, like, you know, the total revenue or daily active users, amount of time they're spending, amount of people they're amount of activity they are doing. So, so there is like a holistic metric that will be used for optimization. So, that will be the outcome metric of the experiment. Yeah, that makes sense. Yeah, and being able to calculate that is a better towards an overall improvement there, so. Yeah. Cool. Awesome.
