How many impressions before a banner's CTR is considered statistically significant? (6)
How many impressions before a banner's CTR is considered statistically significant?
I've seen this mentioned a couple times, but I don't quite understand the math behind it. I'd assume there is a threshold at which CTR is pretty well defined for most banners based on a certain number of impressions having been served. Where exactly is that line drawn? 10k? 50k? 100k? 200? More? I've seen CTRs bounce all over right up to the point where they've had 200k impressions, which is generally the point where I decide to cull a creative, not counting obvious flops. If the level of statistical significance is generally considered much lower than that though, I see no purpose in paying for on a creative longer than need be, which is what I'm worried I'm doing in a lot of cases now.
There's a lot of these but I'd get in the habit of plugging in numbers and letting the calculators do the math for you to call a winner.
http://www.splittestcalculator.com/
Alright, someone is going to have to correct me on which point of this I have interpreted wrong, because as I'm looking at this it makes no sense whatsoever. Everything looks straight forward, so clearly there's confusion somewhere along the line here.
Statistical significance is the probability that a result has not been concluded upon simply by chance. So say I run 1000 impressions one one creative and get 0 clicks. The next 5 impressions may all enter my conversion funnel and the CTR for that creative would sway wildly. There's no significance to that "test". In order to get a true reading on what my actual average CTR is I need a valid sample size to achieve what would be considered a "statistically significant" output. Pretty sure I am clear on that much.
This is where I am getting lost though. I want to see how many impressions it will take for me to determine that my CTR is somewhat accurate. How many impressions does a banner need to see, before you can be confidant that it's CTR is considered statistically accurate? I mean, how many variables are involved here? It seems like a question that would have a pretty straight forward threshold for an answer. I don't see how one banner would require a different sample size than the next to establish confidence in your test results. It's not like I'm dealing with advanced mutlivariate testing across my entire conversion funnel here. I just want to know at what point the numbers tell me I can safely decide to chuck a creative with confidence without overspending.
So I'll take your calculator link as an example of what I'm not understanding here. Step 1, "# of visitors" as I follow that would be my number of impressions. Step 2, "# of goals completed" I will fill in as the number of clicks received from those impressions. I plop some numbers into "Panel A" from one of my campaigns. My CTR is .59% on 1854 impressions, which I assume to be a pretty high number that will level off with some time. I already know this sample size is not "significant, so I click "Calculate Results" annnnnnnnd, nothing.
I have to compare it to something now. So let's compare it to one of the creatives I've had running a bit longer that has a bit larger sample size.
I am now told I have "93.12% confidence", but in what? "Sorry, no winner yet". Again, what? I don't even know what that calculator just told me, but I'm pretty sure it has nothing to do with whether or not I have a valid sample size to determine the probability that my CTRs are accurate representations. So now I'm just confused. I don't see the determination of statistical significance, and I don't understand what the confidence level depicted by that calculator is even referring to as having confidence in...
I have no issues calculating my CPC, EPC, CR, CTR, etc etc. I understand these values. Statistical Significance is an all new concept to me right now though, and I'm completely drawing blanks on this.
I'm on the phone right now so it's tricky to write a lengthy piece on this. I'll write in more detail later on. But in short - for banner CTR, rather than a split tester, you're looking to calculate confidence intervals. Specifically, you're looking for a binomial distribution, which allows you to calculate confidence intervals for any event, like a click through, given a number of trials and number of successes.
Here is a good one - http://statpages.org/confint.html
The rules of thumb are based on these calculations and some assumptions from practice running on usual traffic sources.
To learn how to use the calculator, check part 10 of the getting started guide. It's written for CVR calculations, but you can use the same approach for CTR.
Thanks. I was barking up the wrong tree and that's what was throwing me off. A more detailed description is not really necessary for me at this point I don't think. Looking at the numbers again now, things make sense when I see them. A low sample size will have a much wider spread on the confidence intervals. As the sample size grows you tighten the spread as your confidence in the statistics grows with it. The calculator appears to be based on 95% confidence, so keeping an eye on the intervals, you can see when they're ultimately proven doomed to fail or not, lol. \feel free to correct me if I'm wrong. Thanks again.
You're spot on - that's exactly right.
One tip: you can change the confidence interval for that calculator at the bottom. I sometimes set confints a bit lower for banners, particularly for CTR calculations. That's a risk management issue rather than pure stats, though.
Home > Questions and Answers > General Questions