How to calculate Statistical Significance? (12)
Hey there,
I've recently read caurmen's guide (http://stmforum.com/forum/showthread...he-Winning-Ads) about statistical significance and killing an ad which was very eye opening for a beginner like me. On the other hand I couldn't figure that out how did he come up with the following: 1200 impressions -> if no clicks kill the ad.
I have been using this site with no luck yet: http://statpages.org/confint.html.
Example question:
I want to get a 0.150% CTR on my ads with a high confidence (95%? Or whatever you guys are suggesting.) that they keep the same CTR. How much impressions would I need? How to calcluate this?
Basically I would like learn a formula, that can be used with all my future campaigns to decide when to kill an ad and how sure I can be that it's the right time to kill it.
A great explanation would be appriciated!
Cheers,
Zoltan
Originally Posted by Zoltan
Payout $0.3
To make it easier let's say you pay $1 for those 1200 impressions.
Assuming you can get 1%CTR you will then see 12 clicks out of 1200 impressions.
Out of 12 clicks you have 3 conversions. Which translate to 25%CR
And your revenue will be 3 x $0.3 = $0.9
So your profit is -$0.1
How likely it is you get 25%CR?
Even if let's say you can get 1 conversion, which is around 8%CR you will still be -$0.7
That's why with low payout, using the example above is $0.3, if after you spent $1 you did not see ANY clicks at all, it's safe to assume you should cut this placement early, to avoid further bleeding on your budget.
It's just a bit of common sense when you're still trying to figure out how to increase CTR CR etc. While on a limited budget to test.
Even if you have big budget to test it's just dumb to keep running until you spent $10 with no clicks at all.
Moreover there could be 100 placements like that. So multiply those by 100 now. $100 no clicks? Still hoping even after $1000 no clicks?
Hopefully that makes sense.
Along the way, you keep refining your rules to suit your style.
Example: Risk taker? You might be able to stomach until 10x payout before you cut. Because you're experienced enough to keep tweaking the CTR and your gut tells you this might work.
To each his/her own. Based on his/her data.
Originally Posted by Zoltan
Originally Posted by caurmen
Grab the spreadsheet in: http://stmforum.com/forum/showthread...ats-Calculator and plug your numbers.
There's a gem in that thread, some kind user posted another excel for easy calculation, read the whole thread, post-by-post, if still confused, post your questions in that thread.
Hi qureyoon ,
Thank you for your answers. Yeah, I have read all of them thorugh. However what I need is to calculate the statistical significance. In Part 10 he explains a formula to calculate the Minimum Viable Conversion Rate. Statistical significance however is not covered deeply. That's what I'm looking for.
Anyone else can help me to explain this?
Originally Posted by Zoltan
This is great for someone who is already generating leads. I would need a solution for the Ads impressions exclusively as per my example. I need to understand how to calculate if my Ad has reached a statistically significant limit (based on my desired CTR) so I can cut out the non-performers.
Originally Posted by Zoltan
You should not cut ads based on CTR only.
Even a lowly ads that gets only 0.03%CTR if it converts. Then it converts.
You need all the figures all together. Do not isolate each other.
Even if you managed to get the ads to your so called desired CTR, let's say 10%, if it doesn't convert it's pointless.
In the end the simplest formula you should look is revenue > cost.
Originally Posted by qureyoon
Look at:
- Why You Should Care About "Statistical Significance"
- How Many Banner Impressions You Need
- Your First Banner Cut
That is what I need. If the revenue on a low CTR banner is high I obviously won't kill it, but the basic question here is when can i be statistically certain to cut an ad. Please look at my question in the OP. I don't need any calculation based on the cost/revenue. I just need a formula which can decide if the number of impressions and clicks I'm getting is statistically significant. That is all!
It's in Part 10 mate.
http://stmforum.com/forum/showthread...l=1#post133177
Originally Posted by caurmen
But then again, good luck on your journey for the knowledge! Keep us updated on what you've learned
One thing that I should clarify here: the 1,200 impressions rule is a rule of thumb based on a number of generalised assumptions. It's offered in Part 7 as a way for people who aren't as math-happy to get going in a user-friendly way.
If you want to be more precise and develop your own hurdles, you can do that using a binomial confidence interval calculator as discussed in Part 10.
However, do be aware that it's a good idea to cut ads that are unlikely to be profitable just based on the cost of their clicks as early as you possibly can. That's the reason for CTR-based hurdles: if our payout is $5, our expected conversion rate after optimisation is sub-20%, and looking at our statistical significance calculation we're going to end up with a CPC of $1.20 at the lowest possible, there's no reason to run the ad until we have statistically significant ROI. And we can save a lot of money by cutting that no-hope ad early.
Thank you all for your answers. Then I guess I'll just start playing with the binomial confidence interval calculator to come out with my own hurdles. I will also try to keep an eye on the CPC as well, thanks for tip.
Cheers,
Zoltan
caurmen: thanks a lot for that info. I myself test banners buying them at CPC model. This is quite cheap, and, in my opinion safe in order not to screw up, and burn a big budget.
Home > The Newbie Zone > Questions and Answers