is 10,000 impressions a hard number? (4)
hey caurmen,
from your thread http://stmforum.com/forum/showthread...ats-Calculator
Now, simply check whether each ad's calcuated maximum possible conversion rate is higher than your Minimum Viable Conversion Rate. If it's not, kill it. If it is, keep it running.
You can do this at any point whilst running ads, but you shouldn't do this too often. Every time you test, there's a chance that your calculation is wrong. The chance is very small, but it multiplies by the number of times you test.
So just check your ads every 10,000 impressions or so!
1) after reading the above, I've got the impression that i HAVE TO get at least 10,000 impressions before making a decision. But is 10,000 impressions a hard number? Is it possible that it's already statistically significant after for example 5,000 or 8,000 impressions?
also, are you strictly referring to POF only here?2) "simply check whether each ad's calculated maximum possible conversion rate is higher than your Minimum Viable Conversion Rate" why use the maximum possible conversion rate instead of minimum possible conversion rate? To me, the maximum possible conversion rate sounds like a "best case scenario" and the minimum possible conversion rate a "worst case scenario". Shouldn't we be conservative and use the worst case scenario instead?
thanks again for any help, on a Christmas day!
Caurmen's on holiday at the moment so I'll pitch in:
1) Absolutely not. 10,000 is completely arbitrary. Statistical significance has nothing to do with the absolute magnitude of a number. You need context.
If I have ads that have 23% CTR and run at a $100 CPM, if I wait for 10,000 impressions I am going to have spent $1000 and have 2,300 clicks. That's a lot of data and you could readily make inter-comparisons between similar ads much earlier than 10,000 impressions. You'd damn well better, because spending $1000 on an ad before looking at the data is stretching the limits of laziness.
What you should do is set a ballpark based on how much those impressions cost (consider offer payout!) and how long it takes to get them. Run ads, look at their data and make comparisons. At some point you will figure out that at some arbitrary number of impressions you can usually find statistically significant differences in ad performance that lead to optimisation decisions. As a rule of thumb, think about how many impressions you can get for the offer payout and treat that as a target.
2) No... that's not how the math works here. You are comparing ad performance data to a set threshold that you need to break even. What you want to know is, within the realms of statistical doubt, will that ad likely ever reach break even. If you look at the worst case scenario you will actually remove ads that are currently profiting simply because of the uncertainty/distribution in the conversion percentages.
hey zeno, thanks a lot for the help!
Not sure why but i feel Christmas and boxing day have got me and made me stupid....i still have questions after reading your answer several times
1) shouldn't we simply use peakconversion.com/2012/02/ab-split-test-graphical-calculator to decide if it's statistically significant or not? what did you mean eaxctly by "arbitrary number"?
1) Yes, you can use that tool too, they both use binomial distributions. The process here is important, not the particular tool used or the specific confidence interval you use.
By arbitrary number I mean just that... arbitrary. How many impressions you need to make decent judgements depends on the traffic source and only you have to make a guided decision here. For PoF 10,000 may work well, for SiteScout you might want to work at 1,000, it depends.
To give you an analogy, say I want to keep track of how full my water tank is. If I said "It's best if I check it every 72 hours", then to you that's completely arbitrary and wholly depends on the context, no? What size is the tank? What feeds it? Rain? How much does it rain? Who's using the water? How long does it take to empty it if 6 people are showering? What I'm trying to say is, you need to look at the context of your advertising to decide what a sensible number of "threshold" impressions per ad is.
2) No, when I mentioned threshold I didn't mean any rate of conversion. I just meant how many impressions you let ads get before you start making comparisons and kill decisions. Again, this is up to you but you need to exercise thought here. If you get a million impressions per hour and spend $100/hour with an offer payout of $0.20, you can probably start making decisions pretty damn early. If you get 5,000 impressions per day and garner 12 clicks on average per ad, and on average 1 in 15 clicks convert, then you'll want to wait longer/spend more to get enough data to make reasonable comparisons.
There's no magic number for anything here and how quickly you can make comparisons actually depends on ad performance - e.g. if one ad is converting at 43% from click or offer, and all other ads are converting at 3-7%, with 100s of conversions worth of data, you'd be daft to run it until some particular number of impressions before deciding the inevitable - killing everything but the 43% ad.
The best advice I can give is to actually use these calculators on real campaigns and get a feel for it. You'll need at least several conversions on some of your tests for it to work at all and I would advise not killing an advert if a single conversion would drastically alter it's results from the calculator. Depending on conversion rate, you may find that you can make decisions with 95% confidence at 5,000 impressions. One other thing - time of day can have a profound effect on ad performance so I would also recommend pacing things so you don't burst traffic in 1 hour and then make all your decisions then. Not sensible.
3) Lets say you do some calculations and figure out that the minimum viable conversion rate for your traffic is 22%.
Then you have two ads with these data:
They are currently converting at 28% and 24.6% and have a positive ROI.
But the worst case scenario, with 95% confidence, is that the conversion rates would be 19.1 and 16%, respectively. What to do now? Kill these ads because the statistics say that in the worst possible case they might be unprofitable? Holy hell! You'd want to quit affiliate marketing if you were that pessimistic! Hence we do not use the worst case scenario... Especially since there are external factors like traffic optimisation, lander improvement, payout bumps, direct relationships and so on that we have at our disposal.
Home > The Newbie Zone > Questions and Answers