How long do you wait before picking a winner when split-testing? (6)
How long do you wait before picking a winner when split-testing?
I typically wait for 30 conversions before accepting a winner from an A/B test. But I just read this interesting article on ConversionXL that makes me think I'm being lazy with my testing, 12 A/B Split Testing Mistakes I See Businesses Make All The Time - http://conversionxl.com/12-ab-split-...-all-the-time/
For more accurate split-testing, he says:
- You should not call tests before you’ve reached 95% or higher.
- A good ballpark is to aim for at least 100 conversions per variation before looking at statistical confidence. If you have a lot of traffic, go for at least 250 conversions per variations. It’ll be more accurate if it’s 350-400 conversions per variation.
- If we didn’t test for full weeks, the results would be inaccurate. So this is what you must always do: run tests for 7 days at a time. If confidence is not achieved within the first 7 days, run it another 7 days.
How much of this do you guys think is overkill? How many conversions do you typically wait for a winner, and do you also always run tests for a full week?
Picking an arbitrary number of conversions to "wait for" isn't a good idea in my opinion, no matter how high/low the number.
Just use a statistical significance calculator like this one, decide how high you want the confidence to be and act accordingly.
As for running tests for 7 days at a time, I don't see how that makes too much of a difference as long as you run both variations over the same timeframe and not e.g. variation A on Monday and variation B on Tuesday.
Good link there (thanks), caurmen says the same thing in that post though:
Since your ads aren't being shown to perfectly spherical, frictionless visitors of uniform mass, it's best to get to at least 50 clicks and a few conversions before taking the data seriously, ideally over a couple of days.
I'm sure it varies based on the offer/payout too... just wondering if anyone else has some good rules of thumb for testing LP variations for conversions
Why would it be any different for conversions? It's even more useful and actually in my opinion the only metric you should base your split tests on since CTR won't make you any money 
My point was not that running for 100 conversions is too much or too little, but that it entirely depends on the specific campaigns and that general statements like "run tests for xxx number of conversion to be statistically significant" are just bad.
If the calculator tells you that you have 95 % confidence of a variation outperforming the other, it doesn't matter if it's after 100 conversions or 20.
edit: just saw that you edited your second post, so this respnse doesn't make nearly as much sense anymore
btw: the payout doesn't matter at all as long as you don't split test offers with different payouts
Here's a hopefully interesting example as to why looking at CTR (click through rate) vs CR (conversion rate) can be extremely misleading. Results on this test are statistically significant. This was a small test on the order form of an advertiser. Had a huge impact on CTR to the checkout page, but surprisingly didn't lead to higher conversions at all.
CTR:
CV:
Originally Posted by andyvon
My point was not that running for 100 conversions is too much or too little, but that it entirely depends on the specific campaigns and that general statements like "run tests for xxx number of conversion to be statistically significant" are just bad.
@milobanski Yup, exactly what I'm asking this. I'm focused on testing a few variations for high conversions and want to make sure I don't call tests too early.
Home > Technical & Creative Skills > Tracking Campaigns