I've been messing around with this calculator: http://www.raosoft.com/samplesize.html and realised how off I was with running tests.
I guess I just want to clarify some things - best to use an example (assume running one offer)
LP1:
Clicks: 126
Conversions: 17
CR: 13.49%
LP2:
Clicks: 126
Conversions: 7
CR: 5.56%
Now ordinarily to me there's a pretty clear winner (I'm using http://www.splittestaccelerator.com/freetool.php to calculate which gives 98%). Now the sample size calculator I first posted says that, with a confidence level of 95%, with 126 clicks the margin of error is 8.7%.
Seeing that margin of error makes me really confused. Ordinarily I would cut LP2 but... is the sample size enough to determine a winner? I mean... an increase of 6% on LP2 and a decrease of 3% on LP1 changes the whole thing, which is well within the margin of error given.
The split test calculator will also determine a 20 click lander a winner over a 2000 click one - which is clearly a sample size issue - so where is the cutoff? how do I work this?
How do you guys handle this? I don't really understand how the split test statistically significance calculator gets its results - so I'm not sure how to piece them together.
Sure I could just wait for 1000 clicks for each change I make... but that's a lot of money and time spent.
I don't want to sound like a broken record, but please try use the search bar on the top right. There are tons of threads written to explain statistical significance.
Here are a couple of goodies:
http://stmforum.com/forum/showthread...ht=statistical
http://stmforum.com/forum/showthread...ht=statistical
VERY IMPORTANT: Split-test calculators and single-test calculators use completely different maths to determine statistical significance. Don't get them confused. You're comparing results from a normal distribution calculator and what looks like a chi-squared calculator there - they will produce completely different results.
For split-testing LPs, I usually use http://visualwebsiteoptimizer.com/ab...ce-calculator/.
For single tests (like determining the distribution of CTR of an ad) I use http://statpages.org/confint.html