statistical significance calculators & split-test calculators (3)
statistical significance calculators & split-test calculators
hello i got a couple of questions after reading posts about statistical significance.
here's a list of the calculators i've noted down:
abtester.com
abtestguide.com/calc/
splittester.com
adcalc.net
easycalculation.com/statistics/statistical-significance.php
visualwebsiteoptimizer.com/ab-split-significance-calculator
getdatadriven.com/ab-significance-test
cardinalpath.com/resources/tools/ab-testing-tools
hubspot.com/ab-test-calculator
statpages.org/confint.html
peakconversion.com/2012/02/ab-split-test-graphical-calculator
and then analytics-toolkit.com/statistical-calculators after reading http://blog.analytics-toolkit.com/20...a-statistician
1) difference between these two types of calculators (statistical significance calculators VS split-test calculators)?
2) there're too many options out there and i thought peakconversion.com/2012/02/ab-split-test-graphical-calculator should be the one-stop shop as it's more "accurate". but i'm now uncertain after reading the article from that analytics-toolkit.com site. should i use their calculator instead?
3) veterans may have already started yelling at me "you are over thinking!!" yes i am to some degree 
right now i just feel overwhelmed after going through a lot of posts and online resources about "statistical significance" and "split testing". i just wanna know which tool should i use at this stage, that is, relatively new and don't have a deep pocket
any help would be appreciated!
Pick one.
Use it.
There is no 'best' in this case. You'd need to be converting in the area of x,xxx conversions / day for the differences between these tools to matter to you.
Accuracy isn't an appropriate term when dealing with these calculators.
You're comparing invariant numbers to each other - this is 100% accurate as there is no variation in the source data... similarly the results are going to be the same every single time for the same data set.
This probably sounds a bit confusing... but what I mean is you can divide 50 by 2 anyway you like and the answer is always the same, it's just maths, but the uncertainty in the the result can be analysed in various ways by using statistical methods that presume the source of these numbers has some variance to it (e.g. flipped a coin 50 times and by chance you dropped it twice).
The main difference with these calculators will be a) their user interface and what it lets you control e.g. confidence interval, and b) the statistical methods used e.g. Bayesian inference, frequentist inference, etc.
I imagine most use Bayesian.
That article from analytics toolkit is excellent as is their calculator, so just use that if you're comfortable with it. Don't get carried away with the sample size calculator. Note that the stats calculator here doesn't show the upper and lower conversion rate limits at X confidence, rather it shows the % changes vs the baseline. So if you are questioning "is the minimum likely conversion rate of this data above what I need to be breakeven" then the peak conversion calculator is more straightforward.
In the end, any statistical analysis is likely to be better than none.
P.S. these calculators again all just do a similar thing but have a different user interface/back end code.
Home > The Newbie Zone > Questions and Answers