Home > General > Affiliate Marketing Forum

Finding The "Right Sample Size" for Statistic Significance. (5)

07-12-2016 02:53 PM #1 crysper (Member)

Use this calculator
http://www.evanmiller.org/ab-testing/sample-size.html

The sample size depends on the difference you want to detect and the conversion rate. The lower conversion rate the more sample size you need. The lower the difference between variations, the more sample size you need.

07-12-2016 03:26 PM #2 caurmen (Administrator)

I generally avoid using frequentist split-testing approaches precisely because of the need to determine a sample size in advance (and the fact that said sample size tends to be very large, as the link Crysper shares will show).

If you use a Bayesian approach instead you don't need to worry about the sample size in advance - just keep testing and wait for the results to converge.

Having said that, Bayesian calculators will sometimes throw up funny results too, but usually only on very small samples. The sample size you've got there would be fine for a Bayesian test.

07-13-2016 11:40 AM #3 crysper (Member)

Quote Originally Posted by caurmen View Post
I generally avoid using frequentist split-testing approaches precisely because of the need to determine a sample size in advance (and the fact that said sample size tends to be very large, as the link Crysper shares will show).

If you use a Bayesian approach instead you don't need to worry about the sample size in advance - just keep testing and wait for the results to converge.

Having said that, Bayesian calculators will sometimes throw up funny results too, but usually only on very small samples. The sample size you've got there would be fine for a Bayesian test.
Exactly. Shameless plug, on AffKit there is a tool called When To Cut Landers http://www.affkit.com/tools/when-to-cut-landers that uses bayesian model and tells you which lander is the winner and which to stop.

The next version of LPOptimizer will use bayesian by default(with the option to change back to z-test)

You can also use https://www.peakconversion.com/2012/...al-calculator/ . Over 90% probability of being best is a good indication.

07-13-2016 11:56 AM #4 caurmen (Administrator)

Nice, didn't know you were using Bayesian models now!

I've been investigating Machine Learning approaches recently: still comparatively early stages, but I'll be discussing them during my talk at AWE next week.

07-17-2016 09:57 PM #5 bobliu (Member)

Quote Originally Posted by caurmen View Post
Nice, didn't know you were using Bayesian models now!

I've been investigating Machine Learning approaches recently: still comparatively early stages, but I'll be discussing them during my talk at AWE next week.
Look forward to that. Will the talks be available to purchase for non-attendees?

Home > General > Affiliate Marketing Forum