Hello,
Wondering if someone could help me understand why banner2 wins - been trying to wrap my head around this, but having troubles understanding.
I have banner1 - 239,070 impressions, 689 clicks, 3 leads
I have banner2 - 140,710 impressions, 435 clicks, 1 lead
The split test calculator says that banner2 wins by 73.45% Confidence ..
-it obviously does not factor in total leads generated- but what if it did?
or better yet, how could I factor in total leads generated into the formula when calculating the stat?
In your example, banner2 doesn't really "win", it's just a bit more likely to be the better performer of the 2.
As mentioned in this thread you should aim for at least 90 % confidence.
Also, you'll find that the calculator produces weird results at low numbers of conversions. I'd recommend getting some more conversions on both banners before running them through the Bayesian split test calculator.