I have three questions regarding statistical significance:
1. I'm not sure when I should cut carriers. To give you an example, here's my data:
carrier1:
visits: 175
leads: 4
CR: 2.29%
carrier2:
visits: 1077
leads: 10
CR: 0.93%
carrier3:
visits: 603
leads: 5
CR: 0.83%
carrier4:
visits: 154
leads: 0
CR: 0
I'm using these tools to calculate statistical significance:
http://www.usereffect.com/split-test-calculator
http://www.splittester.com/
The second tool doesn't allow entering 0 (I use CTR as CR and clicks as leads) so I need to use the first tool if I want to compare these carriers to carrier4.
Should I compare carrier1 to carrier4 (the best performing one to the worst performing one) or carrier3 to carrier4 (next worst performing one to the worst performing one)? When comparing carrier1 to carrier4 the trend favors carrier (90%), when comparing carrier3 to carrier4 I obviously need much more data. Should I cut carrier4 or wait for more clicks? I've been displaying my ads for a week.
2. When should I cut carriers who have a lower EPC than eCPC? How many clicks approximately should I have to make a statistically significant decision?
3. How many clicks should I get to a landing page to be sure that it performs better? 100, 200, 500? Sometimes with low volume of clicks results can still be statistically significant but as far as I know I shouldn't only take into account statistical significance but also the number of clicks (there's a chance that I will get 5 leads with first 50 clicks and then nothing for 500 clicks; with 500 clicks it isn't likely to happen).
Update:
Okay, I did some digging and found this super helpful gem that answers most of my questions: http://stmforum.com/forum/showthread...l=1#post112625 Seriously, caurmen, you're the best!
Heh, thanks for digging that out - I'm writing the Step-By-Step guide on statistical significance right now, and it's handy to be able to refer back to what I've written in the past!