hello....
been stuck with this and don't really know how to approach it.. maybe some advice from the pros will do.
here goes.
scenario:
- new offer you want to promote and you are building landing pages
- you start with a/b testing to find out which lander (with drastically different designs and layout) gets the best CTR
- then you proceed to do multivariate testing (changing headlines, images but keeping the same layout to find the winning combination of headline + image + whatever) with that specific lander to try to increase the CTR
my questions are:
- why not do a/b testing WHILE doing multivariate testing?
- is it because you need a large sample size/ test budget for this to work?
- rather, do you guys do multivariate testing at all? is it necessary?
ideally, you would want a specific lander for every demographic because it relates to everyone differently... but in reality, this just isn't possible. best thing one can do is dynamic keyword insertion to make it more relevant
i know i might be just thinking too much...
All depends on how much traffic you have.
A/B testing as a whole takes much less traffic to call a winner. We use google weboptimizer most of the time from the start.
The only reason most people don't do A/B and multivariate at the same time is that it needs a lot of traffic to get statistically significant data.
So as you said you go with A/B first and then move to multivariate.
Also , during multivariate , limit the number of tests you do at a time so you can acquire data faster.
2 diff styles x 4 headlines each x 2 dif hero shots x 2 dif call to actions = 32 variations depending on the traffic thats gonna cost a bunch to test out to get a significant result.....
its balance between risk and reward.....and you don't even know if the offer really works or not....also there is offer risk...i.e. the offer goes down just as you finish testing.....
I would A/B test first regardless because it
A) requires less traffic before you chose a more profitable pathway -> losing less money faster
B) allows you to see how well the offer is converting which is probably the most important thing, period. If your offer doesn't convert with your demo/angle/traffic source/basic lander etc then you need to know asap and not beat around the bush waiting for statistically significant data from 10+ variations.
^^^^ This...
Zeno has got it spot on.
Multi variant will in most cases take longer and cost more money.
So - do A/B until your profitable. Then if you want, move to multivariant.
This applies to your ads as well.
1st Image
2nd Headline
3rd Body
If you test all 3 at the same time and have say 10 variations of each, then although you will find the best combo with this approach its more likely that you will be further in the red and therefore take longer to get back to breakeven