Home > Paid Traffic Sources > Mobile

Does anyone run Fractional Factorial multivariate tests? (7)

08-13-2014 10:55 AM #1 jordanfan20 (Member)
Does anyone run Fractional Factorial multivariate tests?

Okay I get it, running a full factorial multivariate test would be extremely expensive and time consuming--additionally, different variables likely have significantly more weight (I.E Banner image probably affects CTR much more than the font-type) so it would be a waste of time to systemically test each variable early on that is unlikely to have a substantial impact.

What I don't get then is why people don't run fractional factorial multivariate tests?

To reiterate what I said above, the principle of Sparsity-of-effects suggests that each variable has an assigned weight. Over years of PPC advertising it seems as though you could get a good feel for which variables have the most impact on a campaign. For example, with FB advertising you could probably create a weighted formula where:

FB CTR = 4 (image) + 3 (header image) + 2 (ad copy) + 2 ( image manipulations)

Then use the weights to determine the elements that factorial testing would have the most substantial impact on.

The big advantage to this approach over A/B testing would you would have a much better idea of how the individual variables contribute to the big picture.

One reason I don't think anyone does this type of testing is maybe because of the lack of availability of calculation tools. What are other reasons?

Also do any type of tools exist that do this, at least that are affordable? My searches have brought me up nothing.

08-13-2014 12:48 PM #2 32rfs23f23f (Member)

Ive been using fractional factorial in my custom system for campaign managament but the results were not. The Frankenstein effect was too big, eg best image for ctr + best headline for cr + best lp for cr, lead to loss. The system did not perform as it should. The problem is the assumption that the variates are not connected and do not influence each other. This is not true. They always influence each other.

Currently I use instead of this predictive analysis. A shitload of variatiosn (banner /lp) combos are made and are killed when 10% of CPA cost for them is acquired (eg as low as 0,2 $). Works much better, predictable and stable.

08-13-2014 01:13 PM #3 jordanfan20 (Member)

Quote Originally Posted by 32rfs23f23f View Post
Ive been using fractional factorial in my custom system for campaign managament but the results were not. The Frankenstein effect was too big, eg best image for ctr + best headline for cr + best lp for cr, lead to loss. The system did not perform as it should. The problem is the assumption that the variates are not connected and do not influence each other. This is not true. They always influence each other.

Currently I use instead of this predictive analysis. A shitload of variatiosn (banner /lp) combos are made and are killed when 10% of CPA cost for them is acquired (eg as low as 0,2 $). Works much better, predictable and stable.
Great post. That makes sense that that is an issue. I sent you a message about a couple question I have about the predictive analysis you use.

08-14-2014 04:05 AM #4 zeno (Administrator)

If it's for Facebook specifically, I can guarantee you that the volatility of Facebook's ecosystem and the delivery systems are going to confound a lot of your tests, unless they are over a substantial time period.

08-14-2014 08:24 AM #5 caurmen (Administrator)

Yes, I'm developing a fractional factorial system right now. Not ready for public use yet, though!

@32rfs23f23f - good experiment design should account for interactions. What set of matrices were you using for the tests?

08-14-2014 10:07 AM #6 crysper (Member)

Quote Originally Posted by 32rfs23f23f View Post
Ive been using fractional factorial in my custom system for campaign managament but the results were not. The Frankenstein effect was too big, eg best image for ctr + best headline for cr + best lp for cr, lead to loss. The system did not perform as it should. The problem is the assumption that the variates are not connected and do not influence each other. This is not true. They always influence each other.

Currently I use instead of this predictive analysis. A shitload of variatiosn (banner /lp) combos are made and are killed when 10% of CPA cost for them is acquired (eg as low as 0,2 $). Works much better, predictable and stable.
I got the same results with the fraction factorial system. I get users asking why multi-variate, why I don't show the % weight for each element in conversion and when I tell them that it doesn't really matter they get cold. The assumptions simply don't work as it should because the user reaction is totally different for each combination, the % of the weight of the winner element should be very high which is rarely the case. Of course, we are not talking about changing a color of a barely visible line in a lander and for this you need to test twice versions.

I haven't set up a system yet, but with some basic statistics you can easily tell from early on which versions won't do good, based on % of CR/CTR between the top converting versions of the lander. When there is a very low chance to beat the top performers, just cut it out.

08-14-2014 05:05 PM #7 alexpte (Member)

Quote Originally Posted by 32rfs23f23f View Post
Currently I use instead of this predictive analysis. A shitload of variatiosn (banner /lp) combos are made and are killed when 10% of CPA cost for them is acquired (eg as low as 0,2 $). Works much better, predictable and stable.
This sounds really interesting but I'm not sure I understand how you can cut variations after only 10% of the CPA, would you mind elaborating a bit? Do you have a model of successful creatives from previous campaigns and cut by comparing the performance these previous tests?

Thanks!

Home > Paid Traffic Sources > Mobile