Home > The Newbie Zone > Questions and Answers

---

Always split-testing 50/50? Why not 30/70? (10)

---

Hi there.

I have a question. Lets say you have a winning lander/offer combination that brings you a decent ROI and you want to put another offer in the mix and split test them. Do you always do 50/50 split test or maybe sometimes 30/70?

If you are spending like 100-200$ a day on a traffic, then 30% of that for the new offer should give you enough data to see how well it performs without sacrificing the profits if the offer is not good.

Anyone doing it like this, or do you always do 50/50 tests only?

---

You can do both actually, but be aware of some pros/cons.

50/50 testing will give you the most accurate data, period. So if you wanna run quick split test with as balanced results as possible, you need to go for 50/50.

However, assuming you already have a winner, running the test on a lower % will not cost you so much in lost revenue. Keep the % on a decent level, so not 5%, and you will still get an idea about the performance of the tested offer.

I suggest to do a combination : start on lower % (at least 20-30%) and wait for enough data. If the offer shows potential, increase to 50/50 and run a proper split test to see if it can outperform the original offer or not.

This is in my honest opinion the best way to do it.

---

Yes, my question was based on the fact that I have a winning lander/offer combination and network just introduces new(similar) offer that I would like to test.

Makes a lot of sense to test it like you suggest. Thanks.

---

30/70 tests require a lot more data and time to achieve the same level of significance as 50/50 tests.

The same thing with multi-variate VS A/B tests.

---

True, I wouldnt even suggest trying to reach significance and compare 2 offers on 70/30, just use it to determine if the offer has potential or not, then run at 50/50 to see which of the offers is the winner.
70/30 or 70/20 just to minimize costs of the initial testing.

---

I've definitely done things like 30/70, 20/80, and even less then that before to test a new offer on some of my profitable campaigns. It all depends on your goals, and if your goals involve keeping a maximum of profits (however much you want) while still split testing an offer, there's no bible or hard/fast rule that says you have to do it 50/50 -- it's simply a matter of preference if you understand it'll take longer to reach statistical significance and if you're okay with it, no reason why you can't imo. There are considerations like, if they're a new direct advertiser and you want to see how the offer performs even with a little data before dedicating more resources to it, or your time i.e you're going to be super busy the next day and won't have time to work on the campaign like if you're travelling or going to a wedding and wanted to stretch it out intentionally.

---

Some of the comments on this thread are not exactly correct, but it would really take too long to write out how to do it properly (and you would need a very strong background in maths and stats to follow).

The optimal way to do this is via what is called an epsilon greedy multi-armed bandit approach which dynamically skews most of your traffic to the campaigns which are performing the best (exploitation), and less traffic to find ones that may be better (exploration).

Google's Steven Scott gives a pretty clear discussion https://support.google.com/analytics.../2844870?hl=en if you are so inclined.

---

I agree with this, and keeping it simple. The goal here is ROI, not statistical significance. lol

---

Read a little about an epsilon greedy multi-armed bandit approach, but it seems to me that that method is more useful when your traffic is very limited and it would take a very long time to use conventional 50/50 method to hit statistical significance. When you get a couple hundred thousand impressions per day you can reach statistical significance in just a day or two.

---

No. It is a function of what you are maximising for: profit instead of the right answer. These are two different things. You can be right and lose money, since there is a cost associated with each suboptimal test. Unless your traffic is free, which it isn't, you are losing money every time you are pushing traffic to a nonconverting page within the context of an A/B test.

Most people, even on STM, don't know how to do even simple A/B testing properly. Instead, they engage in what is known as p-value mining or data peeking, which by its very nature completely invalidates the statistical test.

---

Home > The Newbie Zone > Questions and Answers