Home > The Newbie Zone > Questions and Answers

A/B Split Test Calculation taking revenue into account (5)

12-08-2015 07:53 AM #1 stuckinaction (Member)
A/B Split Test Calculation taking revenue into account

I can't seem to figure out how to approach this.


A/B significance tests like https://vwo.com/ab-split-test-significance-calculator/ tell you which ad is better based on clicks.


But heres my question.

Lets say I have this situation.

600 clicks, 5 conversion, $42.00 total revenue from all conversions (amazon ad for books)
800 click, 1 conversion, $72.00 total revenue from 1 conversion. (amazon ad for a tv)


What calculation takes into consideration these 3 variables?

12-08-2015 02:06 PM #2 matuloo (Legendary Moderator)

These calculators work the best when you have just one type of product, ideally with the same payout, in your example there are two very different types of products, books vs TV.

Assuming that the first ad is promoting some book and the other one is promiting a TV set, you are actually comparing apples and oranges here as these two should have been completely separate funnels from the begining.

Or am I missing something here?

12-09-2015 07:04 AM #3 thuglife (Member)

Quote Originally Posted by stuckinaction View Post
I can't seem to figure out how to approach this.


A/B significance tests like https://vwo.com/ab-split-test-significance-calculator/ tell you which ad is better based on clicks.


But heres my question.

Lets say I have this situation.

600 clicks, 5 conversion, $42.00 total revenue from all conversions (amazon ad for books)
800 click, 1 conversion, $72.00 total revenue from 1 conversion. (amazon ad for a tv)


What calculation takes into consideration these 3 variables?
Option A: 600 clicks for 5 conversion means it takes 120 clicks to get a conversion, and you've proven that 5 times.
Option B is still unproven as far as repeatability is concerned.

I'd use a Bayesian calc instead and only test one thing at a time. (either test the traffic source, or test the offer, doing both means now there are 4 cases you'll need to account for). This: http://www.peakconversion.com/2012/0...al-calculator/

Just run it as long as it's profitable.

12-09-2015 09:53 AM #4 stuckinaction (Member)

Thanks for the reply


I realize I need to word the question better.

How do I determine which ad to continue running and which one to cut when the payouts from conversions are not a fixed and amounts are variable?


Example:

You have 1 site that has 1 placement. You rotate the 2 different ads equally and each conversion leads to different commission amounts

A. Traffic source A ----> Ad X (Book) ----> Offer for ad X on Amazon leads to Q $ amount


B. Traffic source A ---> Ad Y (TV) -----> Offer for ad Y on Amazon leads to Z $ amount


So now how do you determine at point it's statistically significant to stop running either ad?

Obviously ad ad A is profitable... but when it comes to Ad B, how much more do run it and at what point do I cut it?

12-09-2015 01:18 PM #5 matuloo (Legendary Moderator)

If I understand it correctly, you are basically trying to figure out, whether its better to promote TV sets or books on one single spot on one single site, am I right?

In this case, you need to treat the two cases as completely separate campaigns - You have different ads and different products, its like 2 totally unrelated campaigns so you need to judge them separately too.

You need to run both long enough to get statistically significant results for each of them SEPARATELY. Then you compare the final ROI of these two ang go on with the one where it was higher.

Home > The Newbie Zone > Questions and Answers