Home > General > Affiliate Marketing Forum

How to Cut Offers with Different Payouts (8)

09-30-2016 05:44 AM #1 vortex (Senior Moderator)
How to Cut Offers with Different Payouts

For a long time, I didn't have a tool for cutting offers I was split-testing.

It wasn't too bad when the offers all had similar payouts - I would just use the calculator at peakconversion and cut at a higher probability of being best to offset the inaccuracy.

However, when the offers had very different payouts, the only option I had was to generate lots of conversions, until the difference in performance was so drastic that even a turkey with half a brain missing would be able to tell without a shred of doubt which offer was the winner.


And then...edgekaos saved the day with his discovery of the win-vector calculator (thanks edgekaos!):

http://stmforum.com/forum/showthread...ou-pick-offers


I've been using it ever since. For those of you that have been using the peakconversion calculator, you won't even need to read this brief tutorial. For the rest of you - please read on.



**************************************

Basic Instructions

**************************************



1)Go to this tool:

https://win-vector.shinyapps.io/CampaignPlanner_v3/

(Note: If the tool happens to be down, please see this post.)

2)Click on "Evaluate Campaign".

3)Input the stats for the 2 offers you want to compare.

"Actions" are the number of impressions on the offer - so if you're direct-linking in pop, it would be the same as the number of "Visits" in your tracker; if you're using a lander with pop, "Actions" will be the number of "Clicks" in your tracker; if you're running display traffic without a lander, it will be the number of banner clicks; and if you're running display traffic and using a lander, it will be the number of lander click-thrus.

"Successes" are the number of conversions for each offer.

"Success Value" is the payout for each offer.

Just leave the "Target Value per Action" and "Scale Factor" at default values - we don't need to use those for what we're trying to do.

4)There's no button to press - just scroll to the bottom of the page and you'll see a percentage - this represents the probability that the top offer is better than the bottom offer.





**************************************

Case 1: When Comparing 2 Offers

**************************************

Example 1: Statistical Significance Not Yet Reached

Offer 1: Actions=10000, Successes=32, Success Value=0.75
Offer 2: Actions=10000, Successes=75, Success Value=0.25



As you can see, Offer 1 has an 88% probability of being best compared to Offer 2. In this case you'll need to keep running traffic to reach at LEAST 90% probability of being best, to cut the inferior offer. Actually, I would strong recommend that you wait until 95%+ probability of being best before cutting the inferior offer. The offer can make or break the campaign, so it would be good to be more accurate when it comes to cutting offers.


Example 2: Statistical Significance Has Been Reached

Offer 1: Actions=10000, Successes=35, Success Value=0.75
Offer 2: Actions=10000, Successes=75, Success Value=0.25



As you can see, Offer 1 has achieved 95% probability of being best, which means you can cut Offer 2.


Important: In both examples above, the top offer happens to be the better offer. In cases where offer 2 is better, you'll see a percentage of LESS THAN 50%. In those cases, you'll need a probability of being best of BELOW 10%, or better, below 5%, in order to cut the top offer.



**************************************

Case 2: When Comparing >2 Offers with Similar # of Impressions

**************************************

The tool only allows you to compare 2 offers at a time. So what if you were running more than 2 offers?

In that case, you'll need to:

1)Find the current-best or in-the-lead offer.

2)Compare EACH of the other offers to that offer, and cut those that meet the >95% / <5% criterion.


This process is the same as the one I've described in this thread:

http://stmforum.com/forum/showthread...Banners-Part-1

Because the process is the same, I won't waste your time by doing another step-by-step example. (I could do this upon request - please let me know below if you'd like to see a detailed example tailored to this calculator.)



**************************************


And that's it - short and sweet! Questions? Comments? You know what to do!



Amy

09-30-2016 09:58 AM #2 matuloo (Legendary Moderator)

Thanks for posting this Amy, working with offers that have different payouts can get very confusing, especially for newbies! I know what I'm talking about, been there, done that

10-13-2016 08:40 AM #3 Mobidea (Veteran Member)

Hi, vortex!

Hope you´re having a great day!

I really like your explanation of AB testing. It's very useful and I take advantage of this practice quite often. Nevertheless, it's normal for me to receive questions from newbies regarding the amount of offers to be tested at once, the procedure, etc. That's why I wanted to sum up my approach to this kind of tools and solutions as quickly as possible.

I usually do this by split testing with an equal distribution of traffic where I simultaneously test 2 or 3 offers maximum. You can test as many offers as you want. However, don't forget you need reliable data and time is always a key factor. After we gather the numbers, I check the statistics and replace the worst offer by a new one or simply reduce the amount of traffic reaching the worst offer.

What if the performance is the same? In that case - depending on the granularity of data you've got - I'll optimize the offers according to the creatives and target their aim. They might be the same overall. However, if adumbrated in detail, you can make sure they're optimized and working perfectly!

Moreover, it´s important to realize that sometimes reality can pull the rug from under out feet and you don't get the chance to achieve the 90-95% probability or even distribute the traffic equally depending on the platform used. In that scenario, I'll usually just take a look at the offers' performance, analyzing the CR and EPC and decide whether or not to cut. Another tricky thing is that marketers frequently choose not to wait for the data to be reliable enough due to the sample you obtain. In that case, calling the AB test might be too soon since the inferior variant may appear as a superior one and you'll move forward with the better, winning variant which will damage your performance.

Summing up: all methods of split testing should ultimately mean more profitability to you. Using this method depends a lot on your traffic, needs and resource capabilities.

Hope this helped!

Cheers!

10-13-2016 11:26 PM #4 vortex (Senior Moderator)

Quote Originally Posted by Mobidea View Post
Hi, vortex!

Hope you´re having a great day!

I really like your explanation of AB testing. It's very useful and I take advantage of this practice quite often. Nevertheless, it's normal for me to receive questions from newbies regarding the amount of offers to be tested at once, the procedure, etc. That's why I wanted to sum up my approach to this kind of tools and solutions as quickly as possible.

I usually do this by split testing with an equal distribution of traffic where I simultaneously test 2 or 3 offers maximum. You can test as many offers as you want. However, don't forget you need reliable data and time is always a key factor. After we gather the numbers, I check the statistics and replace the worst offer by a new one or simply reduce the amount of traffic reaching the worst offer.

What if the performance is the same? In that case - depending on the granularity of data you've got - I'll optimize the offers according to the creatives and target their aim. They might be the same overall. However, if adumbrated in detail, you can make sure they're optimized and working perfectly!

Moreover, it´s important to realize that sometimes reality can pull the rug from under out feet and you don't get the chance to achieve the 90-95% probability or even distribute the traffic equally depending on the platform used. In that scenario, I'll usually just take a look at the offers' performance, analyzing the CR and EPC and decide whether or not to cut. Another tricky thing is that marketers frequently choose not to wait for the data to be reliable enough due to the sample you obtain. In that case, calling the AB test might be too soon since the inferior variant may appear as a superior one and you'll move forward with the better, winning variant which will damage your performance.

Summing up: all methods of split testing should ultimately mean more profitability to you. Using this method depends a lot on your traffic, needs and resource capabilities.

Hope this helped!

Cheers!
Hi Mobidea and thanks so much for sharing your approach!

What you said has triggered a couple of thoughts:

1)I find that testing a lot of offers at the same time, often doesn't take as much test budget as many people may believe. What I do see happening more often than not, is that a couple of the best offers of the bunch would make the first few conversions, reaching statistical significance for a lot of the inferior offers to be cut.

2)The problem with "replacing" an inferior offer with a new offer, is that once a new test candidate is introduced to the test, it starts a new split-test, and all the data collected from before that point will go to waste. This is because day-to-day and hour-to-hour fluctuations can skew results quite significantly (no pun intended!), so that once a split-test begins, no candidates should be added until the current round of testing is finished and a winner (or more than one in some cases) is declared. Comparing stats between landers that have been collected during different time periods would be way too inaccurate.

This kind of discussion is invaluable and enjoyable! Thanks very much and it's always a joy bumping into your valuable insight on various threads.


Amy

11-04-2016 03:27 AM #5 victor s (AMC Alumnus)

Hi Vortex,

Quick Q, If I'm running display traffic with banners & a landing page, I should use the number of click-thrus from the landing pages to the offer to decide what lander to cut? Is that correct?

So like on this image below, I would type in Clicks (265 - 229) into the "Actions" bar? Or should I type in the number of Visits?



Look forward to your reply.

Thanks.

11-04-2016 04:01 AM #6 vortex (Senior Moderator)

Quote Originally Posted by victor s View Post
Hi Vortex,

Quick Q, If I'm running display traffic with banners & a landing page, I should use the number of click-thrus from the landing pages to the offer to decide what lander to cut? Is that correct?

So like on this image below, I would type in Clicks (265 - 229) into the "Actions" bar? Or should I type in the number of Visits?



Look forward to your reply.

Thanks.
Absolutely correct! You need to use the number of lander views - for pop traffic this will be "Visits", but for display traffic this will be "Clicks".

However, for landers you'll want to use the calculator described in this thread instead:

http://stmforum.com/forum/showthread...Banners-Part-1

When cutting landers using the peakconversion calculator: If running multiple offers, they should have similar payouts, because the calculator assumes all conversions to have the same value. So if the payouts are very different, then your results will be inaccurate.

Thanks for your question!


Amy

11-04-2016 04:13 PM #7 victor s (AMC Alumnus)

Thank you for the reply!

I made a mistake and meant to write "offer", not lander.

...from the landing pages to the offer to decide what offer to cut?
Nonetheless you answered my question, so thank you for that!

11-07-2016 01:05 AM #8 vortex (Senior Moderator)

Quote Originally Posted by victor s View Post
Thank you for the reply!

I made a mistake and meant to write "offer", not lander.



Nonetheless you answered my question, so thank you for that!
For cutting offers please see this thread instead:

http://stmforum.com/forum/showthread...211#post289211

And yes - you'll use clickthroughs from lander to offer, i.e. offer views, when deciding which offer to cut.



Amy

Home > General > Affiliate Marketing Forum