Home > General > Affiliate Marketing Forum

---

How to Cut Placements, Landers and Banners - Part 1 (41)

---

NOTE: If you only want to cut placements, please see Part 2.


**********************************
Intro
**********************************

A question I hear a lot of people ask is: "How do I know when to cut banners/landers/placements?"

Caurmen has explained the statistics involved in AM in more than one thread in the past:

http://stmforum.com/forum/showthread...he-Winning-Ads

http://stmforum.com/forum/showthread...ata-Read-This!

http://stmforum.com/forum/showthread...ats-Calculator

Now, I'm FAR from being an expert in statistics. In fact, I failed my intro to stats course in first year university decades ago...

However, building on what I learned from Caurmen's wisdom, I was able to figure out some basic stuff as I gained practical experience in analyzing my own campaign stats. I still don't know much, but enough!

I must have answered the question above for at least 10 friends in skype chats over the past 2 months, and have seen at least 2 threads in this forum asking the same over the past 2 days - so here I am writing a 2-part series to share what I know.

Let's get started shall we?



************************************************** ********
PART 1: How to Identify the Best Banners and Landers
************************************************** ********

Oftentimes we need to compare a group of something to know which one is best. For example:

-Created 5 different types of landers and want to see which one performs best.
-Made 2 versions of a well-converting lander to split-test, to try to improve CR further.
-Have 20 banners running and want to identify the best one, then pause the rest to maximize ROI.

To identify the best in a group of anything, this would be the tool to use:

http://www.peakconversion.com/2012/0...al-calculator/


Here's how to use it...


Step 1 - Identify the current-best

Say we're testing 7 landers, landers A/B/C/D/E/F/G. We have the following data from our tracker:




What the fields in the calculator mean:
"Trials" = number of impressions on the lander (or if you're using banners, the number of clicks to the lander)
"Successes" = number of conversions

The split-test calculator can only test 4 items at a time. If we had 4 landers or fewer, we'd simply input all the data, click "Calculate", and the one with the highest "Apprx probability of being best" would be our "current-best".

However, since our example has 7 landers, here's what we can do - test A/B/C/D, make a note of the best (lander C). Then test E/F/G and make a note of the best (lander F). Then test the best from each lot (lander C vs. lander F) to find the winner - in this case, lander F.

Round 1 - Landers ABCD:


Round 2 - Landers EFG:


Round 3 - Lander C vs. F - K.O.!


It's not unlike how tournaments are set up, where winners from the previous round are set to compete in the current round and so on until the final winner emerges.


Step 2 - Identify items ready to be cut

Next, we use the calculator to compare the "current-best" from the first step - in this case lander F - against EACH of the other landers. If the "Apprx probability of being best" of a lander is less than 10%, it should be OK to cut that lander from rotation. If none of the landers are at less than 10%, DO NOT CUT - we allow it to collect more data.

Lander F vs. Lander A:

Verdict: Cut Lander A

Lander F vs. Lander B:

Verdict: Cut Lander B

Lander F vs. Lander C:

Verdict: Let Lander C Continue

Lander F vs. Lander D:

Verdict: Cut Lander D

Lander F vs. Lander E:

Verdict: Cut Lander E

Lander F vs. Lander G:

Verdict: Let Lander G Continue

Note: The lower the "Apprx probability of being best" of a lander is when compared to the current-best, the more certain you can be that it will continue to be worse than the current-best in the long run. For example, say I was testing a bunch of landers that were the same except for the image, < 10% would be enough for me to cut a lander. However, if I was testing 2 landers I spent 2 days to mod, I may test until one of them falls below 2% or even reaches 0% "Apprx probability of being best" before I'd cut it - because I put so much effort into the landers I want to LITERALLY be 100% sure not to cut the wrong one!

Another Note: You may be tempted to cut something before it reaches 10% or less. Please REFRAIN from doing that! You may feel that by cutting earlier you'd be saving money, but what you'd actually be doing is make an ad hoc decision that is not statistically sound, which effectively renders your entire test a waste of time and money. I've been guilty of that so many times I've lost count! When your "winners" turn out to be "losers" you'll kick yourself for it!

Either test until results reach statistical significance, or don't bother to test at all!


Step 3 - Repeat until only 1 item is left

Now, simply repeat Steps 1 & 2 every so often for the remaining landers, to cut gradually until you're left with the winner!

After that, you can make more landers to test against your winner and start the cutting process all over again. Keep beating your winner to continuously improve on your ROI!

NOTE: If you see 2+ landers going head-to-head, you may not want to spend a ton of time and money just to wait for a winner! Please see caurmen's post:

http://stmforum.com/forum/showthread...ugh-Clicks-Yet

Specifically, read this section:

****************************************
Finally...
****************************************

The same process can be applied to banners as well, where "Trials" would be the number of impressions on a banner.

That was simple, right? Told you I only knew just enough!

Questions? Comments? See errors in my logic? Have additional knowledge to impart on the subject? Please post below!

EDIT: Part 2 is now DONE! You can find it here.



Amy

---

OH?

So instead of putting in Trials as clicks, we can put impressions instead?

Shieet... Okay, now I got everything I need. Thanks Amy. I think I'll be restarting my campaign again

---

Hey Letsgo!

Yes "Trials" are basically the number of times the lander or banner is seen (depending on which one you're wanting to cut). Clarification: If you are running display traffic where visitors are clicking on banners to arrive at landers, and you're trying to cut landers, then "Trials" would be the number of banner clicks, which equals the number of visits to the lander.

Just remember "Trials" are always the number of impressions to whatever it is you're trying to cut, and you'll be fine. If you have some actual stats, feel free to post them here and I could run them through the calculator for ya.

Amy

---

Nice, thanks for the effort this is a good guide!

---

Have you looked at this split tester http://abtester.com/calculator/

It tests against a control and gives you Confidence percentage against the tested control.

I believe the peakconversion split tester ranks the testing against each other rather than a control.

---

They're actually the same sort of tool. Actually "Step 1: Identify the current-best" is essentially my way of establishing a temporary "control". Then, at the end of the first round of testing, the winning lander/banner becomes the real control. In the next round of testing you'd be trying to beat this control. Every time you're able to beat and replace your control, you increase your bottom line.

All split-testers serve the same purpose: compare and find the best.

I'm also enjoying our discussions in the other thread - thanks for making me think!


Amy

---

I was thinking maybe using both, because in the beginning when you are just testing LPs for a new campaign, you don't have a control. So you may use the peak conversion's split tester to decide 1 LP out of 4 being tested to go with.

After you got your first LP, you start split testing against that one as the control using http://abtester.com/calculator/

I hope I'm not mixing two different kinds of calculations. Because if you plug in the numbers into each calculator they do spit out different numbers that mean different things.

I was planning to go with http://abtester.com/calculator/ and use the PHP code provided to automate things slightly. Did you have any luck with getting something in Excel?

---

The only difference between them, is that the peakconversion calculator compares up to 4 things against each other and tells us which one is best, whereas the abtester calculator compares each "treatment" to the control.

I do agree with you that for "Step 1 - Identify the current-best", peakconversion would be better, and that for "Step 2 - Identify items ready to be cut", abtester may be more convenient.

However you choose to do it, the results are COMPLETELY THE SAME. I'd settle for keeping things easy by using a single calculator, but if you prefer using 2, there's no reason why you shouldn't!

As for automation - I didn't even bother to automate this, because it only takes a minute to do it manually. Unless you're testings hundreds or thousands of different banners/landers, there would be little point in automating the process IMHO. Unlike in the case of cutting placements where we need to check thousands of placements repeatedly to see which ones are ready to cut - now that would be impossible to do accurately and quickly without a spreadsheet to automate the process.

As always, nice discussion!


Amy

---

Amy - Thanks for the great stuff. I have been scratching my head the past couple of days regarding cutting criteria and stuff and this post comes at an opportune moment for me.

I would like to add the motivation as to why STEP-2 above is required. The grouping of A/B/C/D in group 1 and E/F/G in group 2 is actually arbitrary. An alternate grouping could be A/B/C/F in group 1 and D/E/G in another group. And that grouping would knock out lander C. So, by bringing in STEP-2, we recheck which lander apart from F could pass muster.

(Regarding different groupings, I am too lazy to go into how many permutations/combinations could be achieved. Moreover, it would only serve an academic purpose.)

TENNIS ANALOGY
If very strong players (say Pete Sampras and Andre Agassi) were made to play a championship match in round 1 of Wimbledon, one of them is going to get knocked out. But we don't want that. We want the best players to "bubble up" from the different rounds so that we have an exciting denouement. That is why the top seeded players are staggered across different rounds so that the weaker players get progressively eliminated and we have strong players fighting out in the later stages. Same with strong/promising landers/banners. In fact, we could decide to run the top 2/3 landers/banners instead of just 1.

---

Hi srinivasan! Glad you're finding this info relevant and helpful. You're such an eloquent speaker.

It doesn't matter how the landers are grouped when testing - the same winner will emerge. All we're trying to accomplish in step-1 is identify the currently best-performing lander, so we'd have a "control" to compare each lander to in step-2.

Without identifying a current-best, we wouldn't know which lander to compare to or apply our 90%/10% cutting criterion on!

Hopefully it's clear now as to why both steps 1 and 2 are necessary. If I'm still being confusing please let me know and I'll try to explain this better.

As for your interesting tennis analogy: If the process of analyzing stats was as enjoyable as watching a tennis match, then there would be merits to drawing out the process by introducing as many steps as possible! However, since efficiency is what we're after, we want to get the "fight" over with as quickly as possible, by identifying the strongest player and using him as a benchmark to assess all others and thereby eliminating the weaker.

I completely agree with you on this! The focus of this thread is on using the split-test calculator to eliminate all except the best, so I tried not to get into lander testing methodology too much. Say if I was testing 10 landers that were the same except for the headline - in this case I would use the method above to eliminate all but the best. Or, say I've set up a new campaign for a new offer, and I wanted to use only the best banner to cut placements - I would use the method above to find that best banner. (By doing so I'd be using that campaign to cut placements only - and by putting my best banner to the job I'd be doing so I'd be spending less. I would then set up a separate campaign to test banners.)

However, as you've pointed out, there are times when you DON'T want to eliminate all but the best. Say I'm wanting to test 10 different types of landers to see which ones are more effective. In this case I would use the method above to cut only the worst ones. Then I would start a separate campaign for EACH of the remainders that look promising, and in each campaign I would continue split-testing each lander with versions of itself to optimize it more and more. In fact, I have current campaigns where I'm using this approach.

Thanks everyone for chiming in with your insight - I'm enjoying these discussions.


Amy

---

Hi Amy,

Yes. I forgot to add in my previous post that the process as it stands is simple/self-sufficient and it will work just fine. In making the tennis analogy, I missed adding that.

---

Hey Amy,

Thanks so much for the great tutorial - this is really helpful.

I just have a couple of questions to make sure I'm doing this right. My situation: I'm testing 6 landers that are vastly different from each other (i.e. no incremental improvements yet).

1) To compute the numbers, I would obviously look at the # of impressions and conversions since I started the campaign, not daily stats. This makes sense to me, but wanted to verify that this is correct?
2) I don't have quite as many impressions as you do above - it's more like 1,500 impressions per lander and fewer than 10 conversions. If I get results like one lander has less than 5% probability of being the best vs. the top, I can still cut it - even though the number or trials / impressions is not that high, right?

Many thanks for your advice all over the forum, it's been really helpful.

---

You're absolutely correct!

Right again! The calculator takes all that into account already. And if you're down to 5% probability, you can cut that lander with a pretty high level of confidence. If it's a lander you've spent a lot of time crafting, you could wait until 0% probability even - just to be 100% certain it's bad.

Thanks and you're welcome! Just doing my job.


Amy

---

Hey Amy,

Thanks again for your fast response to my question the other day.

There's another (related) question I was hoping you could help me with:

Currently, I'm running four campaigns to promote one offer in one geo:

1) Domain Redirect / WIFI
2) Domain Redirect / Mobile
3) POP / WIFI
4) POP / Mobile

I'm currently running the same 3 landing page variations for each campaign

As for the landing-page optimisation, would you:

a) Sum impressions and conversions across the 4 campaigns, and then plug them into the tool
-or-
b) Do the analysis individually for each campaign

Obviously, (a) will give you a quicker response due to more data but I might be missing nuances in landing page performance. Then again, is there a logic why landing pages would perform differently across the various variables I'm split-testing?

What's your view on this?

Cheers,

Dan

---

Thanks for this awesome tutorial, this really cleared things up for me.

I just wanted to confirm one thing though. What if I've gotten 0 conversions (or successes in that peakconversion calculator terminology), can I still calculate all this and decide which one of my 6 landers is a winner among those other losers, even though all of them have gotten no conversions at all yet? I just want to make sure I don't cut any lander prematurely because they've not gotten any conversions but at the same time not keep on running a lander which hasn't gotten a conversion yet and most probably won't in the future too.

BTW http://abtester.com/calculator/ isn't opening up for me. Probably it's down.

---

Oh boy I'm so sorry for not having seen this earlier Dan! Don't know how I missed this. I'm assuming you must have figured out a way on how to approach this before now.

What you've illustrated is a common problem faced by anyone who's trying to optimize a campaign. In this case I would probably lump all the numbers together like you described in a). Is this is the most accurate way? Maybe not. But the question to ask when faced with this type of scenario, is "how accurate do I NEED to be without risking sacrificing a lot of profits?" and also "what kind of price will I need to pay for the additional accuracy?" If you have a large-scale campaign that you're making 4-figures/day in profits from, and all other aspects have been optimized, then I'd say do more-refined testing because 1)you'd have a ton to traffic to use for testing, 2)you'd be reinvesting a small portion of profits back into testing, and 3)any additional increase in ROI can translate into an extra 3-figures/day. But if you're still in the initial testing stage then I'd say focus on optimizing the big factors that have the biggest impact on conversion rates first before considering minor tweaks.

I'm facing similar problems in my own camps at this time - so thanks for the discussion! And once again I apologize for the late response.


Amy

---

The short answer is no - you cannot compare 2 landers that have made 0 conversions and make a cut based on that.

It would be like sending two workers out to do a job, and when absolutely nothing gets done, you try to judge which one of those workers is the more incompetent - you simply can't.

There are lots of possible reasons why you're not getting conversions. It may be your landers, it may be the traffic, or it may be the offer, or even tracking issues. The best way I know of to circumvent this "catch 22" is to start by testing 2-3 offers (in your chosen vertical and geo) that are "hot" according to AMs, use a traffic source you know converts at least decent for that type of offer, and then do a lot of spying and rip a bunch of different landers and test those to establish a benchmark. If you use an unproven offer on an arbitrary traffic source and landers you have no idea will perform well or not - when the campaign performs badly there's no way you can know which is the cause. (I'm not saying this is what you did - I'm using "you" in a general sense. )

I would suggest that you test more landers to see if you'll get some conversions. Either that, or do what I suggested in the last paragraph - test a few more offers first, and ask your AM for traffic sources that they know will convert for that type of offer, and go from there.

Interestingly enough, the split-test calculator WILL tell you to cut a lander even if neither lander has made conversions, if the difference in impressions is big enough:



Just don't let that fool you into thinking that comparing landers based on 0 conversions is appropriate.


Amy

---

Thanks a lot for explaining things in such detail Amy!

---

Question. Can we use this method to cut ISPs, browsers, devices and so on?

---

This method with the split-test calculator will only tell you which one of a bunch of things is best (landers or banners typically). It won't tell you which items are likely to be profitable - which is probably what you need to know when cutting ISPs etc.

To gauge profitability please use the calculator spreadsheet from this thread:

http://stmforum.com/forum/showthread...Banners-Part-2


Amy

---

Thanks Man! AM were very dark but with time it's getting much more lighter by the way @vortex I failed on this subject too in my university ;D
Thanks for this first part, now I'll jump for the second

---

Thanks for taking the time and efforts to put together this guide.

I was wondering if this solution could work when testing multiple offers?

Example
Lander 1 - 9 conversions but Average Payout higher
Lander 2 - 10 conversions but Average Payout lower

How that is going to count if not taken into consideration?
Thanks

---

Great question! For offers that have approximately the same payouts, I usually just assume they are equal. When payouts are really different, I would use the revenue as "successes" instead of conversions, and wait until the "probability of being best" reaches a higher value such as 95% to hopefully compensate for any inaccuracies in my method. I don't know how good this offer evaluation approach is, but for better or worse, have been doing it this way for a while now.

If anyone of you know of a better way to analyse offers with different payouts, please do let me know!


Amy

P.S. Why would landers 1 and 2 be testing different offers? If testing landers, each lander's exposure to each offer should be the same. If testing offers, each offer's exposure to each lander should be the same. Otherwise you would be comparing apples to oranges.

---

Very nice article. Its important to have your own set of rules and to be very consistent with them.
You have to apply the rules all across the board and let the numbers decide whether to cut or push a placement or to keep it stable.

My rules are, before making any decision on Push vs Cut vs Maintain:
- Run as broad as you can, on every placement you can get ahold of
- Banners: minimum 100 clicks --> ROI- cut, ROI+ push or maintain depending on what your ROI% treshold is
- Pops: minimum 10,000 impressions --> ROI- cut, ROI+ push or maintain depending on what your ROI% treshold is

Also remember that its better to make 10% on a 10k$ placement than to make 50% on a 1K$ placement so there is definitely a trade-off between volume and ROI%. This will determine your decision to Push vs Maintain.

---

I agree that having rules and sticking to them would be better than making arbitrary decisions - but only if those rules will allow you to be right in your "predictions" more than 50% of the time (i.e. better than making decisions by tossing a coin).

Rules of thumb can be helpful, but they need to be changed to adapt to different situations. Applying the same set of rules to different bids and offer payouts may not be the best way to go. But that's just a personal opinion.


Amy

---

Hi Amy

I have the stats for all the week on Voluum 24,162 visits and 137 conversions and when i input this into peakconversion i get: You have a row with more successes than trials! Clearly i'm doing something wrong

---

Did you switch the 2 values when entering the values into the calculator? The 137 conversions should be entered into the "Successes" field.

Please upload a screenshot if you're still experiencing problems.


Amy

---

Hi Amy

Just a question out of this thread

http://prntscr.com/97pmzl

Cr and Cv confuses me in Voluum. What is the difference between the two?


Thank You

---

In Voluum i can see all the Landing pages and the CTR/CR/ROI etc... Isnt this enough to see which one perform better?

---

CV = Conversions / Visits
CR = Conversions / Clicks



Amy

---

Actually came across this yesterday. Thanks anyway

---

So impressive Amy. To be honest, I'm really afraid of numbers, but this game is all about numbers and feel like I learn more from this thread than the whole stats course at my uni . Gotta apply it to my campaign now.

---

Haha thanks Tien! It's interesting that you'd bring up the stats course at your uni. I actually failed Stats 101 and had to retake the course. I learned everything I know about stats from Caurmen's threads when I was a newbie. My contribution is taking the same concepts and methods and breaking them down even further - as there are so many more newbie members in the forums now compared to when caurmen originally wrote those stats threads.

Glad you enjoyed it, and best of luck with your camps!



Amy

---

Hello Amy,

Do you use any tool other than https://www.peakconversion.com/2012/...al-calculator/

Looks like the website is down.
Looking to Cut offers and then landers.

Regards,
gaurangv

---

It's back up now.

For cutting offers with different payouts, use this method instead:

https://stmforum.com/forum/showthrea...211#post289211




Amy

---

It is down again :-( Is there any offline or self-hosted version of this tool?

Edit: I just found a webarchive version of the page https://web.archive.org/web/20190404...al-calculator/

---

Thank you Amy for this little gem! I've got a noob question, when cutting creatives, should I still use impressions when running a push campaign on CPC. My instinct tells me that it should be the number of clicks but I'd rather ask a stupid question than get burned.

---

With push traffic, the CTR can be all over the board, depending on the rotation, age of the subscriber etc... Personally, I don't cut push ads based on their CTR as per default. However, if the ad CTR is to low, the algo of the network will stop server such ad on it's own, so it will simply stop receiving traffic or it will be very low. In such case, it's pointless to try and run such ad as it won't get almost any traffic anyways. So I could say, the algo will optimize based on CTR itself instead of you.

Obviously, the low CTR will tell you something about the AD, so it's still a valuable piece of info. Low CTR ads are not a good match for your current targeting so you need to change them in most cases.

---

Thank you for your insight matuloo. I think I failed to express my question. I'm not trying to optimize on CTR. I'm wondering if it makes more sense when using this calculator https://peakconversion.com/2012/02/a...al-calculator/ to input the number of clicks a creative received instead of the number of impressions, since that's what I'm really paying for.

---

Ah gotcha Yup, you should use clicks in case you're using the CPC model.

One more thing though, the CTR will still play some role as a low CTR ad will not get any traffic. So in case of low CTR ads that are pretty much ignored by the algo, it's pointless to use a calculator to determine if they are "winners". To put it simple, work only with ads that stand a chance in the bidding battle.

---

Perfect. This is clear now thank you

---

Home > General > Affiliate Marketing Forum