Trying to unravel the Gordian Knot! (14)
Hey everyone - so after a several month hiatus post the 6WAMC, my wife/partner and I have been heads down trying to get to green. We're kind of at the point where we have some data, but don't know what comes next. Advice appreciated - here's the sitch:
1) Our AM recommended a series of sweeps offers - 6 of them.
2) We ran those 6 offers RON on ZP with a variety of angles, and settled on 2 angles/offer approaches.
3) The first “offer” is actually a multi-offer angle
4) The second offer was one of the 6 that seemed to do well in the initial round of testing
5) After running for awhile with no conclusive results, we also started creating landers for one of the offers that our AM insisted was doing well.
6) We’ve now made about 50 landers, with one of the multi-offer ones showing promise at a 14.51% CTR and about a .56% CR.
7) We have not optimized placements yet other than to blacklist clear losers, allowing ourselves to vary only landers.
We are asking ourselves:
- Do we have enough data to pick a lander? We've been using the Kill/Whitelist calculator but nothing flashes "whitelist" yet.
- How do we know if we picked the right offers to focus on given that we we were also testing landers to start?
- We have made about 50 landers. We let each run until we have 80% confidence that it will NOT have a 30% ROI before we kill it - roughly 1500 impressions at a $3.00 CPM with no conversions on a $2.40 payout. Is this the right way to test landers?
- More specifically - given the ROI performance of our top landers so far - should we just start focusing on one of them?
- How long should we keep making landers for our three “chosen” offers before we decide it’s not our ability to produce landers, but the offers that are bad?
- In particular, Ruby Tunes advises spending about 10-15x payout on each lander before killing it - this is way more than we are spending and also we don't know if the offers are good.
- We’ve spent about $600 so far and made back about $200. How much do we spend on this round of offers before looking at new ones?
Here's what our our top landers look like...
Originally Posted by fernando
But what seems crazy to me is the sheer amount of landers you're running. Are they 50 unique landers? Or is it 50 slight modifications to 1?
I think you'd be better off running 3 or 4 completely unique landers and then optimizing the winner to profit.
@ssmarketers - i think that's the direction we're heading, e.g stick withthe offers and start optimizing the best lander.
You guys have been hard at work! Nice!
I have a few questions...
- Firstly, am I right in assuming this is just in one geo?
- Secondly, what have your landing pages been like? Many totally different variations? Only a couple of total variations with a lot of slight changes?
- Thirdly, I'm slightly confused by the offers you're testing.... You tested six, then focused in on two of those, then focussed in on a third later?
If I'm reading correctly, it seems like there are quite a few variables within your tests, with offers switching in and out...?
Ideally, you want to eliminate as many variables as possible. And this ties directly into your question of "how do we know if we picked the right offers to focus on given that we we were also testing landers to start?" Because I'm guessing you now you have many different results on different landers that featured different offers..?
Once you have eliminated variables, it's a lot easier to have benchmarks to compare to. E.g. you run your best lander to a few different offers, you'll be able to compare these fairly and from there have one less variable.
Regarding my suggestion to spend 10-15x the payout on a lander - this is definitely not to be taken as a strict rule. It's more of a general rule of thumb that I use in my own campaigns. Ideally, you'll want to test to the point of statistical significance, but of course this is not always practical and not always an economically beneficial rule to follow. There was recently a very good in depth discussion about rules of thumb, significance of payouts, and statistical significance HERE.
Another possible approach is instead of being stuck here trying to optimise forever, take your winning landing pages and offer types and test a bunch of different geos, or even sources!
Hey there bud! Good idea starting a follow-along!
Some answers to your questions....
- Do we have enough data to pick a lander? We've been using the Kill/Whitelist calculator but nothing flashes "whitelist" yet.
That's not what you should be using that calculator for! Use this one instead. Cut any lander that falls to <10% "probability of being best" when compared with the "current-best" lander until you're left with landers you feel are promising, then keep improving on those. See this post on how to use that split-test calculator.
- How do we know if we picked the right offers to focus on given that we we were also testing landers to start?
If you're only testing a few landers, and each offer is getting equal exposure to each lander, then the data won't be skewed by much. However, if you're testing 50 landers, you may want to test offers and landers separately (this is what I do). Initially you could throw all your offers and landers into the same camp for 1-2 days, then 1)pick the most promising offer and start a "lander testing campaign" to test landers, and 2)pick the most promising lander and start an "offer testing campaign" to test offers.
- We have made about 50 landers. We let each run until we have 80% confidence that it will NOT have a 30% ROI before we kill it - roughly 1500 impressions at a $3.00 CPM with no conversions on a $2.40 payout. Is this the right way to test landers?
Nonono....again, use the split-test calculator instead! Your goal here is to keep improving your landers, not cut landers that have negative ROI. Don't fixate on ROI right now as long as your offer is good enough such that you're not wasting excessive money on lander testing by using a bad offer. Judging by the ROIs I'm seeing for your landers, I think your offer may already be good enough.
- More specifically - given the ROI performance of our top landers so far - should we just start focusing on one of them?
Again, use the split-test calculator to cut the lander types/designs that are the worst. Then make different versions of the promising landers to keep improving on them.
- How long should we keep making landers for our three “chosen” offers before we decide it’s not our ability to produce landers, but the offers that are bad?
I don't think anybody will be able to answer a question like that. We just have to test different things - as long as the testing is always improving at least one of the factors that affect ROI (landers/offers/banners/traffic quality/etc.) then we're headed in the right direction. If you have the budget though, you could consider setting up separate campaigns to test landers and offers separately, and have these camps set up for multiple geos. It really depends on whether you want to focus on improving one factor at a time or several (budget is a consideration here as well as the amount of time you have).
- In particular, Ruby Tunes advises spending about 10-15x payout on each lander before killing it - this is way more than we are spending and also we don't know if the offers are good.
Rules of thumb can only act as rough estimates. Also, everyone will have their own personal preferences. Another thing to consider is how much effort you've put into a lander. Again, I want to mention the split-test calculator. If the lander is a completely original one that I've spent hours/days to create, then I would let it run to 0% probability of being best (compared to current best lander) before I would cut it, because I'd want to LITERALLY be 100% sure it's a losing lander. However, if you're testing 10 landers that are the same except for the headline, then 10% probability of being best should be enough to cut.
- We’ve spent about $600 so far and made back about $200. How much do we spend on this round of offers before looking at new ones?
Hard to say. I'm seeing a couple of your landers at -2x% ROI but they only have like 1 conversion each. I'd keep collecting data for a while to see if those will hold. If so, I'd say the offer is decent. Of course you can always test new offers if you like! At any rate, start using the split-test calculator to cut your worst landers and your numbers will improve for sure. But you have to be patient because you have 50 landers running - make sure to wait until statistical significance before cutting a lander (i.e. <10% probability of being best compared to current-best lander).
Looking forward to your next update!
Amy
As you may recall from your Hellenistic Greek History studies, the ultimate solution to unraveling the Gordian knot was not to unravel it at all, but to cut right through it with a sharp blade and move on.
There are theoretically an infinite number of variables and landing pages you can test. If you test ALL of them under ALL possible conditions past and present, then yes, you will find THE best performing lander EVER.
Unfortunately you will be completely broke and penniless well before you get there.
The goal in testing in conversion economics is NOT to find THE best lander EVER, but rather to find profitable placements as quickly as possible.
So follow the lesson of Ἀλέξανδρος ὁ Μέγας and start cutting.
Focus your energies on the placements that show some signs of working better, and then create variations of those placements while pushing yourself to understand what may be preventing these placements from performing better.
Oh boy, lots to respond to! Thanks everyone for the detailed advice. I'll try to take these in
@ Ruby Tunes:
- Yes, just one GEO
- 25 totally different variations with one or two versions each.
- Yeah, we kept on too many offers :-).
@ Vortex - some major learnings in here! As usual you are generous with your advice.
- Do we have enough data to pick a lander? We've been using the Kill/Whitelist calculator but nothing flashes "whitelist" yet.
That's not what you should be using that calculator for! Use this one instead. Cut any lander that falls to <10% "probability of being best" when compared with the "current-best" lander until you're left with landers you feel are promising, then keep improving on those. See this post on how to use that split-test calculator.
>>This is a way better approach - thanks! I think that this single change will matter the most to how we move forward, along with picking a single offer.
- How do we know if we picked the right offers to focus on given that we we were also testing landers to start?
If you're only testing a few landers, and each offer is getting equal exposure to each lander, then the data won't be skewed by much. However, if you're testing 50 landers, you may want to test offers and landers separately (this is what I do). Initially you could throw all your offers and landers into the same camp for 1-2 days, then 1)pick the most promising offer and start a "lander testing campaign" to test landers, and 2)pick the most promising lander and start an "offer testing campaign" to test offers.
>> If nothing else I guess we now have enough data to properly set up these campaigns.
- We have made about 50 landers. We let each run until we have 80% confidence that it will NOT have a 30% ROI before we kill it - roughly 1500 impressions at a $3.00 CPM with no conversions on a $2.40 payout. Is this the right way to test landers?
Nonono....again, use the split-test calculator instead! Your goal here is to keep improving your landers, not cut landers that have negative ROI. Don't fixate on ROI right now as long as your offer is good enough such that you're not wasting excessive money on lander testing by using a bad offer. Judging by the ROIs I'm seeing for your landers, I think your offer may already be good enough.
>>"Your goal is to improve landers, not cut landers that have negative ROI." Let's please post that somewhere prominent!
- More specifically - given the ROI performance of our top landers so far - should we just start focusing on one of them?
Again, use the split-test calculator to cut the lander types/designs that are the worst. Then make different versions of the promising landers to keep improving on them.
@ cmdeal -
Yeah, we haven't even addressed placements yet - honestly we were trying to hold that variable fixed while we looked at landers. We'll go there next!
I'll post an update on this when we've re-structured things and gone through the next round of data!
Also @cmdeal I didn't know Alexander was known for his optimization skills :-)
Originally Posted by cmdeal
Originally Posted by fernando
Hi all - ok, we have some updates.
When we posted on June 10 our daily stats looked like this:
Spend: $37.72 | Revenue: $0| Net profit/loss: -$37.72 (-100% ROI)
As of yesterday, our stats look like this:
Spend: $27.135 | Revenue: $24.00 | Net profit/loss: -$3.35 (-12.26% ROI)
So we are moving in the right direction! Here's some of what has happened, and, of course some additional questions.
Based on Vortex’s advice to keep improving on our existing landers rather than killing landers based on arbitrary ROI criteria, we created 10 or so versions of our winning lander. Given that we have so many versions we killed the ones that had a less than 20% chance of beating the leader, rather than a less than 10% chance. That worked really well, with about 5 of them showing green for the first day or so. Most of them seem to lose profitability over time. Here is a screenshot of our top landers over the past 30 days.
So for the past several days we have been optimizing the winning landers. We are trying to get to the point where we have one that has been consistently profitable so that we can scale.
The other thing that happened is that the average bid in our market seems to have increased, so we have had to increase our bid to keep getting traffic, which has lowered our profitability.
Also, we discovered that Explorer and, to a lesser extent, Chrome, drive the bulk of our conversions, so we cut Safari and Firefox.
Here are some questions:
1) We keep introducing variations on landers because our "winners" have a roughly equal chance of being the winner - or within 10% of each other in the Baysian calculator. Is the thing to do to keep introducing variations, or wait and buy more data on the same landers?
2) Our RON campaign is 6% profitable overall on Explorer alone (and red on all other browsers). The data is cumulatively statistically significant, however several of our current batch of landers are profitable on Chrome, as well (though not all). In other words, there is a possibility that our early shitty landers worked exclusively on Explorer, but our better new ones have a shot on Chrome as well. Should we create a whitelist campaign for Internet Explorer RON? Here is a screenshot of the performance of the various browsers on the campaign over the past 30 days.
3) As mentioned before, we haven't done any placements optimization. We are finding it takes a long time to get enough data on placements to know whether to whitelist or kill. We're thinking about converting our RON campaign to a Targeted campaign that just targets the top 100 placements in our market in order to get data more quickly. Is there a better way to do this?
Thoughts appreciated...you guys really helped us break through last week!
Wow, that's a pretty big improvement. Well done!
Keep cutting.
The next best thing to do would be to scale using your best 2-3 landers instead of retesting all of them. Therefore there's no need to cut down to the last best lander - because you'll probably want to scale using your top few landers anyways.If it's your decision to target just Explorer - you could either create a new campaign or just change your current campaign's settings to exclude other browsers.
If you want to target both chrome and explorer and want to optimize landers for each browser separately, you can set up a new one so you'd have 2 camps each targeting a single browser. Alternatively, you can use a single campaign, and just specify
Another thing worth looking into would be WHY a lander is performing badly on certain browsers. May be a worthwhile exercise to find out how the lander looks/works when you browse to them using those browsers - you may find that certain functionalities may be lost, or the lander doesn't display correctly, for example - and it may take a minor tweak to improve the conversion rates for those browsers so you can monetize them as well. (Of course, this is assuming the traffic you get from them is significant enough to be worthy of the detective work.)
Although, in the testing stage, you'll probably want to use RON to maximize the traffic you get. Wouldn't hurt to set up a TARGET camp and assign it the same bid and see what the difference is in traffic levels.I'm just blown away by how much action you two are taking and the progress you've made in such a short time. Should start to see green real soon!
Amy
Originally Posted by vortex
The next best thing to do would be to scale using your best 2-3 landers instead of retesting all of them. Therefore there's no need to cut down to the last best lander - because you'll probably want to scale using your top few landers anyways.That would really depend on your strategy. You can choose to maximize your traffic by targeting all browsers - in which case you'll want to cut landers based on total data across all browsers. If you want to just target certain browsers, this may help you see green faster but you will get less traffic when you scale. So you pick your battle.
If it's your decision to target just Explorer - you could either create a new campaign or just change your current campaign's settings to exclude other browsers.
If you want to target both chrome and explorer and want to optimize landers for each browser separately, you can set up a new one so you'd have 2 camps each targeting a single browser. Alternatively, you can use a single campaign, and just specify
Another thing worth looking into would be WHY a lander is performing badly on certain browsers. May be a worthwhile exercise to find out how the lander looks/works when you browse to them using those browsers - you may find that certain functionalities may be lost, or the lander doesn't display correctly, for example - and it may take a minor tweak to improve the conversion rates for those browsers so you can monetize them as well. (Of course, this is assuming the traffic you get from them is significant enough to be worthy of the detective work.)
No other way to know except by testing and this would definitely be worth a try.
Although, in the testing stage, you'll probably want to use RON to maximize the traffic you get. Wouldn't hurt to set up a TARGET camp and assign it the same bid and see what the difference is in traffic levels.I'm just blown away by how much action you two are taking and the progress you've made in such a short time. Should start to see green real soon!
Amy
We're definitely trying to get our landers successful and winnow down to the best ones. If anything, I'd say we are introducing too many variations - we're testing anywhere from 15 to 20 at any time. But given that none are clearly winners (as in strongly profitable over multiple days) I guess that creating lander variations is the main variable we have to work with.
We're back to about -50% ROI today, most of our stuff seems to be reverting to the (negative) mean. Also I do get the sense that this trade is getting a bit crowded and prices are drifting up for the market and the traffic. Will see!
Will post when something interesting happens...
Home > POP / PPV / Redirect > Follow-along Campaigns