Are day-by-day fluctuations screwing up your tests? How to avoid it? (6)
Are day-by-day fluctuations screwing up your tests? How to avoid it?
This is another follow up post, based on this awesome thread started by Amy (Vortex) : https://stmforum.com/forum/showthrea...Avoid-(Part-1)
I will talk about mistake #3 today : Either Testing Too Extensively, or Not Extensively Enough … and particularly about the later one : not testing enough.
We all know that in order to make qualified decision, we need to base it on statistically significant data. A lot has been posted about significance on STM already, so I'm not gonna get into details again now. In case you're not familiar with the term, check this guide by Caurmen for example : https://stmforum.com/forum/showthrea...nificance-quot
One might think, let's buy a bunch of data, let the campaign run for some time, until it reaches statistical significance … and that's it. However, there is one more factor associated with testing that can royally screw up your data analysis – part of the days and days of the week, or let's say : WHEN you are running the tests.
I've been analyzing the data on a bunch of my stable campaigns and I noticed that quite a lot of them had a few bad days recently. If it wasn't for the fact, that these were long running campaigns, I'd start messing with the setup immediately. But since they were all running for a while, I left them alone and blamed the bad days on outside factors that I cannot influence.
This made me realize, if a new affiliate was looking at the data, they would probably freak out and start optimizing, so I decided to write this article to give you some food for thought.
YOU NEED TO LOOK AT THE BIGGER PICTURE!
Performance of campaigns can fluctuate from one day to the next. A losing campaign, can start making profits the next day. This is quite normal and the differences can be quite big. Certainly not from -70% to +70% but a 20%-40% difference is rather normal. So having a campaign at -20% ROI one day, then moving to +20% on the next one is a standard situation that you will face from time to time.
Now, imagine you've been testing something on the day with -20% results … chances are, your actions would be completely different than they would be on the +20% day, agree? That' why you need to optimize your campaigns by looking at a longer time frame, you simply have to figure out how large the difference between a GOOD and a BAD day will be.
Let me show you a screenshot, it's from one of my live campaigns, covering a bit more than last 48 hours. Take a look at how poor the performance looked yesterday and how much it improved today. Let me add that I did NOT touch anything, that's simply a fluctuation that happened without any action on my end.
Within 2 days, the performance went from a slightly negative numbers to about 40% ROI … I know this is rather big change, but that's why I picked this particular campaign, so it's very visible and easy to spot.
Now imagine that this wasn't an already running campaign, but I was testing a new offer or whatever. Based on yesterdays data, I would probably cut a few banners or LP's or offer variations, because they would be in deep red numbers. We are talking about couple 1000s of banner clicks here, so you couldn't say I didn't wait for significance either. But still, my decisions would be wrong so to speak, I'd most likely kill a highly profitable campaign.
SO HOW TO HANDLE THIS?
The solution is pretty easy, do not try to optimize a campaign within a couple hours or a day. Try to spread it out to run evenly for a couple of days, if you can include both week days and the weekend … that's the best case. In case you budget doesn't allow you to do this, limit your targeting in some way in order to decrease the available volume. Target a smaller GEO, or one OS, or a limited amount of placements. Do whatever is needed to run 24/7 for as long time as possible.
Once you have an idea about how big the fluctuations are, you can use this information in your optimization process. It's very important to understand that these fluctuations are normal and that there is nothing you can do to avoid it, you just need to learn how to work them into your plan.
WHY DOES IT EVEN HAPPEN?
There are several reasons for these fluctuations, let me mention the most frequent ones :
- Offer Type : some offers perform better in the evening or on weekends – a good example could be adult dating. You can be profitable 24/7 with dating, but some hours or days will simply work better. If you try to optimize such campaign based on morning data, you can easily cut a good offer for example.
- Technical Problems : there might be a temporary problem with the offer you are promoting. It only takes a few hours of downtime and the data will look horrible.
- Scrubbing : we all know that some advertisers scrub, whether they have legit reasons for it or not … try to optimize an offer during the “scrubbing” period and the results will be very poor.
- Seasonal Reasons : there are certain periods that affect conversions, some are very obvious, for example Christmas, Easter, Summer Holiday season … but there also less obvious ones that you might not know about : big sport event, local election, natural disaster … These can be responsible for short conversion problems.
- Competition levels : not every day is the same at the traffic sources either, sometimes the competition is bigger, sometimes it's lower… this will directly impact your traffic quality and volumes and it will obviously affect your conversions too.
ALL CLEAR?
In order to prevent bad decisions in your optimization process, make sure you collect data across several days or as long period as possible. You need to identify the fluctuations and understand how big they can be. There are many things that you cannot control, which have direct effect on your campaigns.
As you could see, the differences can be really big, keep that in mind when tempted to make important decisions based on a few hours of data and a bunch of conversions!
Thanks for reading 
Please post questions if you have any.
Great post once again Matuloo!
I have also noticed that my campaigns are on the red line 4 days out of 7, based on a 3 week data analysis... I'm struggling with it since I tried a lot optimizing it without success. But I may be just doing something wrong somewhere, I just couldn't figure it out yet. My campaign have been profitable the first week, I added bid on the good placements and cut the bad ones based on 1 week data(placement with ctr lower than 3% and with more than 80 visits), but I still couldn't get profitable on those bad days. So I'm just running my campaigns 3 days a week now to get some profits out of it. Can my campaign just be dead? Shall I just start a new campaign with the same assets in order to get a new rotation and start optimizing from scratch? Or shall I keep running it on profitable days only ?
Originally Posted by foussama
some days are worse than others indeed, but 4 out of 7 is too much ... the negative fluctuations should not be so severe. This means that there is some flaw in your funnels, you need to improve something.
The fact that you were profitable in the first week and then the problems started, shows us that your creatives simply need improvement. You need a better lure so to speak, check this article of mine for better understanding of what I have in mind : http://stmforum.com/forum/showthread...-down-and-down Your banners or LPs are most likely good, but they still need a small improvement in order to attract more fishes.
Hi,thank you, this is a great help to me.
+ 1 more thing to take into account: CONVERSION DELAYS.
There was 1 offer I run last year where the conversions came back 24 hours later, in a batch and timeframe of about 30 minutes. So I was easily -xxx to -xxxx and suddenly became +xxx to +xxxx just because of this fact. If I wouldn't know there was a such delay (because of testing) I would probably think it's just another bad offer.
Those delays are also geo and advertiser specific, so it's always wise to take it into account - give it at least from couple of minutes to couple of hours.
Originally Posted by erikgyepes
I also see some delays in case of offers with DOI flows, but that's only a minor % so it shouldn't have dramatic effect on the overall ROI, but still, it can make some changes to the final result.
Home > >