Split tested offers from 2 networks, big difference (5)
I've always believed about split testing from networks from the start, but nothing beats seeing it in action in real life.
Identical offers, different payouts, different networks. Both pretty big and known.
Few questions:
1. Does that look like a significant sample size? (Used the AB split calcultor from peak conversion, looks pretty significant, or is there any other methods of measuring when it comes to this?
2. If it is, is there anything I can do or talk about with the bottom network? Or is this the case where the network doesn't even have control?
This happens all the time, but the bigger the sample size, the smaller the differences tend to be. I would suggest re-running the test on another day again.
Different networks perform diferently for me on random days and I cant really find any pattern in it.
You can see a network work worse at the end of the month for example, during certain day of the week... there might also be a server problem... countless reasons to make a network perform worse for a limited period of time.
It is true tho, that after the sample gets big enough and the data is acumulated over a span of multiple days, one of the networks usually does outperform the other one.
If you have relationships with both networks I'd ping the bottom network and tell them your findings. See what they say. There maybe a tracking issue. This happens with us from time to time. Essentially something was potentially setup wrong.
No harm in letting the bottom network know about this IMO.
I would check to see if the conversion flow is really identical, especially as there is a difference in payouts.
There can be small differences in the flow that maybe you arent aware of.
And yes, different days, different hours, different servers. Offers dont always act and convert the same in different networks. This is (one) of the reasons people tell you to split test.
You should also see what the overall revenue is for you after a longer period. Different networks have different relationships with the advertisers and you should see the overall stats when a month ends in order to get a complete picture of things.
I agree it’d help to clarify things by speaking with the network rep of the offer with the lower CR. Sometimes, more transparent communications with your rep can work wonders.
Also, to add on to what has already been mentioned. It would help to split test for a longer period of time, and also to check if either of the networks end up scrubbing during billing. I once heard from my publisher about a split test that he conducted across 3 networks. Unfortunately, the network with the highest CR and payout ended up scrubbing so many leads, that he ended up with a negative ROI. The CR and payout with glispa and the second network are comparable so in the end, he just ran the offer with both of us.
End of the day, split tests and communicating with your network rep are very important.
Home > Paid Traffic Sources > Mobile