Home > General > Affiliate Marketing Forum

---

Weird Logo Splittest Results (14)

---

I've run a splittest testing 2 identical logos on my website. The only difference was the filesize and image quality.

Why is this test weird? Because the difference is barely noticable but one image outperformed the other by 250%!

I tried to recreate the quality different of the logos with some custom images. Here they are:

Image#1:
Original Filesize: 996 Bytes


Image#2:
Original Filesize: 1.35KB



You can see that image #1 has a lower filesize to improve loading speed and takes a slight quality penalty for that.

Now, here are the (weird) results of the test after an even $600 adspend:
Page with Image#1: 1108 LP clicks (1.22% LP CTR), 25 conversions (2.26% CR), $500 revenue (59.70% ROI)
Page with Image#2: 1088 LP clicks (1.21% LP CTR), 36 conversions (3.58% CR), $780 revenue (150% ROI)

Image#2 has a 93% chance of being the best, so it is statistically significant.

This means Image#2, the one with the higher file size and barely noticable quality improvement outperformed Image#1 by 250%!

I usually optimize all my images to make them smaller. But after this test, I wonder how much money I left on the table.

Did the slightly better image quality unconsciously influence the visitors? Or is there another explanation?

Share your thoughts on this.

---

Interesting study, however I would say much more data is needed to say anything for sure. There are just too many external factors at play here - possible server latency, times of day, etc.

That being said, I do recall reading something like this before. An effect similar to the 'loading' trick some affiliates use to increase engagement time. Something subconsciously triggering a blip in the 'day-to-day' browsing of an internet user. Slow loading. What's this about? Must stick around to see if it's high quality.

All that being said, I feel like the data is just too finite to say anything of significance here. The file sizes are just too similar and the data isn't strong enough after only 50 conversions.


When you get to 500.. 1,000, then you might have something Appreciate the share regardless, we need more like this around STM, props!

---

Desktop or mobile?

If mobile, wifi or carrier?

Load time isn't as important for desktop.

---

Are both of them using the same file format? That can play a huge role for mobile, ISP's will treat them differently when they get "minified" or compressed. STM had an interesting case study about this topic before.

---

The bulk of conversions came from desktop.

Yes, both were gifs.

---

I would say keep optimizing them. I also think its some other factor.

---

Also what bobliu said is very true. Many external factors play into it and the sample is very small. If we have three or four times the sample and the relative gap remains, then it becomes interesting looking for which external factors might influence this.

---

The conversion amount is not that large indeed but the conversion path is more complicated, giving each conversion more weight.

Furthermore, the results are pretty much statistically significant.

I will test boblius theory next with this page and delay the logo with some javascript to further investigate.

---

What external factors might affect a splittest in a "controled environment" where traffic is split 50/50 between two pages on the same server/domain?

---

That also means that 7/100 you could be wrong although, it does make sense since desktop traffic are usually Wi-Fi with fast internet speed and having a high quality logo/images can convince them that your offer is legit.

Versus mobile where most people are on 3G/4G and will close your site before it finished loading if it uses too much data.

---

I would say this is just coincidence and too small data sample since the quality or size difference isnt even notable. Even if you take for example 2 completely identical banners, one of them will have higher CTR and one will convert better. The more traffic you run, the less significant the difference will be - this happened to me just a couple days ago when I uploaded the same banner twice by accident.

You can split the traffic 50/50 but there is no guarantee that the split was even on a website level or creative level, because you can only split the actual clicks, not the rotation. This is what makes optimization harder sometimes and one of the factors why we can see different results from day to day.

On a side note, I'm not a big fan of too much optimization, I always choose higher quality over a few bytes saved.

---

I've encountered a similar effect in the past, testing 24-bit PNGs against JPGs. The PNGs outperformed the JPGs - on mobile - even given the much larger file size.

OK, thoughts:

1) The visual improvement between the two is noticable, so I'm not surprised there was some effect.
2) The results do look somewhat statistically significant to me. However, they also look very high for a minor change - that's a 50% lift in CVR.
3) The file size change will have resulted in no loading time change at all on desktop, unless they're on dial-up internet (and not really even then).
4) Given that, I'd suggest the next step is probably a vigorous attempt at falsifying the data. Was the traffic definitely identical in all ways? How did you run the split test - were they seperate pages? If so is there any way one of them could have had different content or loaded faster or slower? Etc.
5) I'd also recommend doing another split-test to see if the results remain, preferably on a different page if you can.

I'd be very interested to see the page you were testing, in order to see the logo in context.

Great case study and interesting result!

---

Try running it a bit longer , when having fascinating results like this in most cases it's not enough data yet.

It's a 93% chance it's the best - the best means it can be just 1% better than the other.

http://www.peakconversion.com/2012/0...al-calculator/ is great because it also gives you a potential conversion rate range.


Here an example from a lander split-test I did a few days ago:

1. First a short time range of one day , the tool says it's 92% chance of being the best , it seems clear lander A is much better:




2. And here the exact same lander split-test , just for a longer time range of 3 days with more data - suddenly they're pretty head on head ..




My guess would be that if you run longer , the winner still stays the winner but the huge performance gap in between the two landers will become much smaller.

Thanks for sharing your results by the way! Landing page split tests are always interesting

---

great insights guys, what a mastermind!

I tested to delay the logo but it actually hurt conversions. I'm not going to pursue this further and just going to assume that the (even if only slight) increase in image quality made the difference. for caurmen it was noticable at least so other people must have felt the same.

matuloo, after this test I feel like going for quality over filesize too. will definitely run a few splittests on this though.

kepe, thanks for your input but your AP is much lower than mine which makes me think that the conversion path is simpler. Thus giving each conversion less weight. In your case you would need more conversions than I had in my initial test to draw conclusive results.

You entered visits:conversions in the calculator whereas I entered clicks:conversions.

I'm glad I posted this case study on here as it gave me great insight with all your input!

Thank you all!

PS: as a side note I love how you propose to falsify the data caurmen. I'm a huge fan of Karl Popper myself and even as a German native speaker his work is really hard to digest

---

Home > General > Affiliate Marketing Forum