3 Identical Banners. VERY different filesizes. You'll never guess what happened next. (41)
3 Identical Banners. VERY different filesizes. You'll never guess what happened next.
One question that has been asked a number of times on STM is whether the filesize of a banner makes a difference, particularly on mobile. After all, it makes sense that a smaller banner would load faster and do better.
So, I set out a simple test to see whether that was true.
I was not expecting the results I got.
The Setup
I ran the test campaign on Decisive, targeting UK carrier traffic on apps, aiming to look at CTR rather than CVR because I was expecting a more direct relationship between clickthroughs and file size.
I chose carrier traffic because it made sense that it'd be more sensitive to filesize than wifi users, who are probably on broadband, and UK because I had a campaign there already which I could break some creatives out from for the test. I targeted apps because I expected them to be an environment where they were more of an interruption, and hence again would be more filesize-sensitive.
I ran three visually identical ads, which all looked like this:
Two of them were JPEGs, with filesizes of
- 5 kb (Small)
- 15 kb (Medium)
I couldn't get a JPEG to come out with a filesize of more than 15kb, so for the last banner I used a 24-bit PNG, with a whopping filesize (for 320x50) of
- 40 kb (large)
I ran the campaign for 42,000 impressions, ending up with a statistically significant differences between two of the banners, and a less certain but still likely difference between all three, according to the usual calculator.
And the results were...
The smallest size JPEG (5kb) performed second best, getting 113 clicks in 13,923 impressions, for a CTR of 0.842%.
The medium sized JPEG (15kb) actually performed worst of all three banners, getting 99 clicks in 14,089 impressions for a CTR of 0.731%.
And in a spectacularly weird turnabout for the books, the winner of the three banners was the HUGE 24-bit PNG (40kb), which got 127 clicks in 14,124 impressions for a 0.934% CTR.
So what does this mean?
Well, we can probably conclude that at least in the UK (which doesn't have spectacularly fast mobile internet, certainly not as fast as Thailand, for example) and similar countries, this initial test appears to show a reasonably strong relationship between CTR and file size for JPEG banners.
The fact that the PNG outperformed both, however, is extremely weird. I checked statistical significance for these results, and the difference between the medium-sized JPEG and the PNG in particular is almost certainly statistically significant - 97% chance that the PNG will retain a higher CTR.
It would probably be worth doing more tests - which I may do in the future - to see if this result is repeatable, but initially, my guess is that the difference has to do with the rendering code needed to render a PNG and a JPG. Either the PNG renders much faster - a possibility - or it renders clearer than a JPEG on most phones.
Either way, it seems like it might be worth chucking a 24-bit PNG version of your best banner into a few of your mobile campaigns to see if you can outperform everything else!
Weird, huh? If you've got any questions, comments, or suggestions on WTF could be happening, post 'em below!
UPDATE: Second Test
As promised, I ran another, more rigorous test on this.
Firstly, I checked for 90% bot-free placements using this technique. Having found one (Grindr), I produced two identical banners, one in JPG at the usual quality I'd use for desktop and mobile banners, and one in PNG.
I then ran both of them on that single placement for approximately 100 clicks. The JPG gave an overall CTR of 0.275%, whilst the PNG gave an overall CTR of 0.309%.
Running that through a Bayesian A/B test calculator, we get a 79% chance that the PNG will be the strongest performer.
That's usually not enough to conclude one way or the other, but given that shows the same anomalous increase in CTR as the earlier test I ran, the two tests together tend to imply that we're seeing some real-world results, albeit not incredibly strong ones.
In light of that, I feel reasonably confident in saying that it seems that if you're running mobile banners, you are likely to see a small increase in CTR by running PNG rather than JPG banners.
Feel free to reproduce this test yourself - I'd be very interested to see any other results!
Further tests? I might run a test on wifi only - if the theory that this a result of how phones render PNGs and JPGs, I'd expect the effect to be stronger on wifi because loading times should be closer to identical. It would also be interesting to segment this effect by phone, and see if one handset in particular, or one family of handsets, shows this effect more or less than others.
Cool test. I'd be curious to see if others can replicate the same results.
+1 also for a very CTR friendly post title 
I wonder if the larger file size caused a bit of a loading delay, and that since it was the last thing to show up, maybe it caught attention a little better than ads that loaded at the same time as the rest of the page? Not sure - pretty strange.
Thanks for the test!
Did you run these on the same frequency cap, on the same placements, and at the same time?
Don't understand how bigger banner size can have a better ctr... was it higher quality image?
mmm, you might saw a random effect ..
maybe the banners actually didnt got traffic from same exact placment,
even if gou set exact placments for same banners,
if you run for short time, for a day you cant really sure
what is the origin of the result
also maybe even hour of day change the results
the bottom line, you can really control that expiriment when you have your own app/game/site
you can set 1 impression to banner1
2nd impression to banner 2
3rd...
4rh impression to banner 1
and on and on
otherwise you its like betting with high probabilities on the winners.. but its still betting and not pure logical decision..
give it another 42 k impressions, let see who the winner will be...
Interesting,
I would test same type of files BTW
Originally Posted by delash
im making some banners right now anyway, I wouldn't mind running the same test myself. maybe i'll start off by running the exact same banner in 3 separate campaigns as a control
although I'm not too sure if running web is as important on the filesize as mobile, but worth the test anyway
@delash - that's a good point, I'll drill into the stats to be sure there weren't placement discrepancies.
I deliberately ran these slow - 5 days' worth of data running throughout the day.
@iamatilla - yep, same placements, same frequency, same days and times.
When ever the network allows me, I always use PNG depending on the actual image.
The banner in your case study would be PNG in my campaign. I use JPG for photographs.
Text and drawings turn out more clear in PNG format.
These type of banners just render more clear in PNG format.
You target a gaming demo: They're hot for visuals.
I don't know if the following has anything to do with it.. but iOS devices just love the PNG format.
Interesting, but I don't know if is the PNG quality that made the difference. I think you should test more as delash said. There are some other variables who might screw the results. It's the same as you put 3 exactly the same banners in 3 similar campaigns on Facebook and you'll get different results..you can try that if you have a budget to throw away.
Theoretically, all external variables should be the same only the one tested should differ, but in this case we can't control the other variables...even if they look similar.
We really need more people to test this and see if this result is robust.
If it is replicable, then this would be a really amazing insight.
Some best results from scientific investigations are those that fit the following "Accepted wisdom says X, but our research and testing actually shows (opposite of X)"
[QUOTE=cmdeal;151031]We really need more people to test this and see if this result is robust.
If it is replicable, then this would be a really amazing insight.
Yea, now we will not just need to make thte banner ugly we will also make it be bigger,
I can see some logic behind it, because it might load slowly and that draw more attention and curiosity
But if I need to put my money I will bet this was just random effect of placement discrepancies
Also the number of the clicks are quite small, it remind me that
post http://finchsells.com/2014/03/26/spo...unky-campaign/
in that post there placement discrepancies, but there is something even with more significant that can change the result, adding one conversion to the second place and remove one converstion can actually change the winner
Actually it remind that rule
If there are 100 impressions in your sample, you need to see a 20% difference between variations to be sure that they actually mean something.
If there are 1,000 impressions, you need a 6.3% difference.
If there are 10,000 impressions, you need a 2% difference.
If there are 100,000 impressions, you need a 0.063% difference.
(taken from http://www.copyblogger.com/split-testing-tutorial/)
Always if something look strange to you, ask why?
So if that really true, it might be because of load speed of banners, or even difference in speed of banners might casue that effect.
Again to test the load speed of the banners you need to control the traffic source (it will be much easier)
Always ask why to understand the root cause of it, because that way you will be able to create new connection
and conclusions that hopefully will end up you getting more green papers..
I can't speak for Decisive on how they handle creatives but MANY mobile networks will automatically scale down your creative to their file size limit. Millenial for instance will scale you down to 10kb. As far as the exchanges go they all have different ways of doing it.
The slight differences in CTR is most likely because of a few possible reasons.
If you really want to mess with your own head, copy the EXACT same campaign + banners 3 times and you will see different CTR's for each. RTB is quite complex and there are a lot of factors that affect your CTR and how/when your ads get shown.
In my opinion the filesize wasnt all that big actually, UK surely has pretty strong mobile internet so 20 or 40kb thats really not that big difference and you usually wont notice any difference in loading speed with such a low amount of data. You would have to use a banner like 200kb in size or more in order to really have to wait for it to load. So I would say the loading speeds actually didnt play any role in this test. But maybe you discovered an unplanned trick - the usage of pngs
Originally Posted by matuloo
Science deals with facts and data, not opinions.
In science, opinions don't really matter.
Facts and data do.
It would be awesome if more people try to replicate this test and report back their results.
One methodological improvement that I would suggest for the test, however, is to test the following:
Three JPEGs, with filesizes of
- 5 kb (Small)
- 15 kb (Medium)
- 40 kb (Large)
Three PNGs with filesizes of
- 5 kb (Small)
- 15 kb (Medium)
- 40 kb (large)
This way, we can isolate the PNG effect from the image size effect.
Originally Posted by cmdeal
Science deals with facts and data, not opinions.
In science, opinions don't really matter.
Facts and data do.
It would be awesome if more people try to replicate this test and report back their results.
One methodological improvement that I would suggest for the test, however, is to test the following:
Three JPEGs, with filesizes of
- 5 kb (Small)
- 15 kb (Medium)
- 40 kb (Large)
Three PNGs with filesizes of
- 5 kb (Small)
- 15 kb (Medium)
- 40 kb (large)
This way, we can isolate the PNG effect from the image size effect.
Send the files over and ill run a test.
But was the campaign profitable!?? In all seriousness, it's an interesting question. Lander file sizes and load time are pretty clear cut. Banner sizes... quite a lot depends on how the network serves the files but on mobile I would wager you get some unexpected things going on (e.g. rendering of image, delay catches attention, low quality jpegs on high res displays look crappier + screen brightness and distance to face brings out the artefacting, etc.).
@deondup - I checked with Decisive before running the test, and subsequently checked the post-upload creatives. No file format conversion.
@matuloo - If we're approaching this as a scientific experiment (which is definitely the right way to do it), the interesting thing here is that the null hypothesis was not proved. As you say, the null hypothesis for this test is "the file sizes are too small, so there will be no difference". The interesting thing is that there was a difference, and a big enough one to be statistically significant. Hence, more testing required.
@MrGreen - sent!
@delash - I very much agree with you on the Facebook comparison - we might be seeing random noise from campaign delivery weirdness. Job #2 is definitely to attempt to replicate the results before getting too excited.
As far as statistical significance, the difference between the "medium" JPEG and the "large" PNG - the most interesting result, IMO - is statistically significant to 97% confidence using Bayesian inference (my preferred testing method) and also significant to 93% confidence using chi-squared.
@cmdeal - I agree! I'd love to see some attempts to replicate the results here - whether the result is "yup" or "nope" it'll tell us some very interesting things. I tend to refer to affiliate marketing as "the best-paid branch of science in the world" - and it's tests and results like this that make that so. I'd love to see the scientific process roll into action to either prove or disprove these results - either way, we'll learn some interesting things.
I'm going to run a second test myself to see if the results are replicable.
The one issue we'll run into with doing a comparative file size test using a selection of PNGs and JPEGs is that 24-bit PNGs don't have efficient enough compression to get their filesize down to JPEG-equivalent. However, I'll have a think about ways we can get around that...
Regarding the PNG vs JPEG filesize mismatch, this will depend on banner design. As the image gets less complicated the sizes will decrease and you should be able to find a banner where the combination of complexity + number of different colours leads to a similar 24-bit PNG and JPEG. Of course you could also mess with 8-bit PNGs but then they tend to look different.
Originally Posted by caurmen
This is great!
This is a good example of what I was talking about.
I can't read the text on my mobile phone unless I zoom in.
I don't know if it is related to your test results but this is the reason why I use PNG as much as possible.
Looking forward to more results.
Again, I think you are reading wayyyyy too much into this.
On any RTB platform there are MANY more variables that are far more likely to cause this.
Did you test all the creative variations on the same placement(s)? start them at exactly the same time? Run them for exactly the same duration (time-wise) ?
Even then, the way different creatives are rotated in the same campaign can vary on any number of factors. Initial CTR can have a big effect on when/how your ads are displayed.
If you really want to run this particular test you should have...
- Run all the ads on the same carrier (just one at a time)
- Run all the ads on the exact same placement (not the same App as placements vary on the same App)
- Start all the ads at exactly the same time
- Run all the ads for the same amount of impressions
- Ensure that your bids are exactly the same (on RTB bids are "auctioned" so what you bid is rarely what you actually pay and even then its usually displayed as an average in the platform)
Ofcourse there are many variables. But, PNG vs JPEG. Png is more optimized and standardised over jpeg. Especially on mobile, jpeg can be notorious both in terms of rendering time and quality.
However, I agree, more testing required! I've started a second test, and will report results in a few days.
Just to be clear, since there seems to be some misunderstanding here:
I am not saying that PNG definitely performs better than JPG for mobile advertising.
I have seen some initial interesting results which are worth testing further. It may be worth testing PNGs in your own mobile campaigns, based on those initial results, to see if they perform better than JPEGs. However, nothing's 100% proven yet.
We might be on the path to discovering something interesting - or we might not!
@ Caurmen - what I meant by the same placements was...if you run across 20 different placements its highly unlikely that the impressions will be spread equally across them all. So, to refine further, you need to make sure its one placement at a time and that its the exact same placement.
One thing where PNG can have a definite advantage is if you use transparency. On some placements it can make your banner look like its truly part of the site as there are no definite boundaries. With "missed message banners" a few years ago it absolutely killed on iPhone
@deondup - that's a good point. I shall make sure to check the distribution of placements on the new test - sadly I didn't collect full stats on the first one. I do recall that a significant number of the impressions went to one app, so it's
Any other suggestions for parameters to control for?
Has anyone been able to replicate this?
I have a second test running - I'll report in a day or so when it concludes.
I'd also be very interested to hear if anyone else had been able to replicate!
Very interesting test...
I know that in the UK mobile carrier networks are known for comprising image files over 3G such as jpeg which tend to decrease there quality but they are unable to this with PNG files.
Maybe this is the reason why the png file did pretty good.
@cashmoneyaffiliate - VERY interesting. It was indeed on UK networks.
I found this: http://calendar.perfplanet.com/2013/...recompression/
Salient excerpts:
Most mobile carriers force all HTTP traffic to go through their proxies that—among other things—recompress images on the fly.
All proxies recompress all JPEG images. Unfortunately, even very low quality JPEGs are recompressed to have even lower quality.
PNG images are very rarely recompressed. Static GIF images are getting noticeably posterized/discolored/dithered.
The standard HTTP header Cache-Control: no-transform which tells proxies not to modify responses was respected by all proxies I’ve encountered.
We may have an important revelation on our hands...
This is really getting interesting. Thanks caurmen for the experiment once again.
I like where this is going..
It shows the problem solving mindset of affiliates. Precious info about image formats are being dug up.
And this is just about the banner image. There are so many other small things that are looked over but can have quite an impact when it comes to mobile.
The next piece of the puzzle: do ad exchanges pass a cache-control: no-transform headers? If so which ones?
Was there ever any further validation of this result?
It's STILL on my to do list. Will get to it after my current project :-)
UPDATE: Second Test
As promised, I ran another, more rigorous test on this.
Firstly, I checked for 90% bot-free placements using this technique. Having found one (Grindr), I produced two identical banners, one in JPG at the usual quality I'd use for desktop and mobile banners, and one in PNG.
I then ran both of them on that single placement for approximately 100 clicks. The JPG gave an overall CTR of 0.275%, whilst the PNG gave an overall CTR of 0.309%.
Running that through a Bayesian A/B test calculator, we get a 79% chance that the PNG will be the strongest performer.
That's usually not enough to conclude one way or the other, but given that shows the same anomalous increase in CTR as the earlier test I ran, the two tests together tend to imply that we're seeing some real-world results, albeit not incredibly strong ones.
In light of that, I feel reasonably confident in saying that it seems that if you're running mobile banners, you are likely to see a small increase in CTR by running PNG rather than JPG banners.
Feel free to reproduce this test yourself - I'd be very interested to see any other results!
Further tests? I might run a test on wifi only - if the theory that this a result of how phones render PNGs and JPGs, I'd expect the effect to be stronger on wifi because loading times should be closer to identical. It would also be interesting to segment this effect by phone, and see if one handset in particular, or one family of handsets, shows this effect more or less than others.
Conversely, if carrier-mediated recompression of images was involved, the effect may be larger on carrier traffic.
This is awesome, thanks for the follow-up Caurmen! It not what I'd regard as a statistically significant effect, but it doesn't seem that PNG hurts CTR in any way, and may provide a benefit.
Why did you choose to only run them for 100 clicks though? That's only about 400-500 impressions right? If we're expecting a small effect then a larger sample is better, and would surely increase our chances of detecting the significant effect if does exist. I'd also be interested to see the effects (if any) on conversion rate.
@mcsteve - heh, honestly it was because the traffic I was getting was taking forever to build up the impressions! For some reason Grindr was on a go-slow, and the CTRs were low. I might restart it and run it for a few hundred more clicks to see if things stabilise.
@zeno - hmm, good point!
Home > Mobile >