Home > Programming, Servers & Scripts >

CDN Benchmarks Thread - Testing & Comparing Multiple CDNs w/ Real Traffic (3)

01-11-2016 01:37 PM #1 integrity (Member)
CDN Benchmarks Thread - Testing & Comparing Multiple CDNs w/ Real Traffic

Hey guys, I thought I would put together a little thread we can all update with stats of our own CDN testing.

I've been using Cedexis for quite a while (like Zeno recommended in THIS thread) but unfortunately did not find data for some CDN providers I was interested in. So, I decided to start running my own tests.

Note: The data I've gathered so far is very brief--I understand in order to make any real conclusions we're gonna need a LOT more data, so I'll be updating it as I have more. Here's the first test I ran:

Geo: Indonesia
Traffic: Pop
Device: Mobile & Desktop
DNS: Cloudflare for tracker (No CDN or security enabled), DNSMadeEasy for CDN CNAME.

All landers on each CDN are the exact same .html file, except the one not using a CDN and on the VPS--that one is the same, but it's .php and is hosted on a 512MB/1 Core VPS running nginx and php-fpm in DigitalOcean's NYC3 location.




Now these are the stats for mobile and desktop combined. Here's how the data looks once we separate everything:

(Please disregard the 2 lonely conversions, this campaign wasn't even supposed to convert. But alas, somehow managed 2.)




I know it's not enough data, but I did find it interesting how CDN77 had the highest CTR on mobile, but the lowest CTR on desktop. Almost 2x difference between 1st and last on each test! Perhaps some CDNs had a much better peering to certain mobile carriers, which accounted for the increased mobile clicks? Not 100% sure on this. Really curious to know how this would change after running for weeks like this.

The other thing that really surprised me was that my US hosted VPS did so well compared to the CDNs, especially in such a far-away country like Indonesia. I suppose any time you have a winning campaign somewhere, it's worth testing a local VPS/dedicated server that has been optimized well. But seeing how well this US VPS fared against the CDNs, I probably wouldn't mind running my pages off there from now on. Keep in mind, this is a $5/mo VPS.

Before this test I was running almost exclusively on Rackspace's CDN, but now I'm inclined to change my setup. I'll definitely have to test more geos out to decide which CDN to stick with, but for now I think S3 + Cloudfront is a great bang for your buck.

If anyone has some CDN benchmarks, feel free to share!

01-11-2016 03:24 PM #2 zeno (Administrator)

Nice initiative mate.

It's taking the time to do tests like this that can teach you things that forever give you a small boost.

I would run this kind of test for a lot longer and try to get statistically significant comparisons on both a CTR and more importantly CVR basis.

The main problem is... the results cannot be transferred from one country to another, but you can at least do it for large geos.

The point of Cedexis is that it can replace your typical DNS platform and redirect traffic based on the community data of fastest response times for CDNs per region - so it basically would make a decision based on thousands of tests similar to this. Pity it's quite expensive and you'd need landers distributed to all CDNs (though probably easy to automate).

01-11-2016 04:52 PM #3 integrity (Member)

Quote Originally Posted by zeno View Post
Nice initiative mate.

It's taking the time to do tests like this that can teach you things that forever give you a small boost.

I would run this kind of test for a lot longer and try to get statistically significant comparisons on both a CTR and more importantly CVR basis.

The main problem is... the results cannot be transferred from one country to another, but you can at least do it for large geos.
Thanks Zeno, great point. I just wanted to post this up to get the thread going, but definitely need a lot more data to reach any solid statistical significance.

The point of Cedexis is that it can replace your typical DNS platform and redirect traffic based on the community data of fastest response times for CDNs per region - so it basically would make a decision based on thousands of tests similar to this. Pity it's quite expensive and you'd need landers distributed to all CDNs (though probably easy to automate).
Definitely something to keep in mind for those that can benefit from such setup. A gentleman from Cedexis called me about it, but it was a bit much for my current needs. However, I can totally see the advantages from something like that.

Home > Programming, Servers & Scripts >