Most Efficient Server Setup (Need some Tips for Restructuring!) (15)

I am pretty hopeless when it comes to analysing the best server technology, or the best server setup.

I'm also having a lot of difficulty scaling my current setup to deal with millions of clicks. Was hoping a few of you could chime in with the optimal setup for what I'm looking for...

So my questions:

1. Is it best to have your tracking installed on one server, and your landing pages on another?

Right now, this is the case for me. I have a dedicated server for CPVLab, and another for my LPs (several actually). Is it better to split like this than having both tracking and LPs installed on one machine?

If it's better to split, which server should be more powerful for most noticeable gain? The one with the tracking, or the one with the landing pages? (Redirect speed vs. Page load speed.) At the moment, my server that deals with the tracking is more powerful.

2. Is Litespeed better than Apache for handling redirects?

I'm running Litespeed but I stupidly didn't benchmark speeds properly before making the switch.

3. Is Imobitrax better than CPVLab for handling heavy loads of traffic?

CPVLab has fared me well, but since I've started hitting it with millions of clicks rather than hundreds of thousands, it's slowed to a crawl. I have to empty the database regularly. And navigating the main control panel is a genuine pain in the arse.

I haven't used Imobitrax, but I've heard that it's good. Would making the switch help me scale more easily?

4. In your humble opinion, which is the best hosting company to provide all of the above?

I am a complete technotard, so good support is a must. I fear having to make my own customisations like the plague.

Guess what I'm looking for is a simple breakdown of the most scalable setup...

For example:

One dedicated server for tracking (Beyond Hosting)
One dedicated server for landing pages (Behond Hosting)
Apache
CPVLab

vs.

One heavy duty Cloud VPS package for both tracking and landing pages (Storm on Demand, etc)
Litespeed
Imobitrax

I want a configuration that is fast and scalable to millions of clicks, but I also want to avoid my current situation where I have multiple dedicated servers, multiple VPSs and even some shared hosting accounts from way back in the day that I still haven't cancelled. Host Gator/GoDaddy FML.

Help appreciated!

Are you using some kind of CDN (content delivery network) for all your static elements? This should take the read/write load off your servers hard drives.


We run Ngix and have found it the fastest web server.

There comes a time in every mans life when you have to consider having something built. We were serving 1.5 million clicks an hour at 1 point and pretty much had to scrap all tracking. Depending on how much data you want to keep this can really add up quickly. I don't think any of the tracking systems are going to support this kind of load. So maybe look at core functionality you need/want. Then look at building a system based on MongoDB and Map Reduce. That's a lot of data to process. You're moving into big data realms which 99% of things aren't built for.

Feel free to hit me up on skype if you want and I can steer you as much as possible.

We run nginx, imobitrax and a CDN but as Smaxor said you may have to look into a custom solution utilitizing a NoSQL db like mongodb or apache cassandra for tracking.

Here's my $0.10 on this:

1) You definitely want to have your LPs and tracking on separate servers. The LPs don't put a lot of strain on the server, but every little helps.

The beefier server should definitely be the tracking server. Note that you want to consider disk speed (SSDs), CPU (tracking beats the crap out of your CPU, so the more powerful the better) AND memory.

2) I've not benchmarked it either, but unless you have found an absolute genius sysadmin to configure Apache, Litespeed is almost certainly faster.

3) I've not tried iMobiTrax either, so I'm not sure there. One thing that will help a LOT is updating your database software. If you get someone to install MariaDB, you'll probably find CPVLab responds a lot faster and more smoothly - that would definitely be my first port of call before changing tracking server. When I last tested, the switch to MariaDB gave something like a 5x performance improvement. After that, the next thing I'd advise would be finding a database expert and getting them to tune the hell out of your server.

4) Given you're in the UK, I'd highly recommend having a chat with Bytemark. They're primarily a more techie company, so may not be a perfect fit for you, but they do offer managed servers, they're very friendly and helpful, and they Seriously Know Their Shit.

I'm also going to PM you about one or two things that I can't talk about publically yet on this front

As Smaxor said.. you might need to run your own tracking system now. Which, by chance, I discovered isn't that hard to set up. I built a cloaker, for my own reasons, but built in tracking based around ISP and GEO/ORG data.. and as I progressed, I realised that I had built something not too different from iMobitrax in terms of tracking and redirection features. It all runs on XML files, no mySQL.. so super fast at processing the clicks that come through it..

Also, you must go CDN.. at least for images.. pretty easy and cheap to set up..

Originally Posted by Finch

I am pretty hopeless when it comes to analysing the best server technology, or the best server setup.

CPVLab has fared me well, but since I've started hitting it with millions of clicks rather than hundreds of thousands, it's slowed to a crawl. I have to empty the database regularly. And navigating the main control panel is a genuine pain in the arse.

I haven't used Imobitrax, but I've heard that it's good. Would making the switch help me scale more easily?

4. In your humble opinion, which is the best hosting company to provide all of the above?

I am a complete technotard, so good support is a must. I fear having to make my own customisations like the plague.

Guess what I'm looking for is a simple breakdown of the most scalable setup...

For example:

One dedicated server for tracking (Beyond Hosting)
One dedicated server for landing pages (Behond Hosting)
Apache
CPVLab

vs.

One heavy duty Cloud VPS package for both tracking and landing pages (Storm on Demand, etc)
Litespeed
Imobitrax

I want a configuration that is fast and scalable to millions of clicks, but I also want to avoid my current situation where I have multiple dedicated servers, multiple VPSs and even some shared hosting accounts from way back in the day that I still haven't cancelled. Host Gator/GoDaddy FML.

Help appreciated!

I'm using Voluum at the moment for high traffic campaigns and prosper202 for low volume stuff like search ppc.

For self-hosted there's Avazu202, a mongodb implentation of Prosper202. Might be worth checking out. Haven't used it myself though. One of the developers posted about it here:
http://stmforum.com/forum/showthread...FOR-GOOD/page2

Building a custom distributed system with java and Hadoop map reduce is definitely something I'm looking into as my traffic grows. At that point it will probably be hosted on something like Amazon EC2 or Red Had Openshift - whatever ends up being more cost effective and easier to maintain.

Originally Posted by caurmen

Here's my $0.10 on this:

1) You definitely want to have your LPs and tracking on separate servers. The LPs don't put a lot of strain on the server, but every little helps.

The beefier server should definitely be the tracking server. Note that you want to consider disk speed (SSDs), CPU (tracking beats the crap out of your CPU, so the more powerful the better) AND memory.

2) I've not benchmarked it either, but unless you have found an absolute genius sysadmin to configure Apache, Litespeed is almost certainly faster.

3) I've not tried iMobiTrax either, so I'm not sure there. One thing that will help a LOT is updating your database software. If you get someone to install MariaDB, you'll probably find CPVLab responds a lot faster and more smoothly - that would definitely be my first port of call before changing tracking server. When I last tested, the switch to MariaDB gave something like a 5x performance improvement. After that, the next thing I'd advise would be finding a database expert and getting them to tune the hell out of your server.

4) Given you're in the UK, I'd highly recommend having a chat with Bytemark. They're primarily a more techie company, so may not be a perfect fit for you, but they do offer managed servers, they're very friendly and helpful, and they Seriously Know Their Shit.

I'm also going to PM you about one or two things that I can't talk about publically yet on this front

Originally Posted by theoptimist

Brilliant tip Caurmen! Can I get MariaDB installed on a Managed VPS server by my hosting company or do I have to install it on a unmanaged server?

service mysql stop
killall mysqld_safe mysqld
ps -A | grep mysql

#CHECK FOR DISK SPACE
echo "YOU NEED THIS MUCH SPACE"
du -sh /var/lib/mysql
sleep(5)
echo "Compare to:"
df -h

service mysql stop
tar pczvf /home/mysql-backup.tar.gz /var/lib/mysql/
 
cp -Rf /var/lib/mysql /var/lib/mysql-old
cp /etc/my.cnf /etc/my.cnf-old

/scripts/update_local_rpm_versions --edit target_settings.MySQL50 uninstalled
/scripts/update_local_rpm_versions --edit target_settings.MySQL51 uninstalled
/scripts/update_local_rpm_versions --edit target_settings.MySQL55 uninstalled

/scripts/check_cpanel_rpms --fix --targets=MySQL50,MySQL51,MySQL55

vi /etc/yum.repos.d/MariaDB.repo

# MariaDB 5.5 CentOS repository list - created 2013-08-23 21:51 UTC
# http://mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/5.5/centos6-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

yum install -y MariaDB-server MariaDB-client MariaDB-devel
/etc/init.d/mysql start
mysql_upgrade
/etc/init.d/mysql restart

/scripts/easyapache --build

Originally Posted by BeyondHosting-Tyler

Do you have the 2.16 campaign cron setup? This will generate your stats in the back ground so its less intense. We've seen 20GB+ databases which are 25 million clicks that still work just fine and are fast.

Awesome tutorial, Tyler - thanks! Added to the newsletter..

Something that is rarely mentioned is that you should have a Gigabit network card, not some low-end 100 Megabit card.

Makes a H U G E difference..

Originally Posted by nissangtr

Is this only available for CPVLab 2.16? I am on 2.15...

Originally Posted by seank

Something that is rarely mentioned is that you should have a Gigabit network card, not some low-end 100 Megabit card.

Makes a H U G E difference..

FWIW, With the right engineer/s building your custom tracking on top of Amazon Web Services you can scale massively. The last company I worked for did an average of 1 billion impressions a day. Some caveats,

- This is not cheap
- This is not easy

If it's built correctly you'll be like Bradley Cooper from Limitless, because you can pretty much scale to the ends of the earth.