Optimizing Thrive for Pure Performance [Self Hosted] (35)
After hearing about Thrive recently, I've been really curious to see how well it performs.
I run a lot of pop traffic, which we all know can get a little expensive with
First Impression
Not gonna lie, the installation was a little tedious. More so than my previous experience using iMobitrax and the STM tracker. But Thrive's excellent documentation (especially the built-in system checker) made everything pretty easy to figure out. David and Aziz on the support team also helped me get the cron jobs (btw, never seen a tracker with cron jobs) working, which was a big help.
Once set up, I was a little lost in the new interface. After actually sitting down to watch the tutorial videos I started to get the hang of things and set up my first campaign.
I like how you can create rotations with specific redirect rules, then re-use those same rotations for multiple campaigns. Big time saver if you have a lot of campaigns with very complex redirect rules. Also really liked the fact you can separate campaigns into categories to make it easier to sort through.
Overall, I'm very impressed with the new feature sets.
Performance
Okay, this is where I can't say I was too impressed. Hopefully some of you guys can chime in with some advice as to how to resolve this.
I used 2 different test servers to run this on. The first one being an 8 core 16GB SSD VPS on DigitalOcean running Ubuntu 14.04 LTS, Nginx, PHP-FPM, MariaDB and Memcache. After testing with that one and not being happy with the performance, I tried again with a beefier configuration, this time going with a 20 core 64GB SSD VPS. And surprisingly, ran into the same performance woes. Below you will find more detailed information on that.
The test ran was a 1 to 500 concurrent users in 60 seconds rush test on Blitz.io pointed at Google.com. This generated roughly 12k clicks in my Thrive stats during those 60 seconds.
David from Thrive's support team was kind enough to go into their setup and provide me with the exact configuration they use on their test server. Here's the exact message he sent me:
The testing that we've done is on AWS as Tom stated. The server we tested between 5 and 10 million clicks a day was an r3.2xlarge which has an 8 core processor and 61 GB of ram. We run MariaDB with the default settings with the exception of the following: max_connections = 300 connect_timeout = 5 wait_timeout = 600 max_allowed_packet = 16M thread_cache_size = 256 sort_buffer_size = 32M bulk_insert_buffer_size = 32M tmp_table_size = 512M max_heap_table_size = 64M query_cache_limit = 128K query_cache_size = 64M innodb_buffer_pool_size = 12G innodb_log_buffer_size = 16M innodb_file_per_table = 1 innodb_open_files = 400 innodb_io_capacity = 400 innodb_flush_method = O_DIRECT innodb_flush_log_at_trx_commit = 2 We also determined that web service is a vital component of making it work. We chose to use Apache Event but Nginx would work as well. David
[Edit] After raising the innodb_buffer_pool_size to aprox. 70% of the server's ram and raising the innodb_log_file_size to 1G, I was able to get a bit of a change in the results:
No matter what I did, I just could not get rid of that lag spike coming in at the ~15 second mark.
After discussing this with Thrive support again, they just told me my best bet would be to use their managed service, as that's where their highest volume clients were. They hadn't run blitz.io on their end so they could not tell me how their results were like.
But based on that, I wouldn't feel comfortable running high volume pops through this Thrive install. I've also never gotten that 502 Bad Gateway error with any other tracker, no not sure what that's all about.
I did run a low-volume mobile campaign to test (1-2k clicks a day) and it was profitable, so I know Thrive works. But it's the high volume and concurrent users I'm afraid of.
And just so you can get an idea of what I'm comparing it to, here's my
Keep in mind, Voluum's clicks are cached, so all subsequent clicks will be much faster than the first one, but the graph is still solid throughout, which is what I'm looking for. I would've liked Thrive to have remained solidly at the 50-70ms mark throughout the test, not jump all the way to 1s+ in some instances.
If anyone has some advice as to what to test here, I'm all ears. Just sharing my experience with Thrive so far. If this can be resolved, I'd definitely give Thrive a shot. Big thanks to Tom, David and Aziz for all the help in getting my Thrive install going, as well as optimizing to as much as possible.
Interesting,
I wonder if you tested the latest P202 1.8.10 to check their redirect speed compared to
I'd wager this has nothing to do with
8 cores / 16 GB RAM and 20 / 64 is absolutely overkill and should handle that fine.
Try doing this test with a flat hit rate, or with the ramp much less steep. You may find the slow down isn't due to load but just a big backlog of connections. Hitting a server with hundreds of new connections per second is going to cause major problems if it's not configured to properly churn connections e.g. drop them and prevent massive backlogs, or produce 1000's of PHP-FPM workers/childs or whatever they are called.
I can guarantee you that PHP-FPM will absolutely screw things over if not configured to expect high volumes of connections. It could be spawning hundreds of processes/workers that are each taking 64 MB of RAM and not dying for minutes.
Also try just doing it with a test file... nothing to do with
Originally Posted by zeno
Also try just doing it with a test file... nothing to do with
Haven't tested a single file bypassing Thrive on this specific server, but I did to it so another on on DigitalOcean with much lower specs and a pretty identical setup which handled the load just fine, so it lead sme to believe this is Thrive/Database related. I can run that test on this same box and post some results.
I don't know much about PHP, but I'll look into how to do something like that. Thanks Zeno.
Between different servers one of the simplest things that will change is the firewall settings - or basic system config like synflood protection, IPtables, etc.
You could take 5 different 1 GB VPS servers from different providers, install the CentOS 6.5 x64 distro they all offer, and I bet you they would all have slightly different settings.
In this case, rather than test configuration first, I would use a simple problem-solving approach to test various elements - much like how you can troubleshoot a car not starting without knowing anything about how the engine works.
1. Reboot the server. Upload a text file. Blitz that text file. Result?
If it handled fine, then it's not because of the firewall and system-level config.
2. Upload a simple PHP file that does some maths, or use <? phpinfo(); ?>. Repeat. Did it die? If so, probably PHP-FPM config. I'd expect it to be this or firewall-related.
3. Upload a simple PHP file that does some database-related stuff. Google around I guess, may need to create a MariaDB database.
4. Test the
5. Test a
By doing something systematic you can identify the likely problem area without knowing much about what goes on under the hood, and it will help you eliminate much of the time spent trying to find the root cause.
Originally Posted by zeno
Will this do? http://www.w3schools.com/php/php_mysql_connect.asp
Looks like the Thrive login.php address got a little laggy there towards the end, although not nearly as bad as the campaign tracking link. Gonna go run that database related script.
Server tested this time was the 8 core 16GB box.
Thanks Zeno.
Interesting post, very curious to see what eventuates out of this.
Will definitely come in handy. I'd love to run a blitz test on my managed service and compare results to see if that shows any difference.
Originally Posted by andy_d
Not sure if running a simple redirect script on a server of yours will allow blitz.io to get to get to the
Mate, here's a question: can your server handle the 300 requests a second that
Maria, MySQL, couch and mongo can all easily read and write 1000-2000+ times a second, I wouldn't be worried. I would work backwards and test the concurrency capability of your server's http and PHP serving (simple pages, do a simple loop with PHP within the HTML, then go from there.
No need to test text file serving as others have suggested. I would look into httperf, you can set the size, requests per second etc. It's the best tool for testing the concurrency limits of your server.
I also don't agree with zeno's suggestion that 8 cores is overkill. Maybe the ram (you won't be using more than a gig) but not CPUs. I have found that cores limit things before anything else. With a service I run, I found 220 clicks the cut-off for 8 cores (before requests started dropping). Thrive is probably not as intensive. In Linux you can assign cores to various functions (affinity) or disable them to test things.
Your machine is probably not optimised correctly (I'm talking nginx but also the network interface).
Might be worth switching from DO to Ramnode or another provider. Each has a different interpretation of "virtual CPU" or "fair share". Some are terrible for hosting trackers. Check serverbear.com for stats on many VPS packages, it's a super useful site when hunting for a VPS.
Btw, DO is not true cloud, not high availability either. Load balancing not as easy as Linode either. Then again Linode isn't an infallible host. If iwStack owned and operated a server in the US like they do in Milan (one of the best in the world, true HA and failover -- actual cloud without the cloud-high price tag) then I'd say go with that. And do if you're running in EU. OVH is the most reliable VPS in the world (also Europe mainly) for a not-bad price.
The reason for the preceding paragraph is that if you are doing this to save on
For a more familiar option, Ramnode doesn't throttle CPU UNLESS you're hogging more than 90% for 45 mins or something. Then your VPS is shut down (process leak? Bad coding on a cron?) And if it keeps happening they send you away for good.
If you can't figure out the benchmarking and what is causing the issue, reply and I'll see if I can dig up some PHP/HTML scripts to help you determine the issue.
Originally Posted by redrummr
Thanks for that detailed post.
The last test I ran as per Zeno's suggestion was to a PHP info page and the server was able to handle it fine. And from previous tests, I too have noticed quite a large burden being put on the processor cores. On the last self-hosted imobitrax install I had, I used a quad-core 16GB dedicated server on LiquidWeb and it was solid for this test. Anything less than a quad core and the performance would start suffering.
And now that you mention it, I don't think I had stress tested a tracker on DigitalOcean like this before.. I know Ramnode is solid as I used them for a while before switching over to LiquidWeb, but their offerings only go up to an 8GB quad core. Plenty for me at the time, but not enough for what I need now.
And you're right, the initial plan was to try and save some money on Voluum events. I have been looking at dedicated servers on Hetzner for my EU campaigns, as well one down south in the US for LATAM. Will take a look at OVH now that you mentioned them, as well as iwStack for testing (great share--they look much better than DO). Although at this point, I would gladly pay the Voluum premium just to not have to worry about any of this stuff. Was never really worried about them seeing my stuff, but it has crossed my mind a few times. I'd be curious to know about that solution you mentioned.
I know that with a pretty much identical setup (even far more underpowered) I was able to run a similar test on imobitrax. The only difference being the server was not on DigitalOcean. If iwStack charges hourly, I may go ahead and set up another test box over there and run the test again. I'd like to know Thrive can handle that much volume before going out and purchasing a dedicated server (or two).
It's about 5am now and I want to finish a lander before I go to bed, so this new testing will have to take a back seat until later on today. Will let you guys know how that goes.
Hey mate, I don't think it's a
Here's a screenshot of testing
It's a 12 Core , 32BG SSD dedicated and fine tuned for optimal performance ( nginx / php-fpm / mariadb )
Originally Posted by bbrock32
I had only tuned the my.cnf (which was all I had to do for imobitrax, aside from changing all the database tables to innodb, so I assumed this was the same) but I guess I'll need to look into the nginx and php-fpm configs as well.
Got some work to do then. I'm a little disillusioned with DigitalOcean from what Redrummr told me, so I'll probably get it set up elsewhere before I take another crack at optimizing it.
Integrity, I can send you my server optimisation sequence. PM me, try that first. Btw OVH's $100~ dedis even if only 6 core (if I remember right) would be better than the highest core-count DO instances.
*deleted*
Hey guys,
First, thanks @integrity for the review. Appreciate you putting time into this.
And appreciate everyone who has had input here. Your comments are very helpful.
I've never used blitz.io, so I signed up for an account today, and it's pretty useful, so I think we'll keep using it.
We did a test on our internal Thrive install, which is on an 8-core processor with 61GB of RAM running on our AWS setup that we do for Managed Users, which means that the install has the full set of cron jobs running just like any other production install. We have our own set of configurations for AWS.
The test results are below. This was done by using a Thrive campaign link as the test, e.g.: /path/lp.php?trvid=10004&trvx=681c3da5 and from Virginia, close to our test server.
Not sure if it matters, but the landing page we redirected to was a pretty simple, fast loading page.
The test equates to over 16 million clicks/day. As you can see, toward the end of the test, when concurrent users were over about 350, the response time starts decreasing. This actually fared better than our own internal tests previously. To more readily handle over 10 million clicks/day and be able to handle spikes and other processes of the app, we always suggest to go bigger if you can.
It's because of internal tests (and now this test) that we claim what the software can do. However, it does depend a lot on the server.
With tracking software is probably 50% of the equation, while the server the other 50%. By self-hosting, you are only allowing us to control half of the equation, and you are essentially outsourcing the other half. Since we can't control the quality of that 50%, performance across different servers can vary and in general, all standard configs will not be able to handle big scale.
To be honest, we are much more software guys than hardware guys. We know AWS, and that's what we use for Managed Clients. It's hard for us to take any server (and without full access) to just make it for high scale easily. And we make our Managed Service, honestly, as cost effective as we can. And we've been pretty successful with those who have tried it. Some of our biggest volume users are on Managed.
I think in the future what we can do is work with @zeno and @goshev to come up with an optimal self-hosted setup for STM. I know there are people who really want the security of self-hosted. We'll have to figure out a price point that works for everyone. Our $99 is an intro rate, and we're going to consider raising rates in the future (current users can be grandfathered in) so we can support projects like these and our growth.
Anyway, in general, love this thread, and we're still in our early days, so it's beneficial to us to get everyone's input and throw around ideas. I know we can improve a lot still, and plan to. It's just a matter of constrained resources and prioritization.
Thank you to everyone's support of Thrive!
Hey Tom,
I really appreciate you taking the time to Blitz test your own servers--those are some impressive results!
Like you clarified over PM with me, the 10-12k clicks a minute does translate into 14+ million clicks a day. Looks like my mental math was a little off in that regard, lol. I was really underestimating the magnitude of the test I was trying to run. Although I did try to run half that on my setup (7 mil clicks/day) and I had the same lag spikes in the same places, so I'm pretty sure something in my setup is not right.
Redrummr was kind enough to PM me a very detailed server optimisation walkthrough that I'll be performing as soon as I get a chance. And from the looks of it, DigitalOcean may not be the best place to host a tracker.
Once I finish all that I'll be reporting back with my findings. Hopefully this can help other guys on the
Definitely better off with a dedicated server vs VPS.
I'd be interested in the sever optimisation steps if you care to share :-)
Andy, sent.
Some VPSes are great though! Depends on your need. The only downside of great VPSes like Ramnode is that, during peak traffic, other customers sharing your CPUs are also likely to experience more traffic, giving you less of a share. Dedicated from OVH @ $100 dwarfs a "high performance" DO instance @ $80 imo. 
Just FYI - from what you've said, I'd recommend looking at your PHP-FPM settings first. In my experience, if an nginx setup is going to have performance issues, that'll often be where they start.
You may even want to try looking at an Apache config - for heavy PHP usage it actually benchmarks pretty well versus nginx because nginx's efficiency is countered by mod_php being more efficient than PHP-FPM. I've not had
One question: how much traffic is actually being sent in a practical application?
Just thinking of the practicalities here... I mean, it's all good to show that a 8 core 32 GB RAM VPS can handle 10 million clicks a day for example - but that traffic might cost you $20,000 a day, or $600,000 a month. At which point spending $50-100 on a VPS would be a pretty nonsensical move when you could buy a dedicated server that would unarguably have a much higher volume tolerance and performance.
But i'm all for these kinds of tests. They bring out the server nerds and I feel somewhat amongst kindred.
@zeno very good point. No reason to really test at a pace that's going to be way beyond your actual number of clicks you need to handle either.
Originally Posted by caurmen
That's one of the things Redrummr mentioned in the server optimisation tips he sent me, and it did make quite a positive impact. Namely raising the number of max children in the conf file.
Now that I've seen you mention Apache for high volume stacks, I'm pretty curious to run a few tests with it. David from
Originally Posted by zeno
Originally Posted by ffclogin
I'll be sharing some new results that I've obtained after implementing the suggestions posted in this thread.
So, I finally got around to setting up a proper server to run
New server has an E3-1240 V2, 32GB ram, and a 463GB SSD RAID10 array hosted on LiquidWeb. Although their Storm Dedicated servers are not true "bare metal" servers, since there is still some virtualization, it's still pretty darn close and performs really well.
Thanks again to Redrummr for his server optimisation tips, as they proved to be a tremendous help getting these numbers. Here are my updated numbers with the new setup:
(Edit: Not as fast as yours Tom, as both your test server and Blitz were on the AWS Virginia location. )
As you can see, the difference is huge from my previous setup. At peak load, the maximum speed gets as high as 341ms, when it used to be 1s+ before.
I will also add that the unoptimized LiquidWeb setup still performed far better than any DigitalOcean VPS I tried. Average redirect speed before making any optimizations to the server were 244ms and the slowest one at peak load was 544ms.
Anyways, I'm pretty happy with my current setup and how much volume it can handle. Thanks again to everyone for their suggestions (especially Redrummr for such a detailed and invaluable guide!), as well as Tom, David and Aziz at
Nice thanks for sharing man!!
@integrity Thanks very much for sharing! It looks like you've gotten your setup to a good spot. There's always room to tweak and improve on performance, so keep adjusting a thing or 2 here and there.
Nothing much to add except that for our users, we have some on Apache, but the Apache is the Apache with the event MPM, which is a more advanced setup.
More recently, we did upgrade most of our Managed Users to Nginx over the last few weeks. I think we'll be sticking with Nginx over the long run since it's performing very well for our highest volume users.
holy hell....
u ain't gonna find this type of conversation on WF...
STM, worth every penny.
Great info guys. Can someone please PM me the server optimization tips? I'm running a similar setup on liquid web and trying to optimize.
i didn't read all the above posts so may be repeating but man that is not NEARLY enough for a million or 2 million hits........I do about 3 million a day and you gotta have some serious hardware if you're going to use your own mysql database and php with regular apache. Now litespeed webserver honestly will take the load down to 1/10 what it was, but you gotta lease it, 20 bucks or so. There's no way around it man if you want to pump that much you need some muscle. I have my own complete dedicated for JUST the DB with 30 gigs of RAM as I don't love litespeed all that much, but get it installed and you will have no issues at all. Same with nginx it's just like litespeed but my host supports litespeed and you can have it turned on in minutes.
i didn't read all the above posts so may be repeating but man that is not NEARLY enough for a million or 2 million hits........I do about 3 million a day and you gotta have some serious hardware if you're going to use your own mysql database and php with regular apache. Now litespeed webserver honestly will take the load down to 1/10 what it was, but you gotta lease it, 20 bucks or so. There's no way around it man if you want to pump that much you need some muscle. I have my own complete dedicated for JUST the DB with 30 gigs of RAM as I don't love litespeed all that much, but get it installed and you will have no issues at all. Same with nginx it's just like litespeed but my host supports litespeed and you can have it turned on in minutes. Seriously I had one server with 2 ssd's one for just mysql and only 8 gigs of RAM and it ran no problem 2 million hits a day with litespeed installed. load was almost nothing. It's great software.
Originally Posted by tosh0us
The noobs over at WireTree totally fucked my server up ($800 a month dedi) and it crashed 3 times in a a few days, so I quickly got a refund.
OpenLitespeed is also avail (free version), albeit, isn't supported and has less features..
Had loads of trouble getting installed and stable myself, though, I'm not a qualified server engineer. Much hair loss was accumulated.
Originally Posted by andy_d
I actually switched over to Beyond Hosting and got set up with a dedicated box + litespeed with them after my liquid server starting having some bugs (noob mistake on my part).
Tyler optimized my server himself (Litespeed and PHP config) and it's been working really well. I originally wanted a box in the EU, but settled for one with BH here in the US and haven't really had an issue with clickloss (only slightly more clickloss than
Great thread!
Awesome thread
Has anyone using
Originally Posted by magnate
These were the best results I was able to get on a fresh
(Test is a 1-100 concurrent users with a 10 second timeout)
This is the same server and tracker install, but with ~5 million clicks in the DB:
(Test is a 1-250 concurrent users with a 10 second timeout)
Home > Tracking Campaigns > Thrive