Make Sure Your Server Can Handle Any Campaign You Throw At It: 2015 Edition (8)
Make Sure Your Server Can Handle Any Campaign You Throw At It: 2015 Edition
Do we still need to test our servers in 2015? Hell yes we do.
Whether it's a CDN that you need to know will work as fast as promised when you're throwing 100 visitors a second at it, a customised tracker that keeps your data private but you need to know will stand up to the heat of a campaign, or a landing page server hosting a complex lander funnel, a server that can't stand the strain can screw your affiliate business.
And whilst Real-User Monitoring is great, if you're operating at scale you need to know if your server's going to die before you place the Insert Order and start the traffic flowing.
Back in the day, I used to recommend a tool called Blitz.io. It's still pretty good but it has become a lot more expensive, and in the last two years a lot of other options have arrived.
Including my new favourite tool for server testing: Gatling.
Gatling's almost as easy to use as Blitz, and at least as powerful as the other top server-testing tool, JMeter. And in this tutorial I'll show you the basics of how to install and use it.
Who will find this tutorial useful, and what do you need to know?
Almost every affiliate can get some use out of load-testing and response-testing their servers.
In particular, load testing is useful for more experienced affiliates running bigger volume, and anyone running a tracker that they're hosting on their own servers.
Gatling is particularly good for affiliates running serious scale, as its more advanced features like sending real-world useragents and running through an entire funnel are great for showing up the kind of performance issues that less thorough load testers won't spot, and that can really cost you a lot of money if you're throwing millions of clicks a day at a campaign.
You won't need any particular coding knowledge for this basic tutorial, and you can go through it on Mac or Windows.
However, if you want to get into the more advanced uses of Gatling - and there are a lot of advanced uses to get into - you'll need to be comfortable with some basic coding and system administration. Either that, or you'll need to pay someone who is.
This tutorial will take you through a comparatively simple test for Gatling, which is still more detailed than the tests most rival platforms can offer.
If you want to test a tracking system, you'll need to set up a campaign to test before you start this tutorial. I'd recommend using a second landing page or a non-functional URL for your offer URL unless you're sure your affiliate network is OK with you stress-testing their systems!
If you just want to test a landing page, either on a CDN or on a single server, you'll just need the URL of that landing page.
What Is Gatling And What Does It Do?
Gatling's a tool for throwing simulated traffic at your servers. Unlike a simple tool like Blitz, it's capable of simulating a Web user very precisely, including sending useragent strings, taking multiple steps and following redirects, and even filling in form information.
Uniquely for this type of tool, it includes a recorder function, allowing you to record your own test "patterns" without needing to code. And it produces a beautiful, very detailed report of the results of the test at the end - far better than most reports from paid testing tools like Blitz:
You can comparatively easily run Gatling on a remote server: just install the Java Development Kit, download the package and you're good to go. However, in this tutorial we'll focus on running it on your desktop or laptop machine.
Installing Gatling And Recording Your Test
First, if you don't have a recent version of the Java Development Kit installed, download the most recent one from Oracle. It's a very simple install.
Then, download Gatling itself from the project download page. Get the "Gatling Bundle (.zip)" from "Latest Release" - you don't want the "Snapshot" version.
Unzip that somewhere onto your computer.
Now, we'll set up your browser for recording. You'll not be able to use your browser for regular browsing while you're doing this, so be sure that you have all the URLs you'll need (campaign URL or lander URL) copied out first.
Gatling works by acting as a proxy between your browser and the Internet. That means it can capture all the data about your browsing session, including timings (so you can wait around or click through quickly) and user-agent data.
To set this up:
- If you're using Firefox, go to Options->Advanced->Network, then click "Settings" under "Connection", and select "Manual Proxy Configuration". Next to "Http proxy" fill in "127.0.0.1" and Port "8000".
- If you're on Chrome, proxy settings are system-specific. Probably the easiest way to set a proxy is to download a proxy management extension - I like FoxyProxy. Once you've got that, set up a proxy with IP 127.0.0.1 and port 8000, and activate it.
Once you've done that, navigate to the folder you unzipped containing Gatling, and then go into the "bin" directory within it. If you're using Windows, double-click "recorder.bat". If you're on Mac, double-click or otherwise run "recorder.sh".
Once you do that, a big scary-looking window will pop up full of various settings. You can safely ignore them for now - just check that "listening port" is set to "8000" and change "Class Name" if you want to name your recording something specific.
Then hit "Start!" and navigate in your web browser to where you want your test to start. If you're testing a tracker, paste the tracking URL into your address bar and hit enter, then click through on the landing page. If you're testing a landing page, just paste the URL into your address bar and hit enter to go there.
You'll see that the Gatling Recorder window has changed to show that it's recording what you're doing. When you're done, hit "Stop And Save". You can set your browser to not use the proxy any more now.
Congrats - you've just recorded your first Gatling test!
Setting Up The Scale Of Your Test
The only thing that we can't do for this basic test in the recorder is set how many users we want to flood our testing server with.
To do that, we'll need to edit the Gatling code. So, head into the user-files -> simulations subdirectory of your Gatling directory - there should be one file in there called "RecordedSimulation.scala" (unless you changed the name you were recording to). Open that up in a text editor.
You'll see a whole pile of code that looks like this:
import scala.concurrent.duration._
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
class RecordedSimulation extends Simulation {
val httpProtocol = http
.baseURL("http://154.62.35.153")
.inferHtmlResources()
.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.acceptEncodingHeader("gzip, deflate")
.acceptLanguageHeader("en-US,en;q=0.5")
.connection("keep-alive")
.userAgentHeader("Mozilla/5.0 (Windows NT 6.0; WOW64; rv:38.0) Gecko/20000101 Firefox/37.0")
val uri1 = "http://154.62.35.153"
val scn = scenario("RecordedSimulation")
.exec(http("request_0")
.get("/tracker/?cmid=NQ==&cvar1={keyword}&cvar2={match}&cvar3={cid}"))
.pause(3)
.exec(http("request_1")
.get("/tracker/gotoof"))
setUp(scn.inject(atOnceUsers(1))).protocols(httpConf)
}
Fairly scary. Fortunately, we only need to modify one line of it, which is the line beginning "setUp".
Start off by making this a bit easier to read: expand that single line from
to
setUp( scn.inject(atOnceUsers(1)) ).protocols(httpProtocol)
The line that controls the number of fake users we're going to send to our tracker is the line starting scn.inject. Currently, it's just set to send a single user - which isn't going to stress-test our server much.
There are loads of options within Gatling for how many users we send to attack our server: here's a full list of the options.
In this case, let's set up a ramped attack where the number of users will start out small but end up pretty large. Replace atOnceUsers(1) with
The syntax is pretty easy to read there - it'll start off with 2 new users a second and add more every second until at the end of the test it's adding 10 new users a second.
That might not seem like many, but remember that our test takes a few seconds thanks to the pause in the middle. You can always increase this value and re-run the test later - or decrease it if you manage to kill your server!
Now we're set up - it's time to run the test!
Running The Test And Reading The Results
Running the test is really easy.
Just go to the "bin" folder and run either "gatling.bat" (Windows) or "gatling.sh" (Mac).
A command-line window will open:
Option [0] should be your recorded test. Type "0" then press enter, then press enter through the other options - and the test will start.
You can watch the progress of the test in the command-line window. If you want to monitor your server with other tools too, that's probably a good idea! Here's my test server's realtime output along with a "top" window and the Gatling monitor during a test:
Once it's done, the Gatling window will give you the name of a folder containing the results. Just navigate to that folder (under "results" in your Gatling folder), open "index" in your browser, and you can view all the results of your test, in aggregate or at each step of the way:
It's also worth having a look at your tracker's results if you were testing a tracker: it should have correctly interpreted the useragent string (your browser, OS, and so on) and should also have picked up any traffic source parameters you sent through, as well as geo-located you accurately.
And that's it - your first test is complete and you now have info on whether your server can handle the load you put it under!
More Things Gatling Can Do
Gatling has an enormous number of options if you want to extend your test, some of which are extremely useful for affiliates:
- You can run multiple different simulations at the same time - so if you want one set of users who arrive at your lander then quit, another set that play with some of the options then click through, and a third set that fill in details on the page, you can do all of those at the same time and see how the server handles the load - and if any of those tasks slow it down in particular.
- You can send multiple useragent strings or query strings, and you can randomise parts of them or simply send random pre-determined ones. That's particularly useful if your tracker has efficient caching of requests and you want to see how it copes with real-world input. Some trackers might cope fine with a test that just sends the same browser, OS and model over and over, but will crap out when presented with dozens of different handsets, browsers, OSes and so on.
- You can run Gatling on remote servers and on multiple servers at once. There's nothing stopping you from running it on 10 different IPs to stress-test a single server, then collating the results.
And that's it! I hope that was helpful - if you have any questions or comments, let me know below!
Epic tread.
NO wonder ad networks have such mega problems finding ad fraud, tools like this are too smart and act like real surfers.
Awesome share. Thanks.
Whoa ... what a post!
If the goal is ultimately to measure a server's ability to handle load, why use gatling?
What makes gatling superior to buying ZeroPark Global RON pop traffic for $0.20 CPM?
@jennatalia - mostly the cost. Running 5-million-visitor tests repeatedly on a server can get expensive even at super-cheap pop prices, wheras it's not very expensive to run even 20 or so Digital Ocean instances around the world to stress-test the server.
@cmdeal, @kash50 - thanks!
@iamattila - Yeah, browser simulation tools are getting very smart. Some of the stuff out there that's more AI-based is pretty terrifying.
Thanks Caurmen, I've just ran my first test with Gatling.
I really like their recorder as it makes testing complex sequences extremely easy.
Just wanted to add the following piece of code for those who want to clear the cookies between each request and send random strings to their tracking tockens:
val rdm = new scala.util.Random(31)
val scn = scenario("NameOfYourScenario")
.exec(flushCookieJar)
.exec(http("your_request_name")
.get("/your_url_path")
.queryParam("your_token", _ => rdm.nextString(10)))
Something like your_domain/your_url_path/?your_token=Qej23ejkds
@vitavee - Nice! Yeah, Gatling's really impressed me since I've been using it with just how customisable it is - their scripting in particular is superb.
If anyone else has any Gatling recipes that fellow affiliates might find useful, please do post 'em here!
Home > Technical & Creative Skills > Programming, Servers & Scripts