Home > Technical & Creative Skills > Tracking Campaigns

Advanced: How To Remove Tracking And Still Track Anyway (10)

06-12-2015 03:49 PM #1 caurmen (Administrator)
Advanced: How To Remove Tracking And Still Track Anyway

One well-known hack to pull a bit more ROI out of your campaigns, particularly mobile and pops, is to remove your tracking once you have the campaign dialled in, and just run traffic straight to the lander.

That means that you get a faster response time from your campaign, which reliably increases ROI, often by quite a lot. But on the downside, you can't optimise very effectively if at all - you're running blind.

What if you could get all the data you'd normally expect from tracking, keep split-testing landers, but also get the speed of just running traffic straight to the lander with no redirects or other code?

Well, you can. And this post shows you how.

What You'll Need To Use This Technique

You know how my tutorials tend to say "don't worry, anyone can follow this!"?

Well, this post ain't like that. This is definitely an advanced tutorial, and I'd expect that experienced affiliates with reasonably large-scale campaigns will be the ones who will get the most out of this technique.

If you're just starting out, I strongly recommend that you do not mess with this stuff unless you're already extremely techie. You have bigger fish to fry and rolling your own complex non-standard tracking systems at an early stage is a really big distraction.

On the other hand, if you're experienced and looking for an edge, and particularly if you have a profitable campaign or five that you'd like to maximise ROI on, this post is for you.

You will need to have or be a sysadmin to implement this approach, and there's a certain amount of programming and/or database work involved. Experienced affs should be able to hire someone on oDesk to implement most of this in a couple of days. Porting the stats into your tracker of choice is likely to be the hard bit.

This technique is currently only usable on a single server, not a CDN. There's probably a way to get it working with at least some CDNs, but I haven't got there yet.

You're also going to need a new server or an existing server dedicated to just one set of campaigns for this trick, because it involves recompiling your Web serving software.

Finally, your traffic source must be able to send a unique parameter with each click. On Decisive, this would be Bidhash. On Zeropark, it would be cid. Etc, etc. If you can't get a unique parameter from your traffic source, this approach won't work without some additional hacking.

So How Does This Miracle Work?

Web servers log all the pages that they serve to "log files". This data can be set to be comparable to the data that your tracker collects, albeit in a much less useful form. However, because your web server isn't written in a slow language like PHP and isn't talking to a database, the log file writing process is much, MUCH faster than tracking.

When you remove tracking and use just an "untracked" server, that server is almost certainly still writing these log files.

Back in the day before Google Analytics all web analytics was done by having programs go through those files, extract the relevant data and present it in a usable form. What we're doing here is going back to that approach.

Specifically, we're going to use the web server "nginx". Nginx is well-known for being incredibly fast and stable at serving "static" files (no PHP, just Javascript and HTML). It's no exaggeration to say that a 512Mb server running nginx can happily cope with a million clicks a day with double-digit millisecond response times.

We're also going to use a couple of little-known features of nginx to allow us to continue split-testing landing pages, as well as allow us to redirect to offers much faster than we usually can with tracking.

Technical Summary Of My Approach

We recompile nginx to allow us to use the Random Index function ( http://nginx.org/en/docs/http/ngx_ht...ex_module.html ), enable this, and also enable Server-Side Includes ( http://nginx.org/en/docs/http/ngx_http_ssi_module.html ) on our server.

Nginx SSIs are way more powerful than most people realise, and are key to doing this efficiently.

We put all the landers we want to split-test in a single subdirectory under our web directory. The Random Index function will rotate between them randomly whenever someone arrives at the URL of that subdirectory.

Then we set up a custom log file format for nginx, which takes all the GET parameters from the incoming URL and stores them as comma-seperated values - IP,date,placement,device,carrier,os,unique_click_i d,lander, for example. We use nginx's $uri variable to pull the lander filename so that we know which of the landers that we're split-testing was used.

We add the unique click ID to the end of our offer URL as a GET parameter in the landing page by using nginx SSI variable echoing ( http://nginx.org/en/docs/http/ngx_ht....html#commands ) and send clicks to /offer/ on our server.

Then we set up an nginx rewrite redirect ( http://nginx.org/en/docs/http/ngx_ht...te_module.html ) to redirect /offer/ to our offer URL. That means that clicks to the offer are logged, with the unique click ID we added above, to our custom tracking log.

We add our URL to our traffic source, using the GET arguments we've already set up in the custom log format, and direct it at the subdirectory where our landers are stored.

Now we're capturing all the information we need for tracking. All that remains is to take our tracking log and load it into our tracker of choice. I demonstrate a simple way to load it into a SQLite database and query it for basic tracking info below, but it wouldn't be much harder to load the data into Prosper, CPVLab, Thrive, etc. Loading it into Voluum would be harder, and would probably involve some kind of fake traffic (unless there's a way to mass-import data - Robert?).

Visitors click through our ad and are sent directly to a randomly-chosen landing page. Meanwhile, nginx logs all their details including their unique click ID. When they click through to the offer, that clickthrough is registered as a click on the /offer/ page, again with their unique ID, and they're sent off to the offer page. Finally, when they convert, their unique ID is registered on the affiliate network as their subid, and we can load that data into our database at a later date.

Step-by-step How To Do It

If you're hiring someone, this is the bit to tell them to follow!

Step 1: Compile Nginx

On your server, compile nginx from the latest source with the --with-http_random_index_module flag set. Here's the guide I follow when I'm compiling nginx on Ubuntu: https://www.rackspace.com/knowledge_...nx-from-source . Note that you'll need to install a whole pile of dependencies if you're doing this on a fresh server, particularly given you're using http_random_index_module - I started with apt-get update and upgrade, then installed make, build-essentials, PCRE and zlib1g-dev . After that it successfully compiled.

Step 2: Configure Nginx

You'll need to enable random-index and ssi in your location section of your nginx config:

Code:

ssi on; 
random_index on;

Now, create a custom log format. I did this in the http block, but you could do it in the location block too. Exactly what this log format contains depends on what you want to log and what parameters your traffic source allows you to send.

Nginx allows you to log any GET param with the $arg_PARAMNAME variable, so include all the variables you want using that. You'll also want to log datetime, IP, and $uri (the landing page that random-index chooses).

Here's the log file format I ended up with for a Decisive campaign:

Code:

log_format tracking  '$remote_addr,$time_local,$arg_placement,$arg_model,$arg_carrier,$arg_OS,$arg_bidhash,$uri,';

Note the trailing comma - it's important if you want to use this as a CSV.

Next, you'll need to enable that log format. I decided to run a seperate log as well as the main access.log:

Code:

 access_log  logs/access.log  combined;
 access_log logs/tracking.log tracking;

Finally, you'll need to enable the offer redirect. In your location block, add

Code:

         rewrite ^/offer/ OFFERURL;

OfferURL will need to include the unique click ID - add that into the offer URL at the appropriate place for your network with $arg_uid .

If you want to run multiple campaigns on this server, you can use /SUBDIRECTORY/offer/ instead of /offer/ .

You'll also want to set up log rotation depending on how much traffic you're expecting.

Step 3: Prepare Landers

On each of your landers, use the following line as your clickthrough URL:

Code:

/offer/?uid=<!--# echo var="arg_UID_ARG_NAME" -->

You'll need to replace UID_ARG_NAME with whatever your unique click ID GET parameter name is. In my Decisive test I used "bidhash".

Upload all the landers to the subdirectory you intend to use for this campaign. Visit that subdirectory a few times in a web browser and verify that it's randomly showing one of your landers each time you do.

Step 4: Set Up Campaign

Now you'll need to set up your campaign on your traffic source.

You should direct traffic to the subdirectory that your landers are hosted in, with no page after the trailing slash. If you're using a directory called "Marvel_Superheroes", for example, the base URL should be http://www.yourserver.com/Marvel_Superheroes/

To that, add your query string. The name of each of the parameters should be exactly the same as the names you gave them in Step 2 (minus the $arg_ , obviously). The values should use your traffic source's tokens - {{{bidhash}}} for Decisive's UID, for example.

Once the campaign is set up and running, it's wise to monitor the output of your tracking log for a while to make sure it all looks right.

You can do that with tail -F LOGNAME to see the data coming in in real-time. It's quite cool, for very geeky values of "cool".

Step 5: Getting The Data Out

Now all you need to do is take your logfile and slurp it into the tracking tool or database of your choice.

Doing that for all possible trackers is a bit outside the scope of this article, but generally you'll want to go through 4 steps:

1) Use a tool like Ruby's CSV parser to convert your log into a format that's easily insertable into your tracker's database. SQL files are probably best. Strip /favicon and /offer/ hits out at this stage but don't lose the original file - you'll need it for step 3.
2) Upload that data into your tracker.
3) Use the /offer/ hits to flip the "clickthrough" bit on your tracker of choice for all those unique IDs. Any hit on /offer/ should be treated as a clickthrough for the UID associated with it.
4) Upload conversion data from your affiliate network as normal.

For my first testing on this approach, I just used a very simple SQLite database with one table with the following schema:

Code:

CREATE TABLE tracking(ip text, time text, placement text, moodel text, carrier text, os text, bidhash text, lp text, clickthrough bool, converted bool);

That meant I could then slurp the data directly in with

Code:

.separator ","
..import tracking.log tracking

and then update clickthroughs and conversions with a similar SQL query - here's the clickthroughs one:

Code:

update tracking set clickthrough="true" where bidhash in (select bidhash from tracking where lp="/offer/");

Then I just queried the database directly to get information on what was going on: checking clickthroughs for my #2 lander, for example:

Code:

select count(*) from tracking where lp="/index2.html" and clickthrough="true";

Obviously, for anything larger than a test campaign I'll just shove that data into my tracker (which uses a custom format very similar to the click format above). But for now it was good enough that I was able to optimise my test campaign, finding that lander #2, with a nice big clear image, was actually heavily outperformed by lander #1, which was text only - surprising given that what I was advertising was a video!

I hope that was useful! If there was anything unclear, or you have any other comments or points you'd like to raise, post below!

06-12-2015 10:08 PM #2 vortex (Senior Moderator)

Another epic thread Caurmen! Will you EVER run out of these brilliant thread ideas?

Thanks so much for this! Bookmarked!

Amy

06-13-2015 08:17 AM #3 h0mp (Member)

I've been using something similar because there weren't any mobile trackers back in the day.

By this time it has evolved into a monstrous beast that laughs in the face of tracking URLs.

06-13-2015 09:52 AM #4 caurmen (Administrator)

@vortex - Can't speak for "brilliant", but I've got a good list - many requested by STM users - to be going on with Thanks - glad it's useful!

@homp - cooool. Would you be up for sharing any of the evolutions and improvements it's gone through from this basic idea? I'd be very interested to hear them!

06-15-2015 08:16 AM #5 cbrughmans (Member)

Running a campaign without tracking is like driving a car blindfolded - sooner or later you'll crash. Seeing the results of your campaign is absolutely necessary to drive ROI, optimize the campaign and scale it up. I would always recommend to fully track any campaign before launching it

06-15-2015 10:09 AM #6 caurmen (Administrator)

@h0mp - that's seriously fascinating - thanks. The peasant advertiser detector is a particularly good idea, and one I might have to look into coding myself...

@katim777 - Ouch! Yeah, that's definitely too slow, particularly for pops. I'd recommend looking into some different CDNs or hosting on a single server there.

If you reduce loading time, I almost guarantee that your CTR will go up, probably by quite a bit. Here's a case study I did a while ago on loading times for pops: http://stmforum.com/forum/showthread...hlight=loading

06-15-2015 10:58 AM #7 memori (AMC Alumnus)

Nice Caurmen,

Have read conversation about this and adding on my whishlist
but you already brought it here!

Thanks

06-15-2015 11:41 AM #8 kepe95 (Moderator)

Amazing guide!

I did a lot of tests with different redirects, and the amount of clicks lost on mobile through redirects is incredible. (and I'm not only talking about those you're in control of)

@h0mp : Wow thanks for the details , looks great - have you tried any split-tests to see how better your systems performs compared to for example Voluum or Thrive ?

06-17-2015 11:29 AM #9 caurmen (Administrator)

@katim777 - hmm, that's a serious issue. Could you start a separate thread on it so we can go into more depth?

@h0mp - YIKES. Oh, advertisers, why do you try us so? Nothing more annoying than working to get a super-slick, super-fast lander and then have the advertiser make a design change that adds 15s to their offer load time...

02-24-2016 03:31 PM #10 taewoo (Member)

This is some silicon valley shit right here...

Home > Technical & Creative Skills > Tracking Campaigns