Home > Affiliate Marketing Forum >

Protect your Nuts - Part 2 of 3 (33)

01-26-2012 12:13 PM #1 tijn (Moderator)
Protect your Nuts - Part 2 of 3

If you havent read it, go over to part 1 and check that out first!

Here is part 2. The first 5 steps you need to take. (Go to Part 3)

1) Switch domains regularly

First thing to realize is that at some point one of your domains is gonna get indexed, spied on, or bid over.

You will not stop this.

So just be prepared and setup your sites & hosting in such a way that its easy for you to swap your domains out.

Don't invest in expensive domain names, like I did with my first tracker domain (gr0w.co cost me like $20....silly me).

Unless of course your building a long term asset and need the brand recognition.

Here is my own internal checklist:



Of course all this changing can turn into a royal pain.

Especially if, like with CPVLab, your licence is tied to a single domain.

Don't worry though - CNAME redirection to the rescue.

For example, say I want to use a throwaway domain instead of my compromised t01.gr0w.co domain, which hosts one of my trackers (in this case CPVLab).

All i do is setup a CName redirect in my DNS records for the domain I want to use. Here is an example of how I did that at Namecheap.



This simply says, when someone accesses my subdomain (s01.gov-new.info) just show them domain t01.gr0w.co.

Then you just need to make sure your server accepts that new domainname.

You do this by changing the virtual host setup on your webserver to make sure the new domain is associated with your old domain.

If your not sure how, dont try to do this yourself as you will break stuff. Just get your hosting provider to add the new domain to the config.

On NGinx you use server_name directive and on Apache use ServerAlias

You will need to restart apache/nginx so the new config is loaded.

This will basically redirect any request from s01.gov-new.info to t01.gr0w.co
The best thing is that the domain visible to your visitors is the new one!!



That's all there is to it!



2) Remove spying toolbars

Toolbars are evil! Dont trust them.

Especially Alexa.

It will swipe links from password protected pages served with https and then send the bots over to take a look!

So uninstall those buggers.

Also...

Switch to Google Chrome, and use its multiple users feature.

I wrote about this a couple of days ago. Its the simplest way I know off to managing your ‘multiple personalities’ without leaving a massive footprint.

Make sure you install an extension to swipe your cookies and Flash LSOs though!

01-26-2012 12:13 PM #2 tijn (Moderator)

Had to break this post up into multiple replies cause it was too big!



3) Try to stop your pages from getting indexed

The reason I say try, is that its very very hard to do. Especially if your not in control of the links to your tracker, campaigns and landers.

From google:

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.
PPV is easier cause you control the path. PoF and Facebook are ok as well because the actual link in your ads is from the site itself, not your own link.

Media buying is the most challenging.

Right now I use 5 methods to reduce the risk of stuff getting indexed that shouldn't.

robots.txt

On all my domains that I don’t want to get indexed, I ensure I have a very strict robots.txt file that excludes all robots from crawling my sites:

Code:
User-agent: *
Disallow: /
IMPORTANT: Make sure you save this file to the root of your domain!

For more information about robots.txt, check out these two links:
http://www.robotstxt.org
http://support.google.com/webmasters...&answer=156449

You can test whether your robots.txt file is working using Google Webmaster Tools (http://www.google.com/webmasters/), or use the following site:

http://phpweby.com/services/robots


stop links from being followed

Google and other search engines will still index your site if someone is linking to it, even if you disallow robots in your robots.txt.

So where you have control over the HTML of a page, and your linking to your landers, tracking domains and offers, always include the rel=”nofollow” tag in the links.

Here is an example:

Code:
<a href=”http://thisisashithotoffer.com/?affid=2938498&subid=lkajdflk” rel=”nofollow”>Click Here And Make Me Rich</a>
Here is a detailed description about the rel=”nofollow” tag on wikipedia:
http://en.wikipedia.org/wiki/Nofollow


Meta tags

Problem with robots.txt is that it has leaks, especially when other pages on the web link to your site and they dont include the rel=”nofollow” tag.

So as an extra layer of protection, make sure you set the relevant meta tag (bold & green below) in your html head section:

Code:
<html>
<head>
<title>Your page title</head>
<meta name="ROBOTS” content="NOINDEX,NOFOLLOW,NOARCHIVE,NOSNIPPET">
</head>
....
....
Here is a short description of the values you can use in your meta tag, courtesy of searchengineland:

NOINDEX – prevents the page from being included in the index.
NOFOLLOW – prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
NOARCHIVE – prevents a cached copy of this page from being available in the search results.
NOSNIPPET – prevents a description from appearing below the page in the search results, as well as preventing the cache link from showing
NOODP – blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
NOYDIR - tell Yahoo to not use Yahoo Directory information to make a title and/or description
NOINDEX, NOFOLLOW,NOARCHIVE are the key ones.

Further reading:
http://searchengineland.com/meta-rob...ges-more-10665
http://yoast.com/articles/robots-meta-tags/


No indexs header

Now the downside with the above Meta Tag method is that for certain URLs you are not sending an actual html page.

Like with your tracking links, its just a redirect.

So the third and final method is to send a “X-Robots-Tag” header.


PHP

You can do this on a file by file basis, by including the following PHP code:

Code:
header("X-Robots-Tag: noindex, nofollow, noarchive", true);
But a better and safer method, is to get your webserver to send that header automatically.


Apache

For apache, this is easy because you just create a .htaccess file in the relevant folder, and include the following in that file:

Code:
Header set X-Robots-Tag "noindex, noarchive, nosnippet"
Or if you only want the header set for certain file types:

Code:
<FilesMatch "\.(htm|html|php)$">
Header set X-Robots-Tag "noindex, noarchive, nosnippet"
</Files>

Nginx

In the case of NGinx its a little more difficult. You will need to change the actual server config.

First you need to make sure Nginx has been compiled with the HttpHeadersModule (with later versions this is the case for the default setup).

http://wiki.nginx.org/HttpHeadersModule

Then you need to include the following configuration line either in your main nginx config, or in the config of your virtual domains:

Code:
add_header X-Robots-Tag "noindex,nofollow,noarchive";
More information about customizing headers with nginx:
http://www.cyberciti.biz/faq/nginx-s...-http-headers/


LightHttpd

And finally if your webserver is LightHttpd (as is the case with the default BeyondHosting setup), you need to include the following line in your webserver config:

Code:
setenv.add-response-header = ( "X-Robots-Tag" => "noindex,nofollow,noarchive" )
More info about sending headers with LightHttpd can be found here: http://www.xarg.org/2008/07/add-http...s-in-lighttpd/


Here is some further reading that you might find helpful:
http://yoast.com/x-robots-tag-play/
http://sebastians-pamphlets.com/stan...xt-directives/
http://www.seomoz.org/blog/12-ways-t...search-engines
http://perishablepress.com/press/200...-x-robots-tag/
http://googleblog.blogspot.com/2007/...with-even.html

01-26-2012 12:13 PM #3 tijn (Moderator)

4) Request URL removal

Of course once your site is indexed you can get it removed.

This does take time. But its becoming a lot easier these days.

For google you just submit a removal request through the Webmaster Tools.
http://support.google.com/webmasters...answer=1663427

For bing, submit this form or go through the the Bing webmaster tools:
https://support.discoverbing.com/efo...&wfxredirect=1

For yahoo, just follow the approach for Bing as its bing results.

Also make sure you take all the steps under 3!



5) Only show your pages to expected visitors!

Also known as cloaking.

There are many solutions available here, from the simple to the advanced.

Here are two simple scripts you can use to do some basic filtering.


User Agent

[php]
<?php
$ua = $_SERVER['HTTP_USER_AGENT'];
$ua_block ='bot|proxy|scrape'; //add footprints to block, separated by | (pipe) symbol

if ( preg_match("($ua_block)i", $ua, $matches) ) {
//match detected, send off to somewhere else
header('Location: http://domain.com/fakepage.html');
die();
}
?>
[/php]

Use this simple php script at the start of your landers to redirect bots and other undesirable visitors elsewhere.

Here si the best site with User Agent strings for bots and the like:
http://www.botsvsbrowsers.com/


Referrer Checking

The second easy thing you can do is to check the referrer string for an expected token. In this particular example, im using the referred token from CPVLab, assuming you include the code below on your landing page.

In the case of CPVLab, this is these are the pages accessed:

Code:
base.php -redirects to-> baseredirect.php -redirects to-> your lander URL
Therefore, on your lander, the referrer will be baseredirect.php.

[php]
<?php
$ref = $_SERVER['HTTP_REFERER'];
$ref_ok = 'baseredirect.php';
if ( preg_match("($ref_ok)i", $ref, $matches) == false ) {
//no match - redirect
header('Location: http://domain.com/fakepage.html');
die();
}
?>
[/php]

There are many variations to this that you can use depending on your tracker, campaign setup and traffic source.



Ok thats the end of Part 2.

Part 3 will follow tomorrow!!

01-26-2012 01:22 PM #4 bbrock32 (Administrator)

Pure gold! Thanks.

There is a lot of things I hadn't thought of and will implement on my domains.

01-26-2012 05:14 PM #5 ppchound (Member)

Top posting again tijin. MANY thanks for these nuggets.

When it comes to STM 'thanks' you're making it a one horse race

01-26-2012 05:37 PM #6 godspeed (Member)

Great stuff, thanks

01-27-2012 05:17 AM #7 tijn (Moderator)

thx guys! ill be posting the final part 3 later today. wont be as huge, but will be as powerful.

anything specific that anyone is looking for, ie a problem or issue to resolve related to this topic?

Tijn

01-27-2012 10:37 AM #8 topgun (Member)

just wondering why is that? thanks!

use sub-domains for each campaign rather then sub-folders

01-27-2012 11:30 AM #9 tijn (Moderator)

google is less likely to link subdomains.

so if one campaign is indexed it doesnt necessarily show the other campaigns when someone does a site: search in google.

with folders, your more likely to show up.

01-27-2012 03:33 PM #10 obolus (Member)

Great post, thanks!

Quote Originally Posted by tijn View Post
CNAME redirection to the rescue.
I'm no DNS expert, but why would you use an alias from the new domain to the old domain instead of pointing the new domain to the actual IP address of the server? This should use less resources and the fact that the old domain completely disappears is probably also rather good than bad. Do I miss something?


Quote Originally Posted by tijn View Post
google is less likely to link subdomains.

so if one campaign is indexed it doesnt necessarily show the other campaigns when someone does a site: search in google.

with folders, your more likely to show up.
As far as I know Google shows all subdomains if you search for "site:domain.com". Only if you search for "site:subdomain.domain.com" subdomain2.domain.com wouldn't show up. Did you observe something else?

01-27-2012 08:40 PM #11 Mr Green (Administrator)

Dang nice write up tijn. You know how to take care of your nuts.

06-26-2012 09:00 PM #12 profitable ()

Quote Originally Posted by tijn View Post
[B]Referrer Checking

The second easy thing you can do is to check the referrer string for an expected token. In this particular example, im using the referred token from CPVLab, assuming you include the code below on your landing page.

[PHP]<?php
$ref = $_SERVER['HTTP_REFERER'];
$ref_ok = 'baseredirect.php';
if ( preg_match("($ref_ok)i", $ref, $matches) == false ) {
//no match - redirect
header('Location: http://domain.com/fakepage.html');
die();
}
?>[/PHP]
^^ is there a way to check for multiple referrers? The code you posted works fine, but i'd like to add multiple referrers that are ok to see real page.

I tried pasting another line:
$ref_ok = 'baseredirect.php';
$ref_ok = 'page2.php';

but doesn't seem to work.

06-26-2012 11:41 PM #13 zeno (Administrator)

Hmmm could you just use the or operator inside the if() part? There must be an elegant way to do it using an array or something similar, but this might work?

[PHP]
<?php
$ref = $_SERVER['HTTP_REFERER'];
$ref_ok = 'baseredirect.php';
$ref_ok2 = 'second.php';
$ref_ok3 = 'third.php';
if (preg_match("($ref_ok)i", $ref, $matches) == false or preg_match("($ref_ok2)i", $ref, $matches) == false or preg_match("($ref_ok2)i", $ref, $matches) == false ) {
//no match - redirect
header('Location: http://domain.com/fakepage.html');
die();
}
?>
[/PHP]

May or may not need extra brackets around each preg_match....false bit.

06-27-2012 04:51 PM #14 tijn (Moderator)

Yeah just add them to the string separated by the pipe symbol:

[php]
$ref_ok = 'baseredirect.php|site2.com|site3.com';
if ( preg_match("($ref_ok)i", $ref, $matches) == false ) {
//no match - redirect
header('Location: http://domain.com/fakepage.html');
die();
}
[/php]

this works because the string is basically a regular expression

06-27-2012 11:54 PM #15 zeno (Administrator)

Oh yeah that makes way more sense, thanks Tijn. That should work in pretty much every situation where you're using preg_match right? E.g. listing subids separated by | that you want redirected somewhere specifically.

06-28-2012 10:03 AM #16 tijn (Moderator)

Yep. Preg match is great.

Also - when you have like 10 or 15 strings you want to check, and lots of volume, preg_match is way faster then a array compare

06-29-2012 07:02 AM #17 vilka9 (Member)

Wow thank you for this information. There seems to be more and more details I get exposed to in the AM world.

06-29-2012 07:02 AM #18 vilka9 (Member)

Where can you find more information about cloaking in general?

06-29-2012 09:29 AM #19 tijn (Moderator)

do a search on this forum....but....its a fine art and a risky game!

07-10-2012 06:08 PM #20 kranium (Member)

Hi, I know this is an older post but couldn't you do something with .htaccess as well to thwart the bots? :


Code:
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^googlebot
   RewriteRule ^(.*)$ http://google.com/
I wonder if you could even get fancier and make a cron job for a script that does a reverse lookup of googlebot's IP address(es) every day or so and dynamically inserts them into your .htaccess and using a flat out IP deny? hmm...

10-03-2012 10:54 AM #21 Majuba (Member)

Hi tjin, I used your CNAME redirect method to get an alternate CPV Lab tracking URL. This worked initially but now I'm getting the error: "The maximum number of domains for this application has been reached" when I try to login tusing the second domain.

Does this still work for you? If not which method do you use?

05-05-2013 04:34 PM #22 karim0028 (Member)

I also have a question...

Do most folks create a separate sub domain for each campaign? So, if your doing Adult, for instance you would do differnt subdomains

like

asian.myadult.com/lp1/index.php
black.myadult.com/lp1/index.php
cheaters.myadult.com/lp1/index.php

etc, etc? That seems hard to keep track of, no?

05-06-2013 09:52 AM #23 caurmen (Administrator)

Do most folks create a separate sub domain for each campaign?
I don't think so - I certainly don't. You can just put separate campaigns in separate folders on your LP server - www.myadult.com/asian/ , www.myadult.com/black/ etc.

05-06-2013 10:03 AM #24 karim0028 (Member)

So, but isnt one of the recommendations to put campaigns on seperate subdomains to "protect yo nutz!" ;-)

05-06-2013 11:17 AM #25 zeno (Administrator)

Putting things on a separate subdomain doesn't really do anything to protect you from anything... In fact linking your campaigns/verticals together so obviously is a bit silly.

03-02-2014 03:13 PM #26 Philwil (Member)

Looking for the best way to use my CPVlab/cloak on FB, don't want to buy multiple CPVlab licenses.

So using CNAME is basically the best way to use CPVlab/trackers on FB?

Each of my FB accounts has a different domain, so I use CNAME to link CPVlab to those domains, that way I can track and use cloaker right?

Been noobing around too much concerning tracking.

03-02-2014 10:25 PM #27 zeno (Administrator)

Yes CNAMEing should work fine in terms of licensing and masking your CPVlab domain - as long as you aren't using https tracking links. Be aware that it is trivial to look up the DNS resolution pathway for a domain to find out where it really goes, so you're hiding in plain sight so-to-speak.

03-03-2014 01:57 AM #28 Philwil (Member)

Cheers Zeno, thanks for answering all my questions so far!

Tracking atm in a tedious way since I can't use my tracker domain in FB.

Il try the CNAME, since its the only method I can find.

03-03-2014 04:43 AM #29 zeno (Administrator)

Why can't you use your tracking domain? If it's blacklisted have you tried using intermediary PHP scripts to bounce traffic to your tracker?

03-03-2014 05:25 PM #30 Philwil (Member)

Yeah I want to keep each account unique with a different domain, so looking for the best way to cloak my tracker for FB. Havant tried intermediary PHP scripts though, not sure what it implies. Too techy stuff
Needs to go from: domain>tracker>cloaker>LP>offer

03-03-2014 11:46 PM #31 zeno (Administrator)

CNAMEing would be the fastest way to do this. Seriously... just buy a $4 blahblah.pw domain name and CNAME it to your tracker domain. I'm unsure if FB sniffs the headers to look for blacklisted URLs but you'll certainly find out.

10-16-2015 11:37 PM #32 TheComedian (Member)

Quote Originally Posted by tijn View Post
4) Request URL removal


Code:
base.php -redirects to-> baseredirect.php -redirects to-> your lander URL
Therefore, on your lander, the referrer will be baseredirect.php.

[php]
<?php
$ref = $_SERVER['HTTP_REFERER'];
$ref_ok = 'baseredirect.php';
if ( preg_match("($ref_ok)i", $ref, $matches) == false ) {
//no match - redirect
header('Location: http://domain.com/fakepage.html');
die();
}
?>
[/php]

There are many variations to this that you can use depending on your tracker, campaign setup and traffic source.
I'm trying to use this approach with Voluum urls but I'm losing the parameters while doing this, is there a way to lose the referer and at the same time maintain the parameters we send through Voluum in order for them to reach the lander?

thank you!

08-30-2016 02:08 PM #33 jessejames (Member)

Not sure if this has been mentioned. But don't use robots.txt along with noindex header tags. Just use the header tags, or x-robots tags in htaccess/nginx-config.

Google will index the site, but with missing meta data, since they can't parse the page (as robots disallow it), they will never see the no-index header tags, and won't know that you tell it to no-index.

Home > Affiliate Marketing Forum >