This is just a quick FYI with something we have noticed with some of the toolbars out there that you may or may not have installed on your browser.
They track your pages and then send bots back over your links.....(hurr durrr you purple unicorn)
But....Let me explain a bit more and you will understand the risk....
say you have the alexa toolbar installed...it pulls all the links form whatever page your on....then later alexa sends bots to down check every link it saw....I believe that google and bing do something similar as well.
What does this mean to me and why the warning?
Well a couple of things since 202 and cpvlab are web apps guess what....they send bots to crawl all over your trackers....even with no index in the robots file...
we have found alexa to be very nasty about this totally ignoring all rules...
So you stand a very high chance that wacky shit from your tracker is gonna get indexed...stuff you prob don't want public. The less people know about your tracker the better.
I would highly recomend that if you have any of these toolbars installed you don't use them at all for accessing any trackers you may use.
Alexa's Clickstream is possibly one of the most useful tools anyone could use as an affiliate. With that being said, cover your tracks like polarbacon said!
oh F me. this explains alot. thanks PB
hmmm time to uninstall some shiaat.
What if not you but an end-user has such a tb installed, it will still sniff all the links and send along their bots. Really nothing you can prevent.
in short - yes!
alexa in this particular case ignored the robots.txt rules / noindex / nofollow meta tags and headers and accessed URLs that were behind a password protected page. Ie these links were only displayed on a site after a user had logged in.
we checked the server logs and could see the alexa bot visiting each of the URLs of the links on the page.