The next trick up my sleeve in the war against scrapers and badly behaved bots is to cloak NOFOLLOW tags into the pages when it's not a major search engine requesting the page..
Basically the concept is by changing the home page, site map and site wide navigation links to contain NOFOLLOW on all major links it will stop legitimate bots at the front door and they will be allowed to crawl a few pages by default. However, any crawler that follows links flagged with NOFOLLOW will get the hammer dropped on their ass.
One added touch is there will be a few spider-trap pages that always have NOFOLLOW on them, never allowed to be crawled by any bot, just like the current robots.txt file has some traps in it as well.
Building a better spider trap, how far will this obsession go?
Wednesday, January 11, 2006
NOFOLLOW Straight To Jail, Do Not Pass Go
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment