Sunday, June 18, 2006

RedKernel is Backasswards

This isn't a new bot but what they did made me laugh out loud: - "GET / HTTP/1.1" RedKernel WWW-Spider 2/0 (+" - "GET /robots.txt HTTP/1.1" "RedKernel WWW-Spider 2/0 (+" - "GET / HTTP/1.1" "RedKernel WWW-Spider 2/0 (+" - "GET /robots.txt HTTP/1.1" "RedKernel WWW-Spider 2/0 (+"
That's right, they asked for the home page and then robots.txt, in reverse order, twice in a row.

Why would you ask for the home page and they check to see if my site permits your bot to crawl?

Do you think you're just entitled to the home page regardless?

How bizarre as their website doesn't even mention robots.txt:
REMOVE a website from our link directory:

Our www-spider (= a bot like googlebot) is an automated crawler/indexer.
So it works with meta names like all other bots.

You just have to insert in your meta names: (only in your index)
<meta name="RK_WWW_Spider" content="noindex">

Then just wait the next crawl of your website. The meta name will be detected and your site will not be indexed. It can takes few months before your site is crawled.
Meta tags?

What a joke, say it boys and girls "block! block! block!", now isn't that better?

1 comment:

MP1 said...

If you use the meta tag they want you to, then its a whole lot easier to find the sites on the internet - the ones you want to spam or scrape! :))