This isn't a new bot but what they did made me laugh out loud:
66.55.143.162 - "GET / HTTP/1.1" RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)"That's right, they asked for the home page and then robots.txt, in reverse order, twice in a row.
66.55.143.162 - "GET /robots.txt HTTP/1.1" "RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)"
66.55.143.162 - "GET / HTTP/1.1" "RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)"
66.55.143.162 - "GET /robots.txt HTTP/1.1" "RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)"
Why would you ask for the home page and they check to see if my site permits your bot to crawl?
Do you think you're just entitled to the home page regardless?
How bizarre as their website doesn't even mention robots.txt:
REMOVE a website from our link directory:Meta tags?
Our www-spider (= a bot like googlebot) is an automated crawler/indexer.
So it works with meta names like all other bots.
You just have to insert in your meta names: (only in your index)
<meta name="RK_WWW_Spider" content="noindex">
Then just wait the next crawl of your website. The meta name will be detected and your site will not be indexed. It can takes few months before your site is crawled.
What a joke, say it boys and girls "block! block! block!", now isn't that better?
No comments:
Post a Comment