Saturday, April 15, 2006

Alexa Hiding in the Shadows

What the hell's going on with Alexa trying to crawl my site with NO user agent string!

Had an attempted anonymous crawl from which is:

Alexa Internet ALEXA-INTERNET (NET-209-237-237-0-1) -
They still read the robots.txt file, but no clue who it was: - - "GET /robots.txt HTTP/1.0" 200 111 "-" ""
What the hell's going on Alexa?

Someone break the crawler or you just trying to sneak under the radar after everyone blocked your ass?

You claim to crawl as "ia_archiver" but I don't see any ID here whatsoever!

Please explain, inquiring minds want to know!


Anonymous said...

I caught them yesterday and it turns out that they've cralwed over 3k pages in the previous two days. .

I have a robots.txt disallow for their bot but the "no UA" crawler never looked at it.

Now its a "deny from"

IncrediBILL said...

BTW, after looking closer I discovered the hidden Alexa has been visiting almost daily!