What the hell's going on with Alexa trying to crawl my site with NO user agent string!
Had an attempted anonymous crawl from 209.237.238.224 which is:
whois 209.237.238.224They still read the robots.txt file, but no clue who it was:
Alexa Internet ALEXA-INTERNET (NET-209-237-237-0-1)
209.237.237.0 - 209.237.238.255
209.237.238.224 - - "GET /robots.txt HTTP/1.0" 200 111 "-" ""What the hell's going on Alexa?
Someone break the crawler or you just trying to sneak under the radar after everyone blocked your ass?
You claim to crawl as "ia_archiver" but I don't see any ID here whatsoever!
Please explain, inquiring minds want to know!
2 comments:
I caught them yesterday and it turns out that they've cralwed over 3k pages in the previous two days. .
I have a robots.txt disallow for their bot but the "no UA" crawler never looked at it.
Now its a "deny from 209.237.224.0/19"
BTW, after looking closer I discovered the hidden Alexa has been visiting almost daily!
Post a Comment