Looks like yet another corporate compliance spybot is hitting our servers, not like we need yet another spybot.
There's only one IP out of this entire range that consistently hits my servers.
OrgName: Corporation Service Company
OrgID: CORPO-9-Z
NetRange: 165.160.0.0 - 165.160.255.255
They claim to crawl the web:
Our proprietary technology scans and digests web pages, images and other Internet content around the clock to locate critical occurrences of online brand abuse.Yet again, nobody has ever seen a crawler name in use so I'll hazard a guess it doesn't read or respect robots.txt when it's crawling, or possibly trespassing, on our servers.
I'd post more about the specifics on this one but I really don't want them to wise up too much because some of the things their crawler does, while pretending to be a browser, trips several alarms in my bot blocker.
Kind of hard to digest web pages when you're busy digesting error pages instead!
Just another day of the internet version of Spy vs Spy.
5 comments:
I saw the same bot come in on 216.158.1.198, which appears to be owned by 'Consult Dynamics' but which tracerts directly back to CSC.
It called robots.txt, but ignored it anyway. It supplied the user-agent " bot/1.0 (bot; http://; bot@bot.bot)", FWIW.
I've seen it before - at least the user-agent; bot@bot.bot, but up until a few days ago, it had stopped coming by. I believe I have Bad Behavior blocking it now.
Didn't it first start from McColo Corp - or something like that?
Bill, this reply is off topic. Hope you don't mind.
WillMacc, be careful with Bad Behavior it will block this:
core-n02.dmoz.aol.com
User Agent libwww-perl/5.79
That is a editors bookmarking tool.
FYI, the CSC bot I see isn't bot@bot.bot, it claims to be Firefox but it's no human.
@ban proxies - I block all libwww-perl too, so should everyone. The DMOZ editors can fix that tool with one lousy line of code so broadcast a UA that isn't the same as every botnet on the planet and until they do, they should be blocked.
'Nuff said ;)
i like u blog.
Post a Comment