Thursday, May 22, 2008

Did CSC's Spybot Get Caught?

Looks like yet another corporate compliance spybot is hitting our servers, not like we need yet another spybot.

There's only one IP out of this entire range that consistently hits my servers.

OrgName: Corporation Service Company
NetRange: -

They claim to crawl the web:

Our proprietary technology scans and digests web pages, images and other Internet content around the clock to locate critical occurrences of online brand abuse.
Yet again, nobody has ever seen a crawler name in use so I'll hazard a guess it doesn't read or respect robots.txt when it's crawling, or possibly trespassing, on our servers.

I'd post more about the specifics on this one but I really don't want them to wise up too much because some of the things their crawler does, while pretending to be a browser, trips several alarms in my bot blocker.

Kind of hard to digest web pages when you're busy digesting error pages instead!

Just another day of the internet version of Spy vs Spy.


Anonymous said...

I saw the same bot come in on, which appears to be owned by 'Consult Dynamics' but which tracerts directly back to CSC.

It called robots.txt, but ignored it anyway. It supplied the user-agent " bot/1.0 (bot; http://;", FWIW.

Anonymous said...

I've seen it before - at least the user-agent;, but up until a few days ago, it had stopped coming by. I believe I have Bad Behavior blocking it now.
Didn't it first start from McColo Corp - or something like that?

Anonymous said...

Bill, this reply is off topic. Hope you don't mind.

WillMacc, be careful with Bad Behavior it will block this:

User Agent libwww-perl/5.79

That is a editors bookmarking tool.

IncrediBILL said...

FYI, the CSC bot I see isn't, it claims to be Firefox but it's no human.

@ban proxies - I block all libwww-perl too, so should everyone. The DMOZ editors can fix that tool with one lousy line of code so broadcast a UA that isn't the same as every botnet on the planet and until they do, they should be blocked.

'Nuff said ;)

Anonymous said...

i like u blog.