LiteFinder Network Crawler Go Fuck Yourself

I don't get too riled up until I read some self-serving pompous bullshit like this that just makes the hair stand up on the back of my neck:

Can I learn the IP addresses, which LiteFinder Network Crawler comes from?
Unfortunately, You can't since it is against the rules of our company.

The user agent for this mess is:

"Mozilla/5.0 (compatible; LiteFinder/1.0; +http://www.litefinder.net/about.html)"

Since they don't feel like sharing the IP addresses, let me do the honors since it's not against MY company policy:

208.101.44.3 -> mybluewine.net.
209.160.65.42 -> hopone.net.
209.62.109.178 -> ev1s-209-62-109-178.ev1servers.net.
216.40.220.34 -> ev1s-216-40-220-34.ev1servers.net.
216.40.222.50 -> ev1s-216-40-222-50.ev1servers.net.
216.40.222.66 -> ev1s-216-40-222-66.ev1servers.net.
216.40.222.82 -> ev1s-216-40-222-82.ev1servers.net.
216.40.222.98 -> ev1s-216-40-222-98.ev1servers.net.
67.19.114.226 -> w103.networkharmony.com.
67.19.250.26 -> 1a.fa.1343.static.theplanet.com.
70.85.113.242 -> f2.71.5546.static.theplanet.com.
74.53.243.226 -> e2.f3.354a.static.theplanet.com.
74.53.243.242 -> f2.f3.354a.static.theplanet.com.
74.53.244.18 -> 12.f4.354a.static.theplanet.com.
74.53.249.34 -> 22.f9.354a.static.theplanet.com.
74.86.209.74 -> templatestill.com.
74.86.249.98 -> westhoste.net.
75.125.18.178 -> ev1s-75-125-18-178.ev1servers.net.
75.125.47.162 -> ev1s-75-125-47-162.ev1servers.net.
75.125.52.146 -> ev1s-75-125-52-146.ev1servers.net.
84.19.176.208 -> ns.km22118.keymachine.de.
87.118.118.111 -> ns.km31417.keymachine.de.
87.118.98.57 -> ns.km22427.keymachine.de.
87.118.98.62 -> ns.km22426.keymachine.de.

There you go, all the IPs I've seen them use and they can shove the rules of their company where the sun doesn't shine.

Attributor Post-Mortem Copyright Compliance Revisited

My first post about the emergence of Attributor was about a year ago and I thought it was time to review and see what we've learned since then.

Here's where they've crawled from that we've spotted:

63.209.14.55 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
Proxy VIA=1.1 ind27.attributor.com:3128 FORWARD=10.50.40.74

63.209.14.10 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
Proxy VIA=1.1 ind25.attributor.com:3128 FORWARD=10.50.40.74

209.51.152.146 "Attributor.comBot"

66.231.188.172 "Attributor.comBot"

63.209.14.53 "Mozilla/5.0 (compatible; dejan/1.13.2 +http://www.attributor.com)"

63.209.14.7 "Mozilla/5.0 (compatible; dejan/1.13.2 +http://www.attributor.com)"

Now the amusing part is the IP 209.51.152.146 as it's a proxy and it appears they aren't any smarter than the rest of the bots as 340+ crawls have come via that IP this year including msnbot, Googlebot, Twiceler, Gigabot, Snapbot and some others so you're in fine company with other stupid crawlers out there.

What's curious is that 66.231.188.172 is one of Gigablast's IPs, and some of the others may be as well but they resolve to Level 3 blocks as do other Gigablast IPs, but I didn't look hard enough to confirm, lazy I guess.

Now let's examine one of my favorite statements on their website:

...you will no longer have to hold back top content or impose technical barriers on its viewing; instead, quality content can be made more easily available to a larger number of consumers.

Excuse me?

My technical barriers [not used on this blog] stop the problem in the first place just so I don't need to pay anyone to go chasing my content around the billions of pages on the web. As a matter of fact, my technical barriers are what trapped your crawl attempts above and identified what IP's your bots were using. That means your technology can't get past my technology so you'll never know if I'm stealing anyone's material but I'm pretty sure you aren't stealing my bandwidth finding out.

So now you have to ask yourself which method is easiest to stop content theft, blocking data centers and bulk downloaders on the fly or scanning billions of web pages looking for theft after the cows have already left the barn?

Bot blocking wins hands down as it's more cost effective without a doubt.

The best part is if someone wants to license your content you'll get 100% of the profits and not share with some company that wants to chase around the vast wasteland of the web looking for violators.

Maybe Attributor has some other places they crawl from without the user agent identifying the source, but that just means the bot blocker will stop and quarantine some anonymous IP address and we may never know it's really them.

Doesn't matter, I'm still banking on proactive content theft prevention technology and not reactive technology as it's easier to keep your cows at home when the fences are all closed and patrolled in the first place than try to round 'em up later.

Websense Stealth Crawler Bypassing Security?

What I find amusing are security companies that claim to be protecting the web while violating access control measures on web servers all over the world.

Here's what I see coming from WebSense that's obvious:

208.80.193.29 Mozilla/5.0 (compatible; Konqueror/3.0-rc2; i686 Linux; 20020108)
208.80.193.30 Mozilla/5.0 (compatible; Konqueror/3.0-rc4; i686 Linux; 20020418)
208.80.193.33 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312462)
208.80.193.34 Mozilla/5.0 (compatible; Konqueror/3.1; i686 Linux; 20020213)
208.80.193.36 Mozilla/5.0 (compatible; Konqueror/3.0-rc1; i686 Linux; 20020328)
208.80.193.37 Mozilla/5.0 (compatible; Konqueror/3.1-rc4; i686 Linux; 20020520)
208.80.193.41 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312466)
208.80.193.42 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312462)
208.80.193.51 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312460)
208.80.193.52 Mozilla/5.0 (compatible; Konqueror/3.0-rc6; i686 Linux; 20020204)

It makes me wonder if deliberately trying to bypass security measures in place that are designed to keep robots like WebSense off a server, such as robots.txt, .htaccess and other access controls, may violate the "Computer Hacking and Unauthorized Access Laws"?

Proving they've been busily sneaking around on lots of servers won't be too hard either.

Maybe WebSense should just claim any site that blocks them is off limits since we don't want them on our servers instead of trying to circumvent our security measures.

That would make too much sense wouldn't it?

Of course someone could claim that bad sites would just cloak clean content if they know it's WebSense. However, I'd rather give explicit permission for WebSense and then it wouldn't bother me so much if they crawled in stealth from a different IP address knowing that I gave permission in the first place.

Here's some of their known IP ranges:

Websense 66.194.6.0 - 66.194.6.255
Websense 74.211.167.208 - 74.211.167.215
Websense, Inc 208.80.192.0 - 208.80.199.255

Not sure these are the same company as a couple are in Canada and the other is in a different city, but what the heck, make up your own mind on these:

Websense Inc 67.117.201.128 - 67.117.201.143
Websense Systems Inc. 64.69.80.104 - 64.69.80.111
Websense Systems Inc. 64.69.80.96 - 64.69.80.103

There you go, some good bot blocking to go with your morning coffee should start off a fine Monday!

Another Stealth Crawler via Extended Host

Here we go with another stealth crawler operating from Extended Host:

194.110.162.19 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.225 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.227 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.228 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.231 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.84 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.85 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.86 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.87 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.88 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.89 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.92 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.93 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.94 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
194.110.162.96 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Here's the Extended Host IP range:

inetnum: 194.110.160.0 - 194.110.163.255
netname: EXTHOST-NET
descr: Extended Host

They just keep coming and I just keep closing more holes they slither through.

Even MORE Stealth Crawling Hosted at Corporate Colo

Here's yet another stealth crawler that came from Corporate Colocations's IP range:

74.124.192.137 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.138 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.161 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.162 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.175 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.181 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.183 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.195 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.198 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
74.124.192.215 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)

Here's their list of IP ranges:

Corporate Colocation Inc. MZIMA01-CUST-CORPCOLO02
64.235.225.8 - 64.235.225.15

Corporate Colocation Inc. MZIMA02-CUST-CORPCOLO04
216.193.219.0 - 216.193.219.255

Corporate Colocation Inc. MZIMA02-CUST-CORPCOLO01
216.193.197.0 - 216.193.197.255

Corporate Colocation Inc. MZIMA02-CUST-CORPCOLO03
216.193.208.0 - 216.193.208.63

Corporate Colocation Inc. CORPCOLO-206-62-132-0-22
206.62.132.0 - 206.62.135.255

Corporate Colocation Inc. CORPCOLO-206-62-144-0-23
206.62.144. - 206.62.145.255

Corporate Colocation Inc. CORPCOLO-206-62-146-0-22
206.62.146.0 - 206.62.149.255

Corporate Colocation Inc. MZIMA02-CUST-CORPCOLO05
216.193.251.0 - 216.193.251.255

Corporate Colocation Inc. NET-216-152-242-0-24
216.152.242.0 - 216.152.242.255

Corporate Colocation Inc. CORPCOLO-NET
205.134.224.0 - 205.134.255.255

Corporate Colocation Inc. NET-216-151-149-0-24
216.151.149.0 - 216.151.149.255

Corporate Colocation Inc. MZIMA02-CUST-CORPCOLO10
72.37.152.0 - 72.37.152.255

Corporate Colocation Inc. MZIMA03-CUST-CORPCOLO09
72.37.131.80 - 72.37.131.87

Corporate Colocation Inc. CORPCOLO-NET02
66.117.0.0 - 66.117.15.255

Corporate Colocation Inc. CORPCOLO-NET03
74.124.192.0 - 74.124.223.255

Corporate Colocation MZIMA01-CUST-CORPCOLO05
64.235.225.224 - 64.235.225.239

Corporate Colocation MZIMA01-CUST-CORPCOLO06
64.235.227.96 - 64.235.227.111

Corporate Colocation MZIMA01-CUST-CORPCOLO08
64.235.238.224 - 64.235.238.231

Corporate Colocation MZIMA01-CUST-CORPCOLO07
64.235.237.64 - 64.235.237.71

That little list of IPs should give you all some fun adding to your firewalls and .htaccess files.

Enjoy.

More Stealth Crawling Hosted at OC3Networks

No clue who or what this crawler is but it's coming from OC3Networks datacenter.

Here's the IPs and the user agent coming from their network:

72.11.155.106 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.112 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.113 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.125 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.131 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.137 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.154 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.197 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.204 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.211 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.219 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.223 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.228 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.236 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.237 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.246 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.34 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.37 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.45 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.5 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.57 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.61 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.63 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.64 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.67 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)
72.11.155.90 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)

Here's their ranges of IPs:

OC3 Networks & Web Solutions, LLC OC3-NETWORKS
66.63.160.0 - 66.63.191.255

OC3 Networks & Web Solutions, LLC OC3-NETWORKS
72.11.128.0 - 72.11.159.255

OC3 Networks ISWT-207-178-200-0
207.178.200.0 - 207.178.200.63

OC3 Networks OC3-NETWORKS-DSLUSERS
66.63.163.0 - 66.63.164.255

OC3 Networks OC3-NETWORKS--DEDICATED-SERVERS-RANGE
66.63.176.0 - 66.63.176.255

OC3 Networks OC3-NETWORKS--DEDICATED-SERVERS-RANGE
66.63.179.0 - 66.63.179.255

OC3 Networks OC3-NETWORKS---COLOCATIONS-VOIP
66.63.178.0 - 66.63.178.127

Not sure I'd block the DSLUSERS range but the rest look like fair game.

Enjoy.

Munax Stealth Crawler

Stumbled upon a stealth crawler hitting my site from multiple IPs and it turned out to belong to Munax who claims right up front that they haven't named their crawler and fake being a legit user which is pretty damned scummy.

My guess would be they figured out they couldn't access sites with good security so they decided to get around it without a bot name, but here's some bullshit excuse they use:

Our crawler does not have a "name", yet. Instead it announces itself to be a standard web browser, a "Mozilla 4.0" kind-of-browser compatible with the browser Microsoft Internet Explorer 6.0, running on the Windows NT 5.1 operating system. The reasons for this are: (a) Today, web servers are intelligent enough to react on the type of user agent. If our crawlers had a name, say MunaxRob or something like that, many web servers would not know about it and would return junk or maybe nothing at all. (b) We want the web server to return a page to us where the page looks as close as possible to a page that can be viewed with a standard web browser. This, to create the best possible indexing in our database and a WYSIWYG experience for anybody that is visiting our search engine.

Well listen up fuckheads, there's a reason we would return junk or nothing at all which is we don't want your goddamn spider crawling our fucking website!

What part of FUCK OFF! don't you understand that drives you to bypass our security and crawl regardless of whether we want you or not?

Amazingly they admit their IP range:

Your site might have been visited by our crawlers, with network addresses in the range of 82.99.30.2 - 82.99.30.73. Here is a short FAQ answering some of the questions you might have:

I've confirmed this crawl range in my logs:

82.99.30.15 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.17 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.21 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.25 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.26 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.30 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.33 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.37 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.45 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.54 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
82.99.30.67 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Well, this fucking crawler is now blocked.

Bunch of bullshit....

TextDigger Caught Using Stealth Shovel

Some semantic search thing called TextDigger stumbled into my spider trap today.

I have nothing against semantic search, I'm not an anti-semantite (that's not the word you think it is, read it twice, i made it up just to be punny), but I'm definitely anti-stealth crawler.

According to the bot blocker, TextDigger requested 136 pages after being challenged while using the following user agent:

64.124.138.164 [nat1.textdigger.com]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

Here's their IP range:

TextDigger MFN-B849-64-124-138-160-28 (NET-64-124-138-160-1)
64.124.138.160 - 64.124.138.175

Not sure if what hit my server was their actual main crawler or not, but they aren't gaining any brownie points with me crawling in stealth for any reason.

Bot Blocker Tracking More Than 80K Unique IPs

Lately I've been doing some analysis work on my database of IPs that I'm tracking for bad behavior and it exceeded 80K unique IPs. Many of these are from data centers, bot nets, home-based scrapers and then some, but it's a staggering number when it exceeds 80K.

People always wonder why I'm such an anti-scrape nazi but it's really not hard to see the problem when you multiply 80K IPs trying to scrape an excess of 40K pages, which is a potential for having over 3 BILLION pages scraped in the last year.

Here's the number with all the zeroes: 3,200,000,000 pages.

OK, that's really a lot of pages and there's no way I'm paying for that kind of bandwidth.

I seriously doubt they would ever hit the maximum pages but there's no way I'm unlocking the doors and let them run rampant just to find out how bad it would really get.

Here's a sample of 3 greedy fuckers that paid a visit just today:

82.34.200.237 [82-34-200-237.cable.ubr05.hari.blueyonder.co.uk.] requested 710 pages as "Mozilla/4.0 (compatible; GoogleToolbar 4.0.1020.2544-big; Windows XP 5.1; MSIE 6.0.2900.2180)"

70.80.186.223 [modemcable223.186-80-70.mc.videotron.ca.] requested 1071 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

201.58.219.234 [20158219234.user.veloxzone.com.br.] requested 329 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1)"

They only got a couple of pages before getting nothing but garbage, but they just keep trying. Based on the location of the IPs, I'm thinking it might be compromised machines in a botnet trying to scrape from stealth locations, hard to say.

The best part is, they're now charter members of my AUTO-QUARANTINE list of IPs meaning they're blocked from accessing any pages on their next trip unless a human is at the controls, and even then, they could get locked out real fast if they aren't careful!

Hosting Company Blocks Bots

Looks like we overlooked this little press release last year when Mecca Hosting announced Mecca Hosting Bounces Bad Bots from their servers.

Here's the good stuff:

Mecca Hosting, a leader in providing customized hosting solutions, has just released a new system to detect and block suspicious automated programs or "bots". Mecca Hosting's new system, by blocking these bad bots, helps protect customer's intellectual property, e-mail addresses, and prevents blog spamming and hacker attacks. This new system can detect good bots, like Search Engines and specialized tools, to allow them access to websites, while blocking the bad ones. This new technology will result in vastly increased website uptime and performance, due to the expected reduction in hack attempts; mainly because hackers use automated tools to find system vulnerabilities.

Sounds good, but how good is it?

Just to see if it basically worked, I used a few tools to try to snag a page or two and got bitch slapped with 403 errors.

Not bad Mecca, not bad.

If we could just get all hosts to do this, and even dedicated server companies to offer this kind of technology, maybe the scrapers would already be out of business.

Greedy French Scraping Bastard

This swine from the land of overpriced wine asked for robots.txt then tried to rip over 1300 pages.

83.198.150.2 "GET /robots.txt HTTP/1.0" 200 146 "-" "-"

83.198.150.2 [ALille-252-1-48-2.w83-198.abo.wanadoo.fr.] requested 1321 pages as "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 4.0)"

Too bad Pepe Le Pew, your feeble scraping attempts SUCK and you got 1300+ pages of error messages so Phuck Off.

[sing a long with apologies to Cheryl Crow...]

All I wanadoo is scrape some pages,
We'll download it, and not take ages.

All I wanadoo is grab your site,
And then cloak it all to Google tonight!

Sorry Pharma Spammer Strikes Again

Some miserable asshole is using "Sorry for subject" as a spam topic and attempting to spam from all over the world. Mainly it's one IP in Germany with some others from other locations.

Most of the links they're spamming are for pharma related sites but there was an actual domain park page thrown in as well which really made me giggle.

The most fun is my spam blocker that I wrote never lets any of this shit through to my website, but just silently logs it so I can go back and see what these shit-for-brains are doing later just for my own amusement, plus collecting the IPs to block.

Here's the German sorry spammer:

62.141.53.139 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; TheFreeDictionary.com; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727)" "Sorry for subject" http://tramadol.4hfs.org
62.141.53.139 "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.7.5) Gecko/20041220 K-Meleon/0.9" "Sorry for subject" http://phentermine.4hfs.net
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a1) Gecko/20051102 Firefox/1.6a1" "Sorry for subject" http://tramadol.hfslink.com
62.141.53.139 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; PeoplePal 6.2)" "Sorry for subject" http://tramadol.4hfs.net
62.141.53.139 "Mozilla/4.0 (compatible; ICS 1.2.105)" "Sorry for subject" http://phentermine.2hl.org
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20041122 Firefox/0.5.6+" "Sorry for subject" http://phentermine.3mac.info
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)" "Sorry for subject" http://tramadol.4hfs.org
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8a) Gecko/20040416 Firefox/0.8.0+" "Sorry for subject" http://phentermine.viphls.org
62.141.53.139 "Mozilla/6.0 (compatible; MSIE 7.0a1; Windows NT 5.2; SV1)" "Sorry for subject" http://phentermine.3mac.info
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6" "Sorry for subject" http://tramadol.medhls.com
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8b5) Gecko/20051019 Flock/0.4 Firefox/1.0+" "Sorry for subject" http://phentermine.3mac.info
62.141.53.139 "Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.6) Gecko/20040406 Galeon/1.3.15" "Sorry for subject" http://tramadol.medhls.com
62.141.53.139 "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126" "Sorry for subject" http://phentermine.viphls.org

Here's the rest of the sorry spammers:

59.93.35.80 "Mozilla/5.0 (Windows; U; Win95; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0" "Sorry for subject" http://phentermine.10pharm.com
60.217.227.141 "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.3) Gecko/20041002 Firefox/0.10.1" "Sorry for subject" http://phentermine.viphls.org
68.10.68.144 "Mozilla/4.0 (compatible; MSIE 6.0; X11; Linux i686) Opera 7.20 [en]" "Sorry for subject" http://tramadol.madnewus.com
69.249.59.232 "Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.6) Gecko/20040206 Firefox/0.8" "Sorry for subject" http://cialis.mednewus.com
86.139.64.133 "Mozilla/4.0 (compatible; MSIE 6.0; Mac_PowerPC Mac OS X; en) Opera 8.0" "Sorry for subject" http://tramadol.hfslink.com
89.208.4.195 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461)" "Sorry for subject" http://ativan.10pharm.com
200.49.102.100 "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.5) Gecko/20031016 K-Meleon/0.8" "Sorry for subject" http://tramadol.4mednew.com
200.55.113.119 "Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1)" "Sorry for subject" http://tramadol.2hl.org
201.17.208.188 "Opera/8.01 (Windows NT 5.1)" "Sorry for subject" http://tramadol.4hfs.net
201.45.41.153 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-GB; rv:1.7.6) Gecko/20050222 Firefox/1.0.1" "Sorry for subject" http:///tamiflu.hlspharm.info
210.87.251.41 "Mozilla/5.0 (Windows; U; Windows NT 5.0; de-DE; rv:1.7) Gecko/20040707 Firefox/0.9.2" "Sorry for subject" http://zetia.hlspharm.info
218.38.9.218 "Mozilla/4.76 [en] (Windows NT 5.0; U)" "Sorry for subject" http://phentermine.cvipm.com
220.78.229.119 "Mozilla/5.0 (Windows; U; WinNT4.0; en-CA; rv:0.9.4) Gecko/20011128 Netscape6/6.2.1" "Sorry for subject" http://tramadol.10pharm.com

You're such a sorry fucking spammer you don't even know you're just wasting your time you stupid fuckhead.

Gigablast Data Trail

While following where my data goes on the internet I found a couple of sites that appear to be using data from Gigablast which include eWoss and searchEstate.

The upside is fewer crawlers as they're leveraging existing data crawls in multiple locations.

The downside is that you have no control where your information shows up so the only way to control that relationship is block the source.

Update: Also found Gigablast content in Webled as well.

Ezilon Also Has Some LookSmart Content

This time my content tracking bugs led me to Ezilon which has content that originated from LookSmart. Don't know if Ezilon is a LookSmart partner or what the deal is, perhaps they scraped LookSmart, but the link to one of my sites in their listings was definitely crawled by LookSmart.

Just goes to show that blocking bad bots from your site doesn't always stop your content from being misappropriated anyway.

Update: Also found LookSmart data in xogger.com.

Webaroo's Content Stealing PulseBot Flatlined

If you've never seen Webaroo before, the concept of copyright obviously has been completely glossed over.

Here's what it says on their website:

Webaroo servers crawl the web, analyze web pages and automatically select the subset of pages with the greatest diversity and quality in the least storage size. These pages are then packaged into topic-specific "Web Packs" that can be downloaded by users onto their devices. Once downloaded, users can search and browse that content on the go.

Here's an English to English translation:

Webaroo takes whatever copyrighted content of yours we want and repackage it for our customers without permission. Of course we do it without permission because nobody knows about Webaroo in the first place so they won't stop us or the many bot names. Isn't it cool how we're going to steal your shit and pack it up so others can download it and now they don't even need to bother visiting your website? Wicked!

Look at the total number of bot names coming from their crawler's IP address.

64.124.122.228 "WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)"

64.124.122.228 "PiyushBot (Piyush Web Miner; http://piyush.com/feedback.html)"

64.124.122.228 "RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html)"

64.124.122.228 "RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)"

64.124.122.228 "SumeetBot (Sumeet Bot; http://64.124.122.252/feedback.html)"

64.124.122.228 "PsBot (PsBot; http://64.124.122.252/feedback.html)"

64.124.122.228 "pulseBot (pulse Web Miner)"

Hell, if you were trying to stop them using robots.txt it's a lost cause as the bot names seem to get changed faster than a baby's diaper.

I would just block their range of IP's, it's more convenient.

Webaroo MFN-B843-64-124-122-224-27 (NET-64-124-122-224-1)
64.124.122.224 - 64.124.122.255

That's how you stop name changing bots, the firewall way.

Package THAT into a topic-specific "Web Pack" and download it.

Swooglebot Thnks My Site Is Anti-Semantic

Here comes the little toy bot from UMBC that's trying to figure out the semantic web and sure enough, my anti-semantic site booted the little fucker.

Here's the 411:

130.85.34.29 [eb2.cs.UMBC.EDU.] "Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html)"

Since my site is anti-semantic, does that make me an anti-semantite?

Hmmm...

Nutch Goes to the Opera

OK, everyone knows I'm a big Nutch fan.

Everyone should also know the above statement is sarcasm laid on thick like peanut butter.

Anyway, back to the topic...

I saw this supposedly Nutch user agent in my logs today which is pretty slimy claiming to be MSIE, Opera, and Nutch all in the same UA.

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.01/Nutch-0.9 (http://lucene.apache.org/nutch/about.html; http://lucene.apache.org/nutch/bot.html; mail@dev.null)

Whoever thought that was clever enough to slide under my radar was wrong as that word Nutch set off alarms and tossed your crawl into the online equivalent of the pit of despair.

Sorry, better luck next time.

WorldWebWide Scrapes LookDumb

Today my content tracking bugs led me to something in the worldwebwide.net which originated from LookSmart.

The IP address where the data was originally crawled from:

60.88.242.64 -> sv-crawlfw4.looksmart.com

This is nothing new as LookSmart seems to be a scraping target as I've already reported same thing happening with GoodBidWords.com containing scraped LookSmart listings.

Hong Kong Cable IP Block Scraping

I've predicted this would happen for some time but have no clue how they managed to utilize and entire block of IPs from a HK cable company to scrape.

Look at the list below and not it's all in the same block, all 1-2 pages max, all the same user agent, and it flew right under the normal radar except my proximity alarm noticed this trend.

222.166.160.1 [cm222-166-160-1.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.10 [cm222-166-160-10.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.100 [cm222-166-160-100.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.101 [cm222-166-160-101.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.102 [cm222-166-160-102.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.103 [cm222-166-160-103.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.104 [cm222-166-160-104.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.105 [cm222-166-160-105.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.106 [cm222-166-160-106.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.108 [cm222-166-160-108.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.109 [cm222-166-160-109.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.11 [cm222-166-160-11.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.111 [cm222-166-160-111.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.113 [cm222-166-160-113.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.114 [cm222-166-160-114.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.115 [cm222-166-160-115.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.117 [cm222-166-160-117.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.118 [cm222-166-160-118.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.12 [cm222-166-160-12.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.120 [cm222-166-160-120.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.121 [cm222-166-160-121.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.123 [cm222-166-160-123.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.124 [cm222-166-160-124.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.127 [cm222-166-160-127.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.128 [cm222-166-160-128.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.129 [cm222-166-160-129.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.130 [cm222-166-160-130.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.131 [cm222-166-160-131.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.132 [cm222-166-160-132.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.133 [cm222-166-160-133.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.135 [cm222-166-160-135.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.136 [cm222-166-160-136.hkcable.com.hk.] pages 3- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.137 [cm222-166-160-137.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.138 [cm222-166-160-138.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.139 [cm222-166-160-139.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.140 [cm222-166-160-140.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.141 [cm222-166-160-141.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.142 [cm222-166-160-142.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.144 [cm222-166-160-144.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.145 [cm222-166-160-145.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.146 [cm222-166-160-146.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.147 [cm222-166-160-147.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.148 [cm222-166-160-148.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.149 [cm222-166-160-149.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.15 [cm222-166-160-15.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.150 [cm222-166-160-150.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.154 [cm222-166-160-154.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.155 [cm222-166-160-155.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.156 [cm222-166-160-156.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.158 [cm222-166-160-158.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.163 [cm222-166-160-163.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.164 [cm222-166-160-164.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.165 [cm222-166-160-165.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.166 [cm222-166-160-166.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.167 [cm222-166-160-167.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.168 [cm222-166-160-168.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.169 [cm222-166-160-169.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.17 [cm222-166-160-17.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.171 [cm222-166-160-171.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.172 [cm222-166-160-172.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.173 [cm222-166-160-173.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.174 [cm222-166-160-174.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.175 [cm222-166-160-175.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.176 [cm222-166-160-176.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.18 [cm222-166-160-18.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.180 [cm222-166-160-180.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.181 [cm222-166-160-181.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.183 [cm222-166-160-183.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.184 [cm222-166-160-184.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.186 [cm222-166-160-186.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.188 [cm222-166-160-188.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.189 [cm222-166-160-189.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.19 [cm222-166-160-19.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.191 [cm222-166-160-191.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.192 [cm222-166-160-192.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.193 [cm222-166-160-193.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.194 [cm222-166-160-194.hkcable.com.hk.] pages 3- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.195 [cm222-166-160-195.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.196 [cm222-166-160-196.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.199 [cm222-166-160-199.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.201 [cm222-166-160-201.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.203 [cm222-166-160-203.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.204 [cm222-166-160-204.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.205 [cm222-166-160-205.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.206 [cm222-166-160-206.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.208 [cm222-166-160-208.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.209 [cm222-166-160-209.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.21 [cm222-166-160-21.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.210 [cm222-166-160-210.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.211 [cm222-166-160-211.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.212 [cm222-166-160-212.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.213 [cm222-166-160-213.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.214 [cm222-166-160-214.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.215 [cm222-166-160-215.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.218 [cm222-166-160-218.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.220 [cm222-166-160-220.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.221 [cm222-166-160-221.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.222 [cm222-166-160-222.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.223 [cm222-166-160-223.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.224 [cm222-166-160-224.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.225 [cm222-166-160-225.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.226 [cm222-166-160-226.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.227 [cm222-166-160-227.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.228 [cm222-166-160-228.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.23 [cm222-166-160-23.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.230 [cm222-166-160-230.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.232 [cm222-166-160-232.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.234 [cm222-166-160-234.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.235 [cm222-166-160-235.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.237 [cm222-166-160-237.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.238 [cm222-166-160-238.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.239 [cm222-166-160-239.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.24 [cm222-166-160-24.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.241 [cm222-166-160-241.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.242 [cm222-166-160-242.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.243 [cm222-166-160-243.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.244 [cm222-166-160-244.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.245 [cm222-166-160-245.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.246 [cm222-166-160-246.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.247 [cm222-166-160-247.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.248 [cm222-166-160-248.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.249 [cm222-166-160-249.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.25 [cm222-166-160-25.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.251 [cm222-166-160-251.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.252 [cm222-166-160-252.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.253 [cm222-166-160-253.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.254 [cm222-166-160-254.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.26 [cm222-166-160-26.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.29 [cm222-166-160-29.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.3 [cm222-166-160-3.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.30 [cm222-166-160-30.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.32 [cm222-166-160-32.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.33 [cm222-166-160-33.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.36 [cm222-166-160-36.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.38 [cm222-166-160-38.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.39 [cm222-166-160-39.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.40 [cm222-166-160-40.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.41 [cm222-166-160-41.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.42 [cm222-166-160-42.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.44 [cm222-166-160-44.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.45 [cm222-166-160-45.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.47 [cm222-166-160-47.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.48 [cm222-166-160-48.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.5 [cm222-166-160-5.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.51 [cm222-166-160-51.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.54 [cm222-166-160-54.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.56 [cm222-166-160-56.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.57 [cm222-166-160-57.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.58 [cm222-166-160-58.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.59 [cm222-166-160-59.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.6 [cm222-166-160-6.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.60 [cm222-166-160-60.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.62 [cm222-166-160-62.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.63 [cm222-166-160-63.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.64 [cm222-166-160-64.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.65 [cm222-166-160-65.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.66 [cm222-166-160-66.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.67 [cm222-166-160-67.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.68 [cm222-166-160-68.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.69 [cm222-166-160-69.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.70 [cm222-166-160-70.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.72 [cm222-166-160-72.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.73 [cm222-166-160-73.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.74 [cm222-166-160-74.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.75 [cm222-166-160-75.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.76 [cm222-166-160-76.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.77 [cm222-166-160-77.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.78 [cm222-166-160-78.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.8 [cm222-166-160-8.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.80 [cm222-166-160-80.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.82 [cm222-166-160-82.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.83 [cm222-166-160-83.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.84 [cm222-166-160-84.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.85 [cm222-166-160-85.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.86 [cm222-166-160-86.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.87 [cm222-166-160-87.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.88 [cm222-166-160-88.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.90 [cm222-166-160-90.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.91 [cm222-166-160-91.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.92 [cm222-166-160-92.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.94 [cm222-166-160-94.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.95 [cm222-166-160-95.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.96 [cm222-166-160-96.hkcable.com.hk.] pages 2- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
222.166.160.97 [cm222-166-160-97.hkcable.com.hk.] pages 1- Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

When I reviewed the physical log entries you can see the data being accessed was sequential.

Not sure if there are other ranges of IP in the HK Cable Company to be concerned with, but 222.166.160.* is definitely one to block for now.

Abotcalypse Now

My goodness, I must be making those people that run bad bots afraid because the number of blog and comment posts decrying how I'm about to destroy the web are cropping up almost daily since my presentation at PubCon.

Assuming that the people making these posts are the same people causing the problems that drive me to the "extremes of mouth-frothing, profanity, and severe bot-blocking" you might think that they would take note that they are responsible for the bad behavior that might stop the next Google from being born.

Unfortunately I'm the "arrogant twit" as I'm so self-obsessed with my mission from hell that I'm oblivious to how my technology will wreck the net.

Riiiiiiiiiiiight.

You people running scrapers need to take stock of the damage you'll ultimately cause and stop trying to pass the blame my way.

I'm just a supplier of weapons in the war on bad bots so surrender now and save the web.

Nah, that would be too easy as they don't think they do anything wrong.

In summary, not being someone that wants to disappoint the author of that amusing post, I laughed so hard reading it because it was just too fucking funny, there's your profanity.

Enjoy.