RED ALERT #2 - Distributed IP Scraper on BBCOM

This one is virtually identical to scraper spotted on Vericenter, same profile to the letter.

Claims to be the same exact browser:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1

This spider was seen operating from these IP addresses:

66.234.139.195
66.234.139.200
66.234.139.203
66.234.139.204
66.234.139.206
66.234.139.207
66.234.139.211
66.234.139.213
66.234.139.216
66.234.139.217

This IP range belongs to BBCOM:

OrgName: Backbone Communications, Inc.
OrgID: BBCM
Address: 515 South Flower Street
Address: Suite 4350
City: Los Angeles
StateProv: CA
PostalCode: 90071
Country: US

NetRange: 66.234.128.0 - 66.234.159.255

I'm blocking 66.234.139.0/24 for the moment, but keeping an eye on the rest of their network in case these scrapers switch to a new block.

Another Proximity Alert on PCCW

Getting multiple hits from the same IPs over and over asking for the exact same web page, no images, nothing else, just the same web page day after day. One day the user agent was blank and another day simply"Mozilla/5.0".

This is the range of IPs that just keep repeating the same request.

210.87.251.111
210.87.251.107
210.87.251.41
210.87.251.106

Appears to be coming from HK, can't tell if it's a shared DHCP situation or what, but I'm blocking 210.87.251.0/24 just to be safe.

Another DUMBASS Crawler

This one hails from South Africa, tried to crawl "/#top" as "GET /%23top"

You poor dumb fucker, BUSTED!

RED ALERT - Distributed IP Scraper Hosted on Vericenter

Here's a real sneaky scraper using distributed IPs that is using a bot that almost appears designed to fly under my bot blockers radar. No single IP address accessed enough pages or did anything obnoxious enough to set off any triggers but the collective accesses set off a proximity alarm and they got nailed anyway.

The scraper is pretending to be Firefox for Linux:

http://www.Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1/

The range of IP's noticed in this scrape attack are as follows:

65.38.102.138 ip-65-38-102-138.hou.vericenter.com
65.38.102.139 ip-65-38-102-139.hou.vericenter.com
65.38.102.141 ip-65-38-102-141.hou.vericenter.com
65.38.102.143 ip-65-38-102-143.hou.vericenter.com
65.38.102.145 ip-65-38-102-145.hou.vericenter.com
65.38.102.146 ip-65-38-102-146.hou.vericenter.com
65.38.102.147 ip-65-38-102-147.hou.vericenter.com
65.38.102.148 ip-65-38-102-148.hou.vericenter.com
65.38.102.150 ip-65-38-102-150.hou.vericenter.com
65.38.102.153 ip-65-38-102-153.hou.vericenter.com
65.38.102.155 ip-65-38-102-155.hou.vericenter.com
65.38.102.157 ip-65-38-102-157.hou.vericenter.com

The host information is as follows:

OrgName: VeriCenter, Inc.
OrgID: VRCT
Address: 757 N Eldridge Parkway
City: Houston
StateProv: TX
PostalCode: 77079
Country: US
NetRange: 65.38.96.0 - 65.38.111.255

Athough the attack seems to be centered on the 65.38.102.0/24 block at the Houston datacenter of Vericenter, I think I'm going to completely block Vericenter as it doesn't appear to have any ISP facilities [ie. NO HUMANS] and see if anything else bounces off the bot blocker from their facilities.

Another Web 2.0 Scraper Company

Don't roll your eyes and think that Bill's just making a fuss as this company claims they scrape:

Real-time Data Collection - Technologies for crawling, monitoring and scraping newly posted web content including the content from the “deep web."

See, I'm not making this shit up, honest to god admitted scrapers!

Not only that, they proudly display the number of sources they scrape updated constantly on their home page.

They appear to crawl without looking at robots.txt best I can tell, don't identify the source of the crawler other than it's the "Jakarta Commons-HttpClient", and their primary interest in my site seems to be attempting to crawl content referenced from my XML feed.

I'm not sure what information they could possibly think is on my site that could help "Institutional Investors leverage the latest technology and data to make better investment decisions" but they'll just have to be in the dark and use the Magic 8-ball from now own.

They have been seen using these IPs:

206.188.0.11 "Jakarta Commons-HttpClient/3.0"
206.188.0.22 "Jakarta Commons-HttpClient/3.0"
206.188.0.10 "Jakarta Commons-HttpClient/3.0"
206.188.0.23 "Jakarta Commons-HttpClient/3.0"

They are all part of the Geometric Group:

Geometric Group DP-206-188-0-0 (NET-206-188-0-0-2)
206.188.0.0 - 206.188.0.63

I'd just block the whole range and be done with it and hope we don't cause the market to crash.

UPDATE: They switched to Java in 2007!

01/22/2007 206.188.0.22 "Java/1.5.0_03"
01/22/2007 206.188.0.23 "Java/1.5.0_06"

Then mysteriously, stopped pinging my server on 03/15/2007 after a year of being fed garbage.

Think someone finally realized they were getting bounced?

Blue Frog Legs and Spam for Dinner

Normally I don't comment on the news and such but Blue Security shutting down their anti-spam Blue Frog operation is stunning. Obviously the spammers were feeling the pressure and it was working or they wouldn't have attacked, the plan was working. Then in this shocking turn of events the anti-spamming Generals leading the attack not only retreated, they resigned from the effort!

So what if the spammers brought down a few servers and services here and there?

Isn't that the whole idea to get those spamming bastards out in the open so people can track them and block their asses once and for all, put them out of business?

When you start a war you certainly don't pack up and go home the minute you get a bloody nose so I suspect that there's a lot more to this story as people just don't fold so easily from a purely technological war. With all the money at stake in spamming, I'm suspecting the threats got a bit more personal which resulted in the sudden shut down, but that's purely speculating on my part.

If this escalated into seriously dangerous territory, where it seemed to be heading, the big service providers, government and everyone else would've gotten involved and put a permanent stop to those responsible.

Unfortunately you Blue Pussies gave up before we ever got a chance for it to get really interesting.

Maybe someone with balls will step up and continue where you left off.

Port 80 Proxies Expose Themselves

Don't know whoever the fucking morons are writing those stupid fucking proxy servers, but a shitload of them were just blocked today when we noticed they were appending ":80" to the URL and it shows up in the HTTP_HOST parameter.

Normally HTTP_HOST just has something like "www.domain.com" but when the connection is initiated from a certain cluster of proxy servers it shows up as "www.domain.com:80" which is trivial to block.

Saved me a shitload of work tracking down the IPs to block.

Thank you VERY MUCH you stupid fucking assholes!

Search Engine Harvesting

Probably never would've noticed this but the referral string changing set off my referral spam trap. Turned out it wasn't a referral spammer at all but someone crawling my site trying to mine specific topic search engine results from both Google and Yahoo.

Very strange behavior too.

Cloaked as MSIE, downloaded all the images as well to remain hidden, yet crawled so fast it also set off a speed trap.

Stupid stupid stupid.

Dragonfly Crawler

No clue what this thing is and nobody seems to have any real information about it, but it's been seen visiting from 5 different IP addresses.

72.29.233.182 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.183 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.185 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.186 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.188 "dragonfly(ebingbong@playstarmusic.com)"

Claims to be associated with www.playstarmusic.com (72.29.233.167) and the IP address is so close it's possible.

If anyone has any additional information it would be helpful.

When Innovations Collide

Some web crawler has hit my site a few times called Heritrix which appears to be written mostly by the team at Archive.org, the same team that created ia_archiver for those of you that haven't had your coffee yet.

Yes, it supports robots.txt, but if you didn't know this damn thing existed you wouldn't bother blocking it now would you?

People writing crawlers wonder why webmasters get pissed tracking and opt-ing out all this nuisance crawling on their websites, but I digress, that's an old rant.

The real amusement is that Heritrix claims their technology is designed to "collect the digital artifacts of our culture and preserve them for the benefit of future researchers and generations" which is a bunch of pretty language to try to sidestep downloading a website without permission, especially when the webmaster probably isn't aware of your crawler, doesn't matter how you try to candy coat it.

Now comes the fun part,
let's see who was using it and why!

Today's attempted crawl was HUGE so it's safe to assume this thing has been on my site in the past and apparently the crawler was even banned on a previous bayarea.net IP address:

209.128.119.46 "Mozilla/5.0 (compatible; heritrix/1.6.0 +http://innovationblog.com)

Today the crawler used a different bayarea.net IP, could be DHCP, could be on purpose to sidestep the previous ban. Who knows, but it only got a couple of pages before the doors were automatically slammed by the bot blocker:

209.128.119.17 - "Mozilla/5.0 (compatible; heritrix/1.6.0 +http://innovationblog.com)"

With my curiosity in overdrive, it was time to research innovationblog.com and see why they were crawling my site. Not a clue as there's nothing but a "This Web Site Coming Soon" site under construction page, but the WHOIS for the site was very revealing.

Registrant:
Michael Osofsky
1758 Shoreline Blvd. Suite B
Mountain View, California 94043
United States

Registered through: GoDaddy.com
Domain Name: INNOVATIONBLOG.COM
Created on: 09-Mar-05
Expires on: 09-Mar-07
Last Updated on: 06-Mar-06

Administrative Contact:
Osofsky, Michael mosofsky@accelovation.com
1758 Shoreline Blvd. Suite B
Mountain View, California 94043
United States
(650) 968-4741 Fax --

Technical Contact:
Osofsky, Michael mosofsky@accelovation.com
1758 Shoreline Blvd. Suite B
Mountain View, California 94043
United States
(650) 968-4741 Fax --

Domain servers in listed order:
WSC1.JOMAX.NET
WSC2.JOMAX.NET

This Michael seems to be involved with a company called accelovation.com and he seems to be big in the innovation circles having founded the MIT Innovation Club.

According to the Accelovation website:

Accelovation is the first and only Market Discovery System (MDS) that allows innovators to mine the online world for insights into unmet needs, trends, innovations and market activity.

Sound familiar?

We crawl you and use your information without permission to make a profit.

Where have we heard this before?

I'll bet they'll be surprised at my attitudes about this but they should try reading some webmaster forums and find out what they're doing probably isn't welcome without permission, some clue posted about what the benefits are to the webmaster to allow his site to be crawled, yada yada yada we've been down this path a few times before, it's getting old.

Sorry, but your innovation collided with my innovation called a bot blocker.

Your crawl is denied, and thanks for playing.

My Family is CRAWLING?

Call me confused, but why is MyFamily.com crawling my site?

Yup, the genealogy people, that's them.

66.43.16.199 [nat.myfamilyinc.com] has requested 110 pages as Mozilla/4.0 (compatible; MyFamilyBot/1.0; +http://www.myfamilyinc.com)

Even claims to be them, looked at robots.txt like they were supposed to, yada yada.

Still went home with a pile of crap instead of my pages.

Put that on your family tree and smoke it.

John has an interesting theory

I'm not sure if any of you have ever seen John Gabriel's Greater Internet Fuckwad Theory but take a gander, quite amusing.

Just in case one of you thinks "Well, that explains Bill in a nutshell!" then think again as I've always been very vocal, blunt, and to the point.

Doesn't mean I'm not a fuckwad, but the internet had nothing to do with that!

Snared a Layered proxy web host

Yup, that's right, someone decided we need to hide on the net so much they actually created a Proxy Web Host and it appears scrapers are using them, what a shocker.

I found them as AdSense was trying to crawl thru the proxy:

BAD_AGENT: 72.232.31.226 [prx1.proxywebhost.com.] requested 3 pages as "Mediapartners-Google/2.1"

So who are these people using for their provider?

Layered Technologies
72.232.0.0 - 72.232.255.255

That's a HUGE block of IP's to just block out of hand, so how much abuse has been coming from this range? Let's search on "72.232." and see what pops up.

First, it appears I already banned a c-block over there running a multi-IP scraper trying random user agents:

BANNED=72.232.67.222 yrkqi3jrmnbrsk3mUpnrwung
BANNED=72.232.67.219 vhsflbuwvLsbwmyp8xse8hvpdpplLdxdx
BANNED=72.232.67.219 utgkm gylmugtdblyppqqu
BANNED=72.232.67.220 pt tkglswaqatq k rfxqolbtqbygxlhvS0qqv
BANNED=72.232.67.221 djpqaegrbxpfbqnkxvqeniqfogyb rnt
BANNED=72.232.67.221 wbdprvjiqbw jbsvqse7
BANNED=72.232.67.220 upehrsqqqevdljtwrgkkbthk e
BANNED=72.232.67.220 sjxgohdtum3yybmbfyembisbxibei
BANNED=72.232.67.219 7jrxquabdwlgn wyjnoxtyxdryvffjbVdjw
BANNED=72.232.67.219 umesjoxmwrwdvjeqmfsreYfenxqel6d
BANNED=72.232.67.219 kdxiqiyu3yicfupymhimbp nlb v oghtqre
BANNED=72.232.67.222 henlvvdiranneq0cddlfdiXeivbwylon bxic
BANNED=72.232.67.222 vdpPPvxlkwmwpPyy8gpshni8y dwe q8lewlhfl
BANNED=72.232.67.219 didII6ye6It wermhvcx 6jmwcblyxj
BANNED=72.232.67.219 jevltpcioxefrooirvcumd
BANNED=72.232.67.222 r8nawcyepuDfymmbdi8xdsfah8sfqkwhuy eu

Why were they banned?

On a different day they claimed to be this:

72.232.67.221 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.219 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.220 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.222 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"

Reverse DNS claimed it was galaxy-webhosting.co.uk which is in Layered's IP block.

Now, for your amusement, here's the same IP within an hour trying more than one user agent:

00:14:06 72.232.67.222 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
00:52:07 72.232.67.222 "plblilwkchhs2qfkv rbXbgveu xsxwsxauspuX"

Hey, if one user agent doesn't work, spin the roulette wheel, right?

Sorry idiot, NO user agents work on my site, so let's move along.

OK, at least this one was creative, someone decided to explain it was a User-Agent:

72.232.77.34 "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;"

Just a few garden variety scrapings:

72.232.52.58 "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)"
72.232.185.170 "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"

Then another random attempt on .58 from above:

72.232.52.58 "qyerqcuylypknmarpuoudyeawwft"
72.232.52.58 "blcrnhbulYypqfbtasciqc"
72.232.52.58 "pwifascemkr4abimihq4ybhbusv"
72.232.52.58 "hbcudylrrturxtxwtMhoqq9sMsr uw pfM"
72.232.52.58 "brn jxvcgitdurvqhivtrhthtknu"
72.232.52.58 "fxddbq qxduqghdpbdgnptqrCtioive"
72.232.52.58 "jni0 kjn0flJxuenr0oek0b0rpjx"

It's just so cute that they fucking don't get it, random user agents or valid user agents, you just keep knocking but you can't come in and play so piss off.

Another proxy event that I banned:

72.232.20.146 "Mediapartners-Google/2.1"

Legit spiders crawling outside their range just scream "BLOCK ME! PROXY!", gotta love it.

Then this idiot thought no user agent would work, WRONG!

72.232.13.2 ""

... and a bunch more IPs doing stupid shit, but I'm too lazy to list 'em all here

Word to the wise, it looks like a scraper haven over there so consider blocking it.

According to their web site it looks like all server hosting so probably safe to block the whole range, but they have provided some amusement with their vaudeville scraper show thus far so maybe I'll just keep an eye on them for now and see if they come up with something new to toss at the bot blocker.

Circle Jerk Scraping

Not sure what's going on with some of the scrapers exactly but it seems my "CHALLENGE" page, which has about 5 links on the page, is sending them in circles asking for the same pages over and over.

Lately I'm seeing a lot of lost scraper bots doing something like this:

GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
... and so on and so forth...

Sometimes they'll ask for hundreds of pages like this before breaking out of that loop.

Quite amusing that I seem to have put a kink in their gears.

Scraping is fucking COMCASTIC!

That's right, most of the assholes abusing my website are using COMCAST!

Let's explore what happened today:

• CHALLENGE: 68.44.91.83 [c-68-44-91-83.hsd1.nj.comcast.net.] requested 229 pages as "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"
• SPIDER: 70.88.200.205 [70-88-200-205-measured-progress-ne-ma.hfc.comcastbusiness.net.] requested 75 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
• CHALLENGE: 68.83.187.18 [c-68-83-187-18.hsd1.nj.comcast.net.] requested 59 pages as "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"
• CHALLENGE: 24.63.11.72 [c-24-63-11-72.hsd1.ma.comcast.net.] requested 25 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
• CHALLENGE: 24.0.49.82 [c-24-0-49-82.hsd1.tx.comcast.net.] requested 44 pages as "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+98;+Win+9x+4.90)"
• SPEED: 68.53.108.148 [c-68-53-108-148.hsd1.tn.comcast.net.] requested 477 pages as "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)"

Let's explain what the reasons were why they were blocked which is the FIRST WORD shown before their information.

CHALLENGE means that after 20 pages downloaded there was a challenge put out that a human could answer but the computer didn't, and it just kept on crawling getting challenges page after page.

SPIDER means they stepped in a spider trap or read robots.txt, something humans don't do.

SPEED means just that, the pages were downloaded faster than Superman could click on them, even with a Kyptonite hangover.

Add it all up and thats 909 pages of COMCASTIC scraping!

That's 1/3 of the abuse to my website coming from a single internet service provider.

Fucking lovely, is it Scraping OnDemand too or is it High Definition Scraping?

Just take your fiber optic cables and give yourself a colonoscopy.

Omni-Crawler now USEFUL?

I was just looking around to see if there was any change in the status of a few old crawlers that never seemed to do anything useful, and suddenly Omni-Crawler's web site claims they actually have a customer!

Holy crap, someone uses it?

They say that their little crawler fuels Vast which looks like a pale Oodle wannabe.

Well guess what?

I don't have any classifieds on my site so leave the shields up set omni-crawler on condition red.

Would hosts care?

What if every time you had an attempted scraping your bot blocker sent an email detailing the specifics to the abuse dept. of the company hosting the scraper?

Would they even care if it weren't for the sudden flood of automated messages?

I'm thinking it might be amusing to give it a shot and see if they block any of these idiots.

Scraping My Big Pipe

And I'm not talking about shaving my cock either, I'm talking about some scrapers running via bigpipeinc.com. From their web site it all looks like pro pipelines but I'd hate to whack an entire network just because of one business connection behaving badly.

Anyone else know anything about these guys?

Can't disable it even to upgrade, this is CRAZY!

To easily performs some upgrades I took my bot blocker offline for a couple of hours. This way I could update the tracking database and software without worrying about accidentally killing the site. Then I completed installing the latest changes and turned the bot blocker back on and the site was running slower than shit.

My first reaction was panic that I'd busted something big time but my local version was faster than hell, so I looked into the current server accesses .... holy crap!

Guess what, while the bot blocker was offline for less than 2 hours, some asshole came and overloaded the fucking server when I wasn't watching. The running task list was enormous, the number of concurrent page requests was HUGE!

I punted the asshole's current tasks off the server and the bot blocker took over from there and restored calm to the website.

Seriously though, it's pretty scary when I can't even disable the bot blocked for a brief time without all hell breaking loose. Kind of reminds me of someone I knew that got hacked in less than 5 minutes while trying to replace a firewall with the machine still stupidly connected to the net.

Guess next time I'll either have to work faster or come up with an update plan that has no down time.

More work, less blogging, less forums

Yup, it had to happen, and people obviously noticed as I'm getting PM's here and there "nice to see you posting again" or "where have you been hiding?". Not like I ever went away, just doing work on trying to make my stupid bot blocker actually see the light of day and doing some R&D with a couple of new concepts that have been very promising.

So if you think I'm slowing down, it's just the opposite, as things are seriously HEATING UP!

If you catch me posting too much, smack me and tell me to get back to work ;)