Call me confused, but why is MyFamily.com crawling my site?
Yup, the genealogy people, that's them.
66.43.16.199 [nat.myfamilyinc.com] has requested 110 pages as Mozilla/4.0 (compatible; MyFamilyBot/1.0; +http://www.myfamilyinc.com)Even claims to be them, looked at robots.txt like they were supposed to, yada yada.
I'm not sure if any of you have ever seen John Gabriel's Greater Internet Fuckwad Theory but take a gander, quite amusing.
Just in case one of you thinks "Well, that explains Bill in a nutshell!" then think again as I've always been very vocal, blunt, and to the point.
Doesn't mean I'm not a fuckwad, but the internet had nothing to do with that!
Yup, that's right, someone decided we need to hide on the net so much they actually created a Proxy Web Host and it appears scrapers are using them, what a shocker.
I found them as AdSense was trying to crawl thru the proxy:
BAD_AGENT: 72.232.31.226 [prx1.proxywebhost.com.] requested 3 pages as "Mediapartners-Google/2.1"So who are these people using for their provider?
Layered TechnologiesThat's a HUGE block of IP's to just block out of hand, so how much abuse has been coming from this range? Let's search on "72.232." and see what pops up.
72.232.0.0 - 72.232.255.255
BANNED=72.232.67.222 yrkqi3jrmnbrsk3mUpnrwungWhy were they banned?
BANNED=72.232.67.219 vhsflbuwvLsbwmyp8xse8hvpdpplLdxdx
BANNED=72.232.67.219 utgkm gylmugtdblyppqqu
BANNED=72.232.67.220 pt tkglswaqatq k rfxqolbtqbygxlhvS0qqv
BANNED=72.232.67.221 djpqaegrbxpfbqnkxvqeniqfogyb rnt
BANNED=72.232.67.221 wbdprvjiqbw jbsvqse7
BANNED=72.232.67.220 upehrsqqqevdljtwrgkkbthk e
BANNED=72.232.67.220 sjxgohdtum3yybmbfyembisbxibei
BANNED=72.232.67.219 7jrxquabdwlgn wyjnoxtyxdryvffjbVdjw
BANNED=72.232.67.219 umesjoxmwrwdvjeqmfsreYfenxqel6d
BANNED=72.232.67.219 kdxiqiyu3yicfupymhimbp nlb v oghtqre
BANNED=72.232.67.222 henlvvdiranneq0cddlfdiXeivbwylon bxic
BANNED=72.232.67.222 vdpPPvxlkwmwpPyy8gpshni8y dwe q8lewlhfl
BANNED=72.232.67.219 didII6ye6It wermhvcx 6jmwcblyxj
BANNED=72.232.67.219 jevltpcioxefrooirvcumd
BANNED=72.232.67.222 r8nawcyepuDfymmbdi8xdsfah8sfqkwhuy eu
72.232.67.221 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"Reverse DNS claimed it was galaxy-webhosting.co.uk which is in Layered's IP block.
72.232.67.219 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.220 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.222 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
00:14:06 72.232.67.222 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"Hey, if one user agent doesn't work, spin the roulette wheel, right?
00:52:07 72.232.67.222 "plblilwkchhs2qfkv rbXbgveu xsxwsxauspuX"
72.232.77.34 "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;"Just a few garden variety scrapings:
72.232.52.58 "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)"Then another random attempt on .58 from above:
72.232.185.170 "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"
72.232.52.58 "qyerqcuylypknmarpuoudyeawwft"It's just so cute that they fucking don't get it, random user agents or valid user agents, you just keep knocking but you can't come in and play so piss off.
72.232.52.58 "blcrnhbulYypqfbtasciqc"
72.232.52.58 "pwifascemkr4abimihq4ybhbusv"
72.232.52.58 "hbcudylrrturxtxwtMhoqq9sMsr uw pfM"
72.232.52.58 "brn jxvcgitdurvqhivtrhthtknu"
72.232.52.58 "fxddbq qxduqghdpbdgnptqrCtioive"
72.232.52.58 "jni0 kjn0flJxuenr0oek0b0rpjx"
72.232.20.146 "Mediapartners-Google/2.1"Legit spiders crawling outside their range just scream "BLOCK ME! PROXY!", gotta love it.
72.232.13.2 ""... and a bunch more IPs doing stupid shit, but I'm too lazy to list 'em all here
Not sure what's going on with some of the scrapers exactly but it seems my "CHALLENGE" page, which has about 5 links on the page, is sending them in circles asking for the same pages over and over.
Lately I'm seeing a lot of lost scraper bots doing something like this:
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
... and so on and so forth...
Sometimes they'll ask for hundreds of pages like this before breaking out of that loop.
Quite amusing that I seem to have put a kink in their gears.
That's right, most of the assholes abusing my website are using COMCAST!
Let's explore what happened today:
I was just looking around to see if there was any change in the status of a few old crawlers that never seemed to do anything useful, and suddenly Omni-Crawler's web site claims they actually have a customer!
Holy crap, someone uses it?
They say that their little crawler fuels Vast which looks like a pale Oodle wannabe.
Well guess what?
I don't have any classifieds on my site so leave the shields up set omni-crawler on condition red.
What if every time you had an attempted scraping your bot blocker sent an email detailing the specifics to the abuse dept. of the company hosting the scraper?
Would they even care if it weren't for the sudden flood of automated messages?
I'm thinking it might be amusing to give it a shot and see if they block any of these idiots.
And I'm not talking about shaving my cock either, I'm talking about some scrapers running via bigpipeinc.com. From their web site it all looks like pro pipelines but I'd hate to whack an entire network just because of one business connection behaving badly.
Anyone else know anything about these guys?
To easily performs some upgrades I took my bot blocker offline for a couple of hours. This way I could update the tracking database and software without worrying about accidentally killing the site. Then I completed installing the latest changes and turned the bot blocker back on and the site was running slower than shit.
My first reaction was panic that I'd busted something big time but my local version was faster than hell, so I looked into the current server accesses .... holy crap!
Guess what, while the bot blocker was offline for less than 2 hours, some asshole came and overloaded the fucking server when I wasn't watching. The running task list was enormous, the number of concurrent page requests was HUGE!
I punted the asshole's current tasks off the server and the bot blocker took over from there and restored calm to the website.
Seriously though, it's pretty scary when I can't even disable the bot blocked for a brief time without all hell breaking loose. Kind of reminds me of someone I knew that got hacked in less than 5 minutes while trying to replace a firewall with the machine still stupidly connected to the net.
Guess next time I'll either have to work faster or come up with an update plan that has no down time.
Yup, it had to happen, and people obviously noticed as I'm getting PM's here and there "nice to see you posting again" or "where have you been hiding?". Not like I ever went away, just doing work on trying to make my stupid bot blocker actually see the light of day and doing some R&D with a couple of new concepts that have been very promising.
So if you think I'm slowing down, it's just the opposite, as things are seriously HEATING UP!
If you catch me posting too much, smack me and tell me to get back to work ;)
Don't know if anyone has ever seen this script vulnerability before but it just happened to me and I almost shit myself.
Who would ever consider that pulling up your log analyzer summary page could fuck you over?
Luckily for me, it was just a simple redirect:203.214.25.21 "<SCRIPT>window.location='http://www.syncrisis.com'</script> (compatible; MSIE 6.0; Windows NT 5.1; SV1; iP-CMPQ-2003; .NET CLR 1.1.4322)"
These fuckers better watch out cause this is fucking BULLSHIT and you're on my shitlist now you cocksuckers.
Something new to add to the bot blocker and just bounce assholes that do this shit.
Remember a while back I predicted that bots would evolve to use random IPs and random user agent strings and it's already happening. Good thing I had already planned for this contingency and was ahead of the curve waiting for them when they tried this trick.
Not 100% positive, but it looks like it all kind of started with this very prolific bot called T8Abot/v0.0.7-beta (3724461@gmail.com) which seems to get around.
The most I can find about it without wasting too much time is this reference:
unknown bot, hosted by FDC Servers, fdcservers.net, US. Massive operation using many IP addresses (66.90.110.199 ... 66.90.110.254)That bot seems to have gone away, at least it doesn't visit my site anymore, maybe they noticed it was being blocked by user agent string.
06:44:13 66.90.95.225 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
06:44:32 66.90.95.249 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:45:44 66.90.95.243 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:45:49 66.90.95.249 "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1"
06:45:53 66.90.95.249 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
06:45:54 66.90.103.69 "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1"
06:45:59 66.90.95.225 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.54 [en]"
06:46:07 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:46:10 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:46:19 66.90.103.69 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Media Center PC 3.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
06:46:27 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50"
06:46:28 66.90.95.243 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:46:37 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:46:39 66.90.103.69 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:46:40 66.90.95.216 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:46:41 66.90.95.211 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; ru) Opera 7.60"
06:46:43 66.90.95.211 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.54 [en]"
Just for giggles I tweaked the parameters of the bot blocker a few weeks ago to see if I could snare any super slow crawlers trying to fly under the radar.
So far, the grand prize winner has been scraping a few pages daily for 13 days now for a grand total of 1567 pages.
What a loser, sheesh.
Unfortunately, it looks like this latest change has snared about 20-30 innocents so we'll be retooling it today, but I learned a lot about slow scrapers during this process and it was quite an eye opener.
Don't know if anyone has been following the trials and tribulations of Blue Security and their anti-spam tool the Blue Frog with over 400K members, but their technique could be useful against scrapers. Basically what Blue Frog does is automatically send opt-out requests to spammers from every member in their network which overloads the spammers servers and sends them crashing or maybe even blocks their ability to spam while the pipeline is choked with inbound requests.
So this concept stewed in my brain for a couple of days and suddenly it occurs to me that bots could be stopped the same way with a large enough network of anti-bot members. The concept is simple in that any active crawlers caught in the act could simply be pinged to death for a brief period to stop them with brute force. That's right, we could aim thousands of servers at unwanted active crawlers and literally DDoS them into leaving us alone.
Worse case, they aren't crawling when their pipeline is choked.
Best case, they take a hint and stop hitting member sites.
That method in itself COULD make an entire community of networked webmasters deal death blows to scrapers, blog spammers, and all sorts of nasty vermin.
Hell, why not fight fire with fire and use the tools of the underground against them?
If you can locate a scraper's site we could also just deploy spiders to simply crawl them offline. That's right, do the same thing they do and crawl them so hard and fast the damn server can't even respond to requests. Hit them with so many different servers at the same time they can't even identify who's doing it in time to stop it.
Remember, if visitors can't get to your web site then the purpose of scraping to build the website becomes meaningless.
Here's where I think this strategy might pay off as scrapers on shared servers just might get the boot when the host figures out that they're the reason for the attacks. Likewise, colo facilities might even boot dedicated servers when a network of unhappy webmasters retaliate and choke a major hosts data pipe repeatedly.
Does is sound like vigilanteism?
HELL YES IT DOES!
Why shouldn't we do it?
Nobody else is helping us as neither their web hosts, the copyright laws, Google AdSense nor Yahoo Publisher Network seem to be interested in helping us stop this plague so maybe it's time to deploy full blown internet warfare tactics like Blue Security in order to stop the madness!
In the immortal words of Mel the cook on Alice:
"The best DEFENSE is a good OFFENSE!"
