Scrapers Switching Bot Software

Here's another example of a scraper that switched software. This time the scraper went from something using random user agents to some other bot that's detectable because of the "compatible ;" flaw, note the space before the semicolon, in the user agent making it easily blockable.

07/13/2006 12.30.91.69 "4jkjhm4lgkualkihgfm nSriexn"
07/13/2006 12.30.91.69 "gu EeoiEmwbrqufoktaitgirh3sso3Ex"
07/13/2006 12.30.91.69 "yjiu k4smuqlvj4qreip l em4vjngywrv"
07/13/2006 12.30.91.69 "tfdgevSbyefsSoevhrrr"
07/14/2006 12.30.91.69 "c xqxrwgt kfrod oUwxmqxbooewtUrxgUplr"
07/14/2006 12.30.91.69 "agnesmynnlihdiunsutxxn5skoY5jsgmx"
07/14/2006 12.30.91.69 "aqtcc2dfklcrdymQQlhqclpcx2km2"
07/14/2006 12.30.91.69 "nBnwsmracuwd7ovdnmgnora"
07/14/2006 12.30.91.69 "vuqWfxtekvxi8relwfx8ejrto"
07/14/2006 12.30.91.69 "9fxebywuwjbpdbfnesfvpqygondkiqtrfdkaskj"
07/14/2006 12.30.91.69 "mst xfwkpktrkfymy2owm wu2"
07/14/2006 12.30.91.69 "xdgdmncxnhjqrvudftxnyrqwqyfiecdclqpmg"
07/14/2006 12.30.91.69 "5sieornr5ksjimfykxyoimyyfuedthnyuuijeb"
07/14/2006 12.30.91.69 "vhryqjhtkmpysfwhmrfcfotgkkkvQdjvtdgyr h"
07/17/2006 12.30.91.69 "ahBuu0xghlmhxaketqo0jjuyxxqxugilvtciso"
07/17/2006 12.30.91.69 "modJjnbqrprdhbwJcpohw prj4"
07/22/2006 12.30.91.69 "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"
07/22/2006 12.30.91.69 "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"

This is a prime example of why I advocate tracking where the little scumbags operate opposed to just blocking by user agent as this cable client might upgrade to something even better and slide under my radar next time.

Another one I spotted switched from something in Java to the "compatible ;" bot:

62.194.21.116 Java/1.4.1_04
62.194.21.116 "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"

I'm sure there's more, still compiling the results, should be interesting.

Huge Yahoo SERP Upheavals or just a Tweak?

The other day I was posting a about my position in Yahoo for the silly term "Lava Pockets" and suddenly today it shot all the way down to #133. I see our old snack food friend Hot Pockets is still holding strong at #20 but they dropped the hammer on my ass.

Could it be they got pissed at my post and gave me a hand job?

Probably not because I started looking around and noticed someone on WebmasterWorld posted today about many blogs appear to have plunged suddenly in Yahoo and others suggested it was a tweak. Not sure it's just a tweak as some of my SERPs not related to any blogs moved up and down all over the place.

Looks like Yahoo has installed the new much anticipated "amusement park" algorithm as that's the only explanation I can find for the constant rollercoaster results this month.

Stay tuned as Yahoo's next algorithm is rumored to be tied to their stock price.

Robots, Spiders and Dumbass Delphi Clients, OH MY!

Never heard of this one but out of the blue someone in France hit me with TALWinHttpClient which is some Delphi plug-in and the French must be the only ones still clinging to Delphi as nobody worth a hookers time still uses that shit.

86.202.105.134 [ANice-152-1-99-134.w86-202.abo.wanadoo.fr.] requested 1 pages as "Mozilla/3.0 (compatible; TALWinHttpClient)"

Figures, something comes crawling from France and puts up a white flag after just one page.

C'est la vie.

VIVA LA GUERRE DES BOTS!

Double Agent Trapped by Bot Blocker

The title wouldn't be so funny except this double agent originated from the Defense Contract Management Agency (DCMA).

Here's the user agent string that caused the problem:

144.183.226.xxx [xxx.dcma.mil.] requested 2 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 3.3; Windows NT; DCMA 5.5 r.2.0); .NET CLR 1.0.3705; .NET CLR 1.1.4322; InfoPath.1)"

Anyone notice the double agent in the string?

It was the second "Windows NT;" without the version number, which is technically invalid, that caused them to get snared.

Oopsy, military intelligence stikes again.

Just hope whoever did that doesn't contribute code to the missile defense systems!

MetagerBot powered by Exalead - First Look

Looks like I'm on the bleeding edge with MetagerBot as there's almost a void of information about this bot that comes calling from the University of Hannover.

130.75.2.12 [kursix.rrzn.uni-hannover.de.] requested 1 pages as "MetagerBot/0.8-dev (MetagerBot; http://metager.de; )"

The part that I'm confused about is what do they need their own bot for as they're claiming to be a meta search engine powered by Exalead?

Some references indicated they were using [big shock] nutch:

metagerbot/0.8-dev-mnebel-20050827 (mgbot; http://metager.de/; __nutch-agent@metager.de)

Played around with their search for a bit and it did cough up a few new things of interest I hadn't encountered, but it was slower than shit most of the time and waiting on results can be a bit tedious.

So does this mean we should let Exalead crawl?

No clue.

StupidHijacking

Here's a simple example of how the proxy sites hijack listings.

72.51.33.237 [server1.stupidcensorship.com.] requested 2 pages as "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Yes, that was really Googlebot that tried to crawl my server via the proxy hosted on that server.

Amazingly you won't see any links for Google to crawl my site or any other site via this proxy server which is why I always assume there is a cloaked list of URLs being fed to Google as this happens all the time from all sorts of proxy sites of this type.

Cell Phone ALMOST passed the test

With the myriad of cellphone user agents it's almost impossible to track them but this one ALMOST got thru with a minor exception:

The user agent was:

"Dopod818Pro/Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; Dopod818 Pro)"

If they hadn't redundantly put "Dopod818Pro/" on the front it would've passed thru to the website.

Oh well, better luck next time.

ActiveTouristBot - First Look

Welcome to the wonderful world of dumb fucking robots that haven't a goddamn clue.

Maybe ActiveTourist was attempting referrer spam as I have no idea what this digital dungheap was actually doing.

See for yourself:

80.229.187.190 -"GET /robots.txt" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /robots.txt" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /robots.txt" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /robots.txt" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 -"GET /AnActualFuckingPage.html HTTP/1.1" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /AnotherFuckingPage.html" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /TheFuckOffMyServer.html" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /YourHeadOutOfYourAss.html" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /robots.txt" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /robots.txt" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /robots.txt" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /MoreRobotsTxtFilesYummy.html" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /MoreShitYouArentGetting.html" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"
80.229.187.190 - "GET /AFuckingClue.html" "Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com)"

This spastic fucking bot eats robot.txt files like they're candy.

Didn't find any specific information on that tragedy they call a website about what they even support in robots.txt, but it doesn't really matter as they got nothing from me except a place of honor on my blog.

Hooters Casino Crawling or Breached?

This is the same exact scenario that I saw with the Tireman site and several others.

All dedicated servers, yet the SAME crawler with the exact same behavior is operating from this server:

24.120.182.161 [mail.hooterslv.com.] requested 29 pages as "Java/1.4.2_06"

I don't think these are a coincidence as it's starting to look like a botnet of some sort, perhaps spambot scrapers, no clue what they're looking for at this point.

SEO'd myself into being an expert geologist

According to the search engines Google #1, MSN #1, and Yahoo #2, I'm an expert geologist on the topic of Lava Pockets at the time of this posting.

Guess this pretty much blows the theory about search engines picking authority sites on a topic based on trust as I don't know shit about lava yet rank high for something that was completely silly rant and off topic for geology. More amusing is that I mentioned Hot Pockets in that blog post and now it's been dragged into the listings currently at #3 on Google, #2 on MSN and #20 on Yahoo.

Just goes to show how easy you can get attention on some lesser fought after keywords without really trying but you can bring along friends like Hot Pockets just by referencing them.

FWIW, I'm still amused that MSN still finds me the number one resource on sex education and they send constant inquires from the middle east, which brings up other issues, but that's for another rant. Not like I tried to get them in the first place, but I lost some of my positions on those lovely terms, but a few of them are still in the top 10.

My mother would be so proud...

Not.

Scraping via Google Translator

The other day I was tightening up bot blocker security just a bit to not only verify requests are coming from the Google IP range but specifically which bots were asking for information instead of the carte blanche approach that "If it's from Google, it must be good" which was a bullshit assumption.

Sure enough, I found something crawling my site at a pretty good pace today and it was someone using the Google Translator to scrape AND translate my site all at the same time.

Isn't that amusing!

Pretty sure it wasn't any type of Googlebot as it didn't ask for robots.txt and requested things like "/#top" which Google doesn't try to crawl, nor would a human in a browser send that request, so it's a bad bot using a loophole.

So follow along kiddies to what I've done to date:

• Locked Googlebot access by known ranges of Google IPs to stop Googlebot spoofing
• Installed NOARCHIVE to stop scraping via Google's cache index
• Blocked PROXY servers when Google comes crawling through one to avoid page hijacking
• Tightened security to specifically look for Googlebot or Mediapartners only to avoid nonsense via the web accelerator or other nonsense services they provide

Then, after ALL THAT, I find Google has yet another vulnerability which is the translator, which has probably been used to scrape me for months now, and they dont seem to care when someone is asking for pages at 1 second or less per page either.

What a joke Google, what a joke...

This is why I keep ranting about PROXY servers being bad, yet ANOTHER example of how any type of proxy, which in effect is what Google translator is, can be exploited.

How can I prove to you it's a bot?

When bad behavior is detected my bot blocker will CHALLENGE the requests with a captcha of some sort, might be a simple one, might be a hard one, but this crawler via the translator asked for 159 pages which, up to a point, were all unanswered captchas, then messages about being blocked for bad behavior, and it still kept going asking for different pages one after the other at a rapid pace.

CHALLENGE: 64.233.178.136 [hs-out-f136.google.com.] requested 159 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1),gzip(gfe) (via translate.google.com)"

Now some of you might point out that it could've been a lot of people going thru the proxy server at the same time trying to translate pages. That's easy for me to refute as I track the proxy information, if present, when I log bogus page requests and most of them came from the same IP address in Brazil.

CHALLENGE 64.233.178.136 "Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.7.8) Gecko/20050511 Firefox/1.0.4,gzip(gfe) (via translate.google.com)"

Proxy Detected -> VIA=1.0 translate.google.com (TWS/0.9), 1.0 proxy.google.com:80 (squid)

FORWARD=201.35.249.163

nslookup 201.35.249.163
name = 201-35-249-163.bnut3703.dsl.brasiltelecom.net.br.

On top of all that, it looks like Google jacks up my javascript in the captcha when they run it thru the translator so if a legitimate visitors, unlike the crawling asshole from Brazil, does something that invokes a challenge you're just fucked as you can't break out.

Oh joy, more shit to debug.

Thank you Google.

You know what's real fucking hysterical about Google breaking my javascript captcha?

The cheap ass CGI proxy servers run by kiddies trying to get to MySpace from school don't even break my javascript, so this is truly some PhD worthy software that broke my shit.

FYI, I asked Matt Cutts to pony up the actual IP's of Googlebot so I could be more precise and his answer was:

IncrediBILL, I don’t think we’ve done so in the past because it changes from time to time, and we didn’t want to give bad/stale information.

Earth to Google, just post the damn IP list for all your crawlers and those of use using it for security will worry about updating our sites. Maybe you should include new IPs with a lead time like 7 days in advance to give everyone a chance to update. Put the list in an XML file and we can automate updating our security, not a problem, really, as it's better than letting idiots scrape my site via your swiss cheese security on your translator!

DIT - Dissident Internet Technology

Don't get me wrong, I have no problem with the Chinese people having open access to the internet, but not via sneaky assed cached DIT proxies and not on my server.

If you want internet freedom you can a) move elsewhere like many of my Chinese friends and neighbors or b) overthrow your damned government and stop poking around in firewalls like children trying to bypass NetNanny.

Not like we have any internet freedom as the fucknuts in Washington state passed a law about online gambling that can have your ass tossed in prison just for WRITING about online gambling, not even doing it, so we have our own internet problems and politicians tossing out the 1st Amendment that need overthrowing right here at home.

The proxy details:

66.98.206.97
[ev1s-66-98-206-97.ev1servers.net.]
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1), DynaWeb http://www.dit-inc.us/disclaimer.php

The amusing part was, unless it was spoofed, the IP being forwarded was 65.59.219.179 which belongs to Level 3 Communications, which certainly isn't in China unless it was yet another proxy server relaying from there.

Sorry people, I know it's harsh, but bad shit goes down via proxies and Chinese IP's have knocked my server offline as long as 90 minutes at a time asking for hundreds of pages a second.

Not that I don't have sympathy for their situation, but I have NO fucking sympathy for anything that has potential to enable my server going offline.

A Daily Rant from a kindred spirit

Looks like somewhere out in the internet someone else has A Daily Rant that is also pissed about bots. For those of you doing it the old fashioned way and still blocking bots one at a time, he provides snippets of code for your antiquated .htaccess files and has a big list of bots to block ready to install.

Maybe some day he'll have an epiphany and come around to the whitelist method and stop wasting his time with the endless list of bots that just don't stop with a new bot found almost daily.

The site has some good information so I think you'll all enjoy it but he needs more cursing, it's pretty tame ;)

1&1 Web Host Goes Spamming

When you want some bad publicity just send some spam and you'll get all you need.

For starters, I'm not in any opt-in, double opt-in or any other kind of bullshit spam list unless someone stuck my name in there. I despise spam so whatever excuse they make about how they sent me this spam is a FUCKING LIE whatever it is so don't even bother trying to post excuses if you read this as I NEVER signed up, no way, no how, not possible, and you spamming morons can kiss my big fat white hairy ass.

It appears this email came from some SPAM operation called DesertWalls hosted on Interland.

Interland.Return-Path:
Received: from desertwalls.com (66.33.17.22)
From: "eMarketing" <SPAMMING_FUCKWADS@desertwalls.com>
Message-ID: <fuck_you_desert_walls@desertwalls.com>
Subject: Exclusively from the World's Largest Web Host
X-Complaints-To: SPAMMING_ASSHOLES@desertwalls.com

This is 1&1's bullshit ad:

The ad was originally loaded from:

http://69.59.147.20/1and1ad4.jpg

When you clicked the ad it went to a tracking script:

http://desertwalls.com/t/xxxxx/xxxxx

Which was redirected here:

http://order.1and1.com/xml/order/HostingHome

Which is owned by 1&1:

domain: 1and1resources.com
registrant-firstname: Thomas
registrant-lastname: Vollrath
registrant-organization: 1&1 Internet Inc.
registrant-street1: 701 Lee RD
registrant-street2: Suite 300
registrant-pcode: 19087
registrant-state: PA
registrant-city: Wayne
registrant-ccode: US
registrant-email: support@1and1.com

Don't know if this was one of their affiliates and I really don't fucking care but it looks like 1&1 sent the email as this was at the bottom:

If you no longer wish to receive messages from 1&1 Internet, please send a blank email with the single word "Unsubscribe" in the subject line to remove@1and1resources.com. Make your mark on the web with 1&1, the only web hosting company trusted by over 5 million people worldwide.

If 1&1 didn't send it then it would be too obvious they had an issue with the affiliate when all the unsubscribe requests started pounding their server, so it's probably them.

So take your webhosting service and shove it up your spamming ass.

Thank you.

Honey Pots are History - Fix Your Site or Block Bots!

There's a really cool anti-spam site up called Project Honey Pot that seems to be snaring a ton of spambots and identifying where the spammers crawl from, spam from and what they are spamming. However, the problem with this valiant but slightly misguided effort is that it assumes you can stop spambots from harvesting if you can out them, which blew up in Blue Security's face.

The true way to stop spambots isn't trying to feed them honey, it's fixing your website so there isn't any email addresses for them to harvest in the first place. Wow, isn't that a revelation, no email address on the page, nothing to harvest, end of spambots.

Two simple ways to accomplish the end of spambots are CONTACT FORMS and BOT BLOCKERS. If you purge your site of all email addresses and use a secure form with a captcha to stop spammers, then you should eliminate email harvesting and form spamming in one shot. Combine this with a bot blocker that stops all the vermin from crawling your site in the first place and you've got a 2-pronged strategy that should stop them cold.

Not that honey pots aren't cute as they snare these idiots, it just makes more sense to EDUCATE webmasters to avoid publicly displayed email addresses in the first place and stop bots from crawling to end the spambots once and forever. Anything else, like a honey pot, is just patching around the problem and not putting a permanent stop to the disease.

NO EMAIL ADDRESSES SHOULD BE DISPLAYED ON WEB PAGES EVER!

GET IT?

GOT IT?

GOOD!

Now move on.

Gaisbot crawls via Yahoo China

I've been waiting for an excuse to block Yahoo China and this shit tips the scales.

202.165.96.134
[r2.mk.cnb.yahoo.com.]
"Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)

What the fuck Yahoo?

Why is GAIS Labs of National Chung Cheng University crawling from your IP block?

This is BULLSHIT!

Tireman crawling or breached?

Here's one that makes no sense to me as it appears the Tireman Auto Centers came crawling today.

72.240.44.21 [thetireman.com.] requested 29 pages as "Java/1.5.0_07"

It appears to be a dedicated from from ev1servers.net which is a hot bed of trouble anyway.

ev1s-207-44-205-180.ev1servers.net

Another reason I block anything from ev1servers.net

Rightscripts are Leakey

Some UK script kiddie Leakey Yung has a website called Rightscripts where he sells a bunch of bullshit scripts like Extract Website, just what we need, along with some SEO tools with live demos you probably don't want aimed at your site.

Their server is:

rightscripts.com (64.202.163.153)

If they try to run the demo from their server it shows as a different IP:

64.202.165.132
wc02.inet.mesa1.secureserver.net
"" [blank user agent]

This mess runs on GoDaddy servers:

OrgName: Go Daddy Software, Inc.
NetRange: 64.202.160.0 - 64.202.191.255

I've seen them appear to use 2 IPs for a proxy also from GoDaddy twice on my site:

64.202.165.201
"Rightscripts"
Proxy VIA=1.0 wc04.inet.mesa1.secureserver.net:3128 (squid/2.5.STABLE12) FORWARD=64.202.163.153

64.202.165.131
"Rightscripts"
Proxy VIA=1.0 wc01.inet.mesa1.secureserver.net:3128 (squid/2.5.STABLE12) FORWARD=64.202.163.153

Then of course, there appears to be some asshole that actually bought this script that aimed it at my site:

65.254.32.58
[sandiego.hostingforme.com]
"Rightscripts"

Of course "hostingforme" is a bullshit shell page that says nothing, another shocker, owned by some idiot scraper from India.

hostingforme.com (72.9.235.194)

Domain Name: HOSTINGFORME.COM
Registrant:
Ankur Motreja
399, 5th cross,
Gokulam III stage
Mysore, Karnataka 570002
India

Reinforces why I blocked GoDaddy's hosting farm in the first place and lock out all these idiot scripts by default.

Juniper Networks Security Lab

Just about once a day something Juniper Networks has been playing with comes and asks for a page or two and goes away. Never asks for robots.txt or anything like that so no clue if it's a crawler or what.

208.223.208.181
[security-lab1.juniper.net.]
"Python-urllib/1.16"

Perhaps it goes away when the user agent is blocked and it gets the "go away you're bothering me" page, but it will come back again and again and has been doing so for months.

If anyone from Juniper wants to comment on what in the hell your "security lab" is doing, perhaps I'll let it pass thru if it's a worthy cause. However, considering they aren't even bright enough to change the default user agent, forget that last thought, we'll just keep blocking it.

First Look - MaSagool spider

Some search engine called Sagool from Japan [translated] sent a spider caledl MaSagool that came crawling from 2 IP addresses. Didn't look for robots.txt or any other damn thing so probably a good idea just to block this thing until they teach it some manners.

59.158.235.21
[59x158x235x21.ap59.ftth.ucom.ne.jp.]
"MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp)"

124.32.246.45
[124x32x246x45.ap124.ftth.ucom.ne.jp.]
"MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp)"