Crawlers of the Dumbed

OK, so I slapped some javascript in all my links just for shits and giggles to make sure it was REALLY a human and sure enough some dumb fuck is now attempting to crawl my site with "OnClick", "OnMouseDown", and other stupid shit as part of the URL.

My god the 12 year olds writing these crawlers aren't as smart as 12 year olds used to be.

FrontPage theft THWARTED

Lazy assed stealing webmaster used to just aim FrontPage at my site and lift a web page, steal layouts, whatever they did, but that was THEN.

THIS is NOW and my bot blocker slammed their head into the wall:

BAD_AGENT: x.x.x.x requested 1 pages as "MSFrontPage/4.0"

Too fucking bad, learn to code your own goddamn webpages you fucking freak!

Blogger ALSO Busted all to Hell

In more news of broken Google property, Blogger's edit mode has been FUCKED for a couple of weeks and I assumed someone would fix it but I was sadly mistaken.

It used to work just fine but most of the time now when I edit a previous post in Firefox or Netscape it just hangs as the Blogger code spins out of control and I have to kill the browser. The only way I can edit in those two browsers most of the time, starting a couple of weeks ago, is to copy the entire article into a new post, edit and save which seems to work.

When you edit in Blogger on Internet Explorer it's doing something completely different and chews up articles, trashes the HTML, jacks up fonts and all sorts of fun shit.

Basically, other than the original first time posting, using Blogger to edit in it's current state can be a death sentence to your post.

Based on all the crap posted in Blogger by illiterate people that can't even tell when they make actual spelling mistakes, let alone attempt to edit them, this problem probably hasn't even been reported as nobody else has noticed.

Funny, I don't see the word BETA on Blogger and if it was, it would be a piss poor Beta at the moment.

Google Exact Phrase Search is BUSTED ALL TO HELL!

Have you done any exact phrase searches in Google lately?

It's serving up all sorts of off topic shit, I mean, it's really busted.

Couple of examples of topics from this blog:

Do a search on "Lava Pockets" and you get crap liks this in listing #13:

Take Five - The Boston Globe(Correction: Because of a photographer's error, the names of two members of the Click Five were transposed in some edition's of Sunday's Arts ...
www.boston.com/ae/music/articles/2005/08/07/take_five/ - 41k - Supplemental Result - Cached - Similar pages

Neither the word LAVA or POCKET are even on that page

Even better is "Florida Drinks Cat Piss" with 99% of the results complete shit!

Let's examine the #2 result:

eg Forums -> Pretentious bottle--ordinary wineOutline · [ Standard ] · Linear+. Pretentious bottle - - ordinary wine. Track this topic | Email this topic | Print this topic ...
forums.egullet.org/index.php?showtopic=72214 - 71k - Supplemental Result - Cached - Similar pages

That's right, nothing on that page about Florida whatsoever but the irony of wine coming up doing a search for cat piss certainly didn't pass me by.

Hey Google, you even know about SQA?

That's Software Quality Assurance for those that don't know, and I can recommend some very good people if you need them down at Google.

Some freaky shit

OK, today when I was taking a shower a very loud noise happened in the bathroom, nearly jumped out of my skin. Looked out and the door was still shut, so the cat hadn't been in the room and the wife was at work.

When I got out of the shower it turned out a tall perfume bottle, one that has sat in the same spot for ages, just randomly fell over and bounced on the counter top.

What the fuck?

I immediately thought perhaps it was an earthquake jolt but the dining room chandelier wasn't swinging, and it swings for the mildest of jolts, so I'm not sure what's going on at this point.

Just what I need, ghosts in the crapper, nothing the aftermath of a good mexican lunch won't scare away.

Jews for sale!

Who'll start the bidding at 10 shekels, do I hear 15?

That's right, some dumb fuck didn't filter out the plural of kike on Google Adwords and some dumber fuck is now trying to sell them on eBay.

Oy vey!

Microsoft Promotes Web Theft

Microsoft is promoting a whole page full of offline browsers just to help people steal shit.

They gleefully promote it too:

Quickly and efficiently download entire sites or sections of a site for viewing and browsing when you are not connected to the Internet.

Just what we need is a major corporation sticking some shit like this in front of people's faces that may not even be aware such tools existed in the first place. That's right, America's largest software company has become the Pied Piper of scraping websites.

Any words of caution about copyright, that your IP may get banned for abusive practices or anything else?

Hell no.

Hey Microsoft, you gonna pay for my web hosting bill when the assholes that download all these offline browsers blow my fucking server out of the water with ridiculously high bandwidth abuse?

I didn't think so.

Well, fuck you Microsoft, just fuck you.

India Link Whores Just Don't Understand

It's the spring link season once again and all the India link whores are out doing their ritualistic website link mating dance. Unfortunately, the shit they want me to link to would be like trying to mate a monkey with a goat, wrong species pal, nothing going to come from this attempt to defy nature.

Here's a slightly modified version of the most recent link blackmail to conceal the identity of the clueless asshole:

Hi webmaster,

We have already added your site to our resource page
http://www.OFFTOPICBULLSHIT.com/ long back.

However, we just visited your site and we still do not find the reciprocal link to our site. If we do not found our reciprocal link we will remove your site link from our resouce page. Please confirm the url where the link is posted as soon as possible to allow us keep your link.

Title :- Spamming link whores at nominal rates.

Description :- We don't fucking understand that people don't like off topic links, link to us or we'll add your link to Jihad.com and then you'll be screwed

URL :- http://www.REALANNOYINGFUCKERS.com

We look forward to receiving this information promptly.

Thank
Stoopidja Dumbfukja
webmaster@IMFUCKINGCLUELESS.com

Not only was the site completely off topic, the whole fucking thing was ORANGE, even a significant amount of the text!

My eyes started bleeding the minute I saw that shit and I couldn't tell if my link was on their site or not, but it didn't really matter as I told them to take their link and shove it up their ass sideways.

Crawl my site, go to jail, it's the law.

Who the hell put open season on my website?

Not talking about this blog, but the website that pays my bills and shit.

It's NOT a free articles site, plainly posted copyright notices, technology is in place to stop assholes dead in their tracks and they just keep coming.

Did someone post a sign on the internet that I've never seen that says:
"HEY! FREE SHIT TO SCRAPE HERE! COME AND FUCKING GET IT WHILE IT'S HOT!"

The other day I wrote some dry ass bullshit about data mining and then followed it up with some boring assed statistics I've been collecting but suddenly something changed and the amount of thieves hitting the site is suddenly going off the charts compared to a month ago.

Let's put this into perspective:

If my website were a grocery store and these assholes were looters or shoplifters the shelves would be bare, the clerks would be guarding the doors with shotguns, the police would have the place surrounded and the parking lot blocked off and a riot squad would be firing rubber bullets and cracking skulls of people trying to get away with th loot.

The local jails, needless to say, would be overflowing.

I realize there is no physical theft involved like the grocery store example above, and I also realize putting something out on a public network invites a certain amount of risk, and lunatics from the fringe, but when you install barriers and roadblocks to stop that activity and they just keep coming it's beyond and above what falls under 'normal access' and is well into some serious realms of abuse and harassment.

But this is just 'copyright infringement' and 'bandwidth theft', right?

Well, maybe it is when you identify yourself as robot and behave in an acceptable way that allows me, the webmaster, to stop you with reasonable efforts. However, when you mask to conceal the nature of your visit, change the identity of the crawler, and then attempt to crawl undetected and bypass mechanisms such as firewalls put in place to stop your activity, then TECHNICALLY this becomes hacking.

Isn't hacking to gain unwanted access a FUCKING CRIME?

I'm thinking someone needs to make a test case on this and just bypass copyright altogether and try filing a criminal complaint against them for hacking and see what happens.

I have a simple case I could use to test this theory already, as initially I put simple roadbloacks in place for humans to get past just waiting to see when the bandits would code around it, and sure enough they did a few months later. Then of course I made it harder and I'm waiting to see if they'll take a shot at bypassing this roadblock as well. Don't worry, there are deadbolts I can install to keep them out if needed, but my current cat and mouse game is more fun and useful to learn from as they expose the level of their technology.

OK, since I put a lock on the door that stopped the unwelcome technology that was hitting my site and someone deliberately programmed to bypass it, isn't that technically hacking?

I'm thinking it's just about time to drop some money on a lawyer to see where this idea stands within the definition of the U.S. laws on hacking as a crime as it would be nice just to scare the shit out of the local scrapers.

Not to mention it would sure be a cool to put a logo on my site that basically explains to these assholes:

"Crawl my site, go to jail, it's the law."

Who am I kidding?

Some asshole in some foreign country would just start a whole new business selling scraping services or pre-scraped websites if they aren't doing it already.

Then we're right back to fighting copyright infringement.

PicGrabAss Image Theft Technology

Some fuckers on a website far, far away have a product called PicGrabber that let's you quickly and easily crawl a website and steal every picture, movie, and any other goddamn thing that isn't bolted to the walls and nailed down to the fucking floor.

Their slogan is:

PICgrabber - The software that finds millions of free movies and free pictures for you!

And catch this "feature" list:

• Scans the web with Keywords
• Scans any given start URL
• Downloads all found Images / Movies / MP3 automatically'

Shows up in your log file as:

Mozilla/PICgrabber - (http://www.movies-free.net/)

Luckily their little grand theft tool got nothing but bitch slapped with error messages.

Hey PicGrabber, try my slogan: FUCK YOU!

Live Servers Update

The more IPs I block from this group, they just seem to move to a new block of IPs. All are being hosted by live-servers.net and they just using block after block and keep on coming.

The current ranges being blocked are as follows:

88.208.192.
88.208.193.
88.208.194.
88.208.195.
88.208.200.
88.208.202.

These ranges never send any traffic with the exception of multiple crawlers from the same block at the same time and all IPs appear to be pointing back to a server farm so I'm recommending you just block 'em.

More IP updates on this group as they become available.

Where are all the kick ass PHP programmers?

Now that I'm ready to actually build my bot blocker for commercial purposes it seems like it's next to near impossible to find any quality PHP programmers, at least none that aren't already booked thru 2010, to help build the damn thing.

For those thinking it's going to be some simple piddly-assed fire-and-forget script think again as we're talking a product of much larger scope, including a centralized server component. The scope is more that just the "bot blocking" that I rant about here as it will be a complete crawler management and protection system. It will allow webmasters to manage access to their site without having to deal with the technical aspects, nor keep up with the latest crawlers. Additionally, there is a security component that will harden web sites against a variety of exploits.

I was prepared to just pay someone to convert it to PHP, or even be an equity partner if they wanted some of the back end action, but it's starting to look like I may just have to roll up my sleeves and do it all myself. That's going to throw a nasty monkey wrench into the time frame as there is still pending R&D that needs to be done.

So much to be done, so little time, and a planned summer BETA release not looking so good at the moment.

Blocked Spiders DO NOT Go Away

There are a few bold, albeit naive, statements by other so-called "bot blockers" that scrapers just go away after you deny them a few pages which is complete and utter BULLSHIT!

Some of the scrapers being blocked on my server have been set to BANNED for months now, haven't gotten a single page of value, yet they just keep coming over and over, attempting to get pages they remember regardless of the outcome.

Most bot blockers I've reviewed just set speed traps or page limits and then throw a captcha in their face to make them go away for a brief period of time, maybe a few hours, maybe a day or two, but many of them will come back over and over and get another chunk of pages when they return. The stakes are high and the scrapers want your content badly so putting silly little bandages on your website for short term solutions do not cure the long term problems.

The only way to truly stop them is to profile their behavior over time as my bot blocker throws all first-time suspected bot IP's into QUARANTINE. Once an IP makes it into quarantine they are immediately suspended for 24 hours and then challenged immediately when they return to the web site after 24 hours. This stops repeat offenders from getting any pages whatsoever when they return and also protects against permanently blocking a DHCP address by accident that is used to scrape only once. After a couple of repeated scrape attempts without breaking thru the challenge, which a human can easily do, the site is escalated from quarantine to BANNED which no longer presents challenges and just gives error messages on repeat visits.

Not rocket science but it has a lot more finesse than some of the more simplistic methods others employ and better hardens the site against repeated attempts at scraping.

Gosh Goes Wild

I blocked one IP address and they came back with a new IP address 72.51.37.210 doing their sneaky masked crawl bullshit.

Go away little crawler, stay away little crawler, or this WILL get ugly when I start cloaking 'YO MAMMA' snaps into your fucking search engine for major keywords like: "Pool Cleaning Services - Yo Mamma so stupid she saw shit in the pool and thought the anal porn queen was trying to teach her baby to swim!"

BTW, fix your dumb fucking crawler so it can properly parse a goddamn web page as hitting my server for "GET / <font" is about as bright as asking for 30 pages in a minute while claiming to be "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Stupid fucking shit really gets on my nerves.

Alexa Hiding in the Shadows

What the hell's going on with Alexa trying to crawl my site with NO user agent string!

Had an attempted anonymous crawl from 209.237.238.224 which is:

whois 209.237.238.224
Alexa Internet ALEXA-INTERNET (NET-209-237-237-0-1)
209.237.237.0 - 209.237.238.255

They still read the robots.txt file, but no clue who it was:

209.237.238.224 - - "GET /robots.txt HTTP/1.0" 200 111 "-" ""

What the hell's going on Alexa?

Someone break the crawler or you just trying to sneak under the radar after everyone blocked your ass?

You claim to crawl as "ia_archiver" but I don't see any ID here whatsoever!

Please explain, inquiring minds want to know!

Cloaked Bots Snared, Charted and Graphed

Since improving the profiling algorithm of my bot blocker so that cloaked bots could be trapped more accurately the statistical data has been piling up and now that data has been compiled and graphed for your viewing pleasure.

Hopefully this will explain to those that think I'm trying to stop beneficial spiders that those spiders aren't what I've been going on about whatsoever.

For those new to the blog we'll quickly define a couple of terms.

• Cloaked Bot - a web crawler that uses the user agent string of a browser like Internet Explorer, Firefox or Opera
• Blocked Agents - those crawlers that plainly identify themselves, such as Googlebot, but are unwanted and typically blocked via robots.txt, .htaccess, or other methods.
  

The chart below shows the amount of requested pages by cloaked bots pretending to be browsers (in purple) compared to blocked agents that have a plainly identifiable user agent string. One thing immediately obvious is that page requests by the cloaked bots far exceed the number of page requests made by all the other blocked agents combined.

The trend analysis of the cloaked bots reveal that, other than a couple of recent spikes, their page requests are on a slow but steady decline over the period charted. This could be a direct result of blocking them and sending them pages of error messages since they aren't getting what they need. Note that in the same time period the normal blocked agents keep attempting to request pages at a steady pace, regardless of the fact they're getting nothing but error pages, with no obvious trend evident at this time.

While some of this may look underwhelming at first, realize that many crawls would've been for hundreds or thousands of pages that are now being stopped before they ever start. Much of the attempted crawling appears to be repeat offenders that already have a complete listing of all the pages of the web site from prior visits. Some cloaked bots do manage to get the site map before being stopped and try to crawl all the links they know about, albeit fruitlessly.

At a minimum this information proved my theory, for my website anyway, that it's not the bots you know and can see that are the real problem, it's the bots you can't see.

With all this in mind, do you think you really know how many actual visitors and pages views your site gets?

Spy vs. Blog Cache

OK, this is getting ridiculous, spy on someone that's stupid and falls for your tricks.

I reported on blog spying the other day and the direct spying stopped showing up in my activity log but now I'm seeing information spies watching my site via Google cache.

Let me be the first to tell you sneaky snoopers that you've met your match this time, the gloves are off so keep fucking with me so I can add every trick in your book to my bot blocker product. Just keep showing off trying to spy on me as every time you play a new trump card you can never use that trump card against me, or anyone that will install my bot blocker in the future.

Talk about playing with fire, sheesh.

Did you think that you could stop hitting my blog and hit my cache and not be noticed?

Look at this listing from Google cache that was scrubbed of identifiable data to mask these people:

http://72.14.203.104/search?q=cache:incredibill.blogspot.com/ %22[THE IP OF THEIR WEB SITE]%22

Very clever, scanning Google for the domain name of my blog, and narrowing the search for LINKS to your site from my blog, by your IP address instead of your domain name hoping it would go unnoticed.

Maybe other people are stupid and fall for this shit but I noticed that IP address, did an NSLOOKUP, and VOILA!, it was the same block of IPs and spybot company name that I've blocked on my main website and discussed in this blog before and now this continuing surveillance is bordering on, bordering hell, it's harassment.

Just go away, we're done already.

UPDATE: After lunch I set the blog to NOARCHIVE so as soon as Google and all the rest update and kill the cache we'll see what they try next.

Beware Blog Spies

This isn't the first time I've noticed this but it's definitely the quickest that my blog has attracted spying operations that monitor information about specific topics.

The first time it was overtly obvious what was happening when I posted about corporate crawlers that spy on websites and the very next day my blog was under daily surveillance from these corporate clowns.

The second most obvious time started yesterday when I posted the blurb "Dubious from Dubai" which immediately started getting hits from what appeared to be automated tools performing automated queries for Dubai on search.blogger.com and one of them was from 217.164.235.140 which according to whois is Dubai: "P.O. Box 1150, Dubai, UAE". The other hits were from several places in New York which make me wonder who's camping on any information about these Arab countries and for what purpose?

I'm not a cloak and dagger kind of guy but sometimes you just see things that raise your eyebrows.

Yugoscrapia My Ass

The "Get a Fucking Clue" award for today goes to whoever the hell is sitting behind 212.102.136.25 hiding as Mozilla/5.0 (compatible; MSIE 5.0) from Yugoslavia that has spent over 8 hours now downloading 1,000+ error messages from my site and is currently still going strong.

Well pal, whoever you are, my automation is better than your automation because mine has a built-in BULLSHIT DETECTOR that stopped yours from doing it's job.

Dubious from Dubai

Just got hit for a couple hundred pages each from this triumvirate of IPs from Dubai:

195.229.241.181
195.229.241.180
195.229.241.187

Reverse lookup said NXDOMAIN and Whois didn't get much either so I'm just gonna block 195.229.241.* and be done with it.