Almost Ten Percent of All Pages Blocked

The total number of pages being blocked from non-humans masking as browsers on my site is about 10% of all pages displaying daily and this rate has been holding strong since I started blocking them.

That's right, for every 48K pages displayed daily between 4K-5K are going to scrapers, crap search engines that send no visitors, and other useless wastes of bandwidth. That's a heck of a lot of pages that are being scraped and the purposes for all of this are still only slowly unfolding as each week something new shows up somewhere on the net.

Simply amazing.

It will be interesting to get more sites profiled moving forward and see if this is a common trend or not as 10% of all site traffic being wasted, not to mention server resources, could take a decent load off many servers starting to feel the pinch.

Will get some more detailed stats together next week hopefully and put them up for everyone to take a gander at as it's blowing my mind for sure.

WebCorp Crawls Why?

Saw this entry on my blocked log a couple of days in a row now:

03/24/2006 193.60.130.67 "WebCorp/1.0"

So I looked it up and sure enough it's WebCorp

webcorp.uce.ac.uk.

Went to their website and they have some mental mind fuck linquistic mashup running they call SEARCH and it's the slowest thing I've ever seen since I used a Commodore-64.

However, it claims to cull results from Google, Altavista, Metacrawler and AllTheWeb so why in the hell is this thing crawling attempting to crawl my website if my content isn't even being taken into consideration for their results?

Sorry, you PhDs in linquistics are just too smart for me so maybe you have some higher purpose for attempting to crawl that just escapes us bot blocking neanderthals.

Here's a phrase you cunning linquists might know - "fuck off"

How Clever Yet so STOOOOOPID!

These assholes that crawl my site must think I'm looking for browser-like behavior such as image loading or something, or they're just using an API to drive Firefox in an effort to completely mask their tracks running complete browser operation including loading ads.

Now let's follow the fun antics of this crawler:

63.197.247.13 - - "GET /robots.txt HTTP/1.1" 200 111 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1"

Reading robots.txt with a browser?

I should've stopped you here but I didn't just because I know you won't get far, 10-20 pages tops, and since I'm a bit on the sadistic side, I've been letting them continue on past robots.txt lately just to see how good the rest of my traps are working.

STRIKE #1

Now you access a page the robots.txt told you not to use?

More importantly, you can only see this page name if you're looking in robots.txt or finding it hidden in my HTML, normal visitors don't see this page.

63.197.247.13 - - "GET /dont_click_this_page.html HTTP/1.1" 200 12895 "-" Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1"

STRIKE #2

Then your stupid ass program continues to attempt to load 60 more pages while not noticing you're getting a captcha after stepping into the spider trap and after a few pages of that, getting error messages telling you that YOU'VE BEEN BUSTED.

STRIKE #3 - YOU'RE OUTTA THERE!

Which just goes to show that even the ones that do things that would bypass my 3 page robot stopping technique still get stopped in record time. Not only that, they could've been stopped before the first page as reading robots.txt is a cardinal sin for a browser.

However, stopping them too fast at this point and I wouldn't have had the fun of getting more profile information from this type of crawler.

The harder they try, the harder they fall, as someone obviously went to some extremes to pull this off and still ultimately failed in the end.

Cyveillance Keeping An Eye on My Blog

You tell the world Cyveillance is snooping on your website and sure enough here they come snooping on my blog too.

Referring Link http://blogsearch.google.com/blogsearch?hl=en&q=cyveillance&ie=UTF-8&scoring=d
IP Address 65.213.208.155
Country United States
Region Virginia
City Arlington
ISP Cyveillance

We know you're watching us watching you spy on us.

FYI, they also seem to have this range of IPs:

CYVEILLANCE UU-65-213-208-128-D4 (NET-65-213-208-128-1)
65.213.208.128 - 65.213.208.159

Stop that bot, 3 pages or LESS

Some days you just wake up and BINGO! you have the best damn idea you ever had in a long time. That's right, something so simple and as plain as the nose on my face just slapped me upside the head today and I'm positive I can shut down bots masking as humans in 3 pages or less into a crawl.

However, it was real a bonus day for me as I came up with not one but TWO new techniques to add to my arsenal of bot blocking weapons. The only downside is both of these tricks require changes in the web pages in order to make it work, but it's well worth the trouble if it can stop bots dead in less than 3 pages.

We'll be doing some testing for a week or so to see if it's really as effective as I think it is and verify it's not snaring humans and let you all know how it works but I'm REALLY excited as this KICKS ASS so far!

Anyone know a good patent attorney that's reasonably priced? ;)

More Fun With Stupid Bots

The website I'm protecting from all these idiots uses javascript navigation for some pull-down menus and it appears that a couple of the scrapers appear to be attempting to scrape something out of the javascript looking for embedded URLs in the script itself.

Unfortunately for them, but lucky for me, their code is mildly brain damaged and doesn't bother parsing the extracted information to see if it's a valid URL and these morons are trying to access a page name from the server like "/getURLfromMenu".

Now that I've noticed this little tidbit I went back and checked my bot blockers error logs and it's already stopped this from about 40 different IPs in the last couple of weeks. A couple of the others attempting this had invalid user agents, meaning they didn't scrape it lately as they haven't been getting into the site for many months, so this has been going on a LONG time before I caught them.

At least I have another bit of criteria to add to my instant block list.

Stealing a line from Seinfeld with apologies to the Soup Nazi:
NO SCRAPE FOR YOU!

Clone Wars

Who are all these wannabe assholes?

I've been the original IncrediBILL online since I first got my hands on a 300 bps modem and if you don't believe ME, then ask my wife FRANtastic!

Most of these fuckers were probably hopping from ball to ball just to keep from landing in their Dad's tissue or being the glue sticking the pages of the Playboy together, or maybe shitting yellow in a diaper at the time.

Look at this shit, IncrediBill's to the left, IncrediBill's to the right, the fuckers are crawling out of the goddamn woodwork!

For the love of god make up your own fucking names.

Educating the Public About Scraping

After talking to a lot of people lately, many webmasters and Silicon Valley internet savvy types, it has become obvious that they simply are oblivious to the entire problem with rogue bots and scrapers. Most people I've been discussing this with are aware of crawlers and they're aware of things like robots.txt, but completely in the dark about what goes on bypassing so-called standards. Ultimately, they leave the conversation with a new level of fear about the security of their online content and run to the nearest console and start searching for unauthorized usage of their content which, as we all know, they typically find without too much trouble.

The real eye-opener for most that aren't building sites that thrive off Webmaster Welfare™ (aka AdSense) seems to be the entire AdSense economy that fuels the bottom-feeding scraper sub-culture that Google has unwittingly created. Once they understand the motivation not only is it clear why scrapers scrape to anyone, but many wonder why they didn't think of it first! Then it's obvious that the low hanging fruit has universal appeal and everything on the net is fair game for the unethical types that pluck that fruit at any cost.

So the question remains, after this small sampling of industry savvy folks, is how wide is the blissful ignorance to this pandemic?

Wonder how many people learn something about this for the first time hitting this web site and just think I'm a paranoid loon with a tinfoil hat dancing with a flute celebrating the summer solstice?

I'm suspecting the depth of the problem is not known by most, even by webmasters fighting one-off copyright infringement, those that even have a hint think it's being overblown and from what I'm seeing in the last week in my banned log files, it will get a lot worse before it gets better.

Spyveillance, Block 'em if you got 'em

OK, this must be a clue that my bot blocker has graduated to the head of the class as I've snared 2 coporations bypassing security measures within 24 hours pretending to be browsers.

Remember what I said about bot blocking being an onion that you keep peeling layer by layer?

The next one in our list of sneaky snoopers is Cyveillance, which apparently has been around for a while but went silently unnoticed until I cranked up the level of bot profiling on my site just a bit to see if I was missing anyone and BINGO! got 2 big fish in a day looking at the next layer of the onion.

According to what I've been reading at linuXgod's site, these boys spy for the RIAA, government and god knows who else or for what purposes. He's been trying to get them to stop crawling his site via a small back and forth of emails and they don't seem to be interested in complying.

My favorite quote is where they justify ignoring internet standards like robots.txt and mask the user agent string as a browser ""Mozilla/4.0 (compatible; MSIE 6.1; Windows XP)".

Because many sites use redirection pages to route robots to special "indexing" pages, we identify our web crawler as an IE browser to ensure it receives the same content as the majority of web surfers on the internet and to allow our programmers to concentrate on a single interpretation of thehtml standard.

Well hell, doesn't that logic just make it fucking OK to ignore whether I want your robot on my server in the first place?

So you're justified in bypassing my security to stop browsers just to concentrate on a single html standard?

Well guess what, NO, YOU'RE NOT JUSTIFIED!

Here you go people, the range of IPs so block them as we're not being given any other means to detect this crawler:

whois 63.148.99.239

Cyveillance QWEST-63-148-99-224 (NET-63-148-99-224-1)
63.148.99.224 - 63.148.99.255

and...

CYVEILLANCE UU-65-213-208-128-D4 (NET-65-213-208-128-1)
65.213.208.128 - 65.213.208.159

Wish I had the bot blocker commercialized now to go mainstream and nail this nonsense.

Corporate Crawler Masking as MSIE

Well, color me stunned shocked and appalled as I ran into an actual real live corporation with a legitimate product that is deploying a crawler that sets the user agent as MSIE ""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; ....)".

Yeah, that's right, forget robots.txt, forget letting you block them by normal user agent filtering means, they're getting into your website whether you like it or not because they have MANIFEST DESTINY!

They are ENTITLED TO YOUR CONTENT!

Not.

These lovely sneaky snoopers that boldly bypass your firewalling efforts are Lightspeed Technologies and they appear to be operating from this IP range 66.17.15.128 - 66.17.15.191.

Just block them now as this is about the lowest I've seen a corporate crawler get and they should be blocked on principle alone by not honoring internet standards.

Terms of Service vs. Fair Use

Here's my next thought about how to combat ill behaved spiders that include snippets from your website and claim fair use. Include something in your TERMS OF SERVICE or LEGAL page on your website that prohibits unauthorized robots.

Therefore, even if they are within their rights of fair use they've violated your terms of service and you possibly have an actionable item on your hands.

Thinking about running this one past a lawyer as we need some boilerplate text like the GNU license that can be distributed and used everywhere as leverage against scrapers.

Film @ 11

Related Movie Bullshit

If my last rant just moments ago about sucky movies didn't get the point across, I was just reading some entertainment news and J Lo may star in a movie adaptation of Dallas and Ice Cube may star in the big screen version of Welcome Back Kotter.

Dallas?

Welcome Back Kotter?

You must be shitting me!

IT'S OBVIOUS WHY PEOPLE DON'T GO TO THE FUCKING MOVIES YOU MORONS!

Looks like I'll need to take up golf or some shit since I obviously won't be at the movies anymore.

Hollywood Blames Piracy Instead of SUCK MOVIES

This rant is rated TV-MA for fucking language.

Pay attention Hollywood, just sit down and listen the fuck up, it's not piracy that's stopping movie goers from watching films and buying DVD's, it's the long stream on non-stop shit you've been cranking out this year stopping people from wasting their money. In case you missed it the first time, listen up you shithead demographic chanting morons, it's your BULLSHIT MOVIES keeping my money in my pocket, not piracy, not video rentals, not On Demand, not cable TV. Don't put the blame on anything but your garbage product as it's ALL YOU, nothing else, causing your decline.

My wife and I used to go see movies 1 or 2 times a week and in the last year we've been having a real hard time finding anything worth wasting money on 1 time a month so obviously we're stealing shit instead of paying to watch shit according to your theory just because you couldn't make a movie worth 2 thumbs up your ass most of the year.

Other than Walk the Line, Matchpoint, Syriana, and Good Night, And Good Luck plus a few others I can't remember at the moment the choices have been real fucking slim this year.

Nothing fun stood out like that anticipated sequel to the First Wives Club those bastards shelved, give me Goldie, Bette and Diane before I get pissed or one of them has a stroke! Nor did they show anything outstanding like American Beauty or being a Nicholson fan we could use more films like Something's Gotta Give, About Schmidt and As Good As It Gets and I could care less if Jack stars in them either, just good quality movies you can watch!

Not to mention the massive vacuum of any real stand-out superhero flicks in a while but the new Superman is on the way, supposedly. Nor have we seen any good SciFi / space epics in a loooooong while but TV's Friday night SciFi line-up is kicking their asses anyway so if you decide to make a new space movie it better kick ass like the original Alien or be fun like Starship Troopers, something special because people running around a rusty bucket ship shooting each other over some fucking conspiracy in a low budget space movie is BOOOORING.

I like to go out, I like a night at the movies, I'd go twice a week easily, but I refuse to ruin my night watching any old shit you think I'll pay for because you're WRONG WRONG FUCKING WRONG so make a good movie or just shut the fuck up and go broke already.

BTW, while I have your attention, spread out the goddamn movie times you assholes. People are still getting off work and want to have dinner when all the movies are starting at 7pm and most people move on and do other things before the next showing at 10pm. It's stupid, it's always been stupid, it will continue to be stupid and you can lose more business until you stagger showtimes a bit more so working slobs can see movies at 8pm and 9pm which is more reasonable.

So BLAME PIRACY when you continue to MAKE SHITTY MOVIES and continue with showtimes on CRAPPY SCHEDULES so.... FUCK YOU, FUCK YOUR PROFITS and STOP FUCKING WHINING YOU RICH HOLLYWOOD GOLD-PLATED GATED MANSION LIVING MOTHER FUCKERS just FUCKING FUCK YOU!

Just make me some good movies and we won't have a talk again, ok?

This Means War

While I was out to lunch this afternoon some nitwits that I've banned over and over trotted out a new IP address and tried to scrape 1,000 pages when nobody was watching.

Sorry pals, someone WAS watching, it was my little silent sentinel buddy that I wrote myself that blocked your ass after about 20 pages and sent you a nice whopping 900+ pages of error messages.

I think I've had enough of your shit though and perhaps it's time we see what ABUSE@SCRAPERHOSTING.COM has to say about your repeated attacks on my server.

Hope you get shut down or at a minimum find yourself in bed with Lorena Bobbitt and wake up with a Frankendick.

Knuckle Scraping Neanderthal

When a scraper reads your robots.txt file don't you think they would avoid the disallowed pages and directories?

Then would you believe the scraper reads your robots.txt file a SECOND time after just downloading a few pages and immediately opens the page that it's told to leave alone and WHAMMO! gets stopped.

How FUCKING STUPID can you be to write such brain damaged code?

Chitika ContentHit IPs

Chitika took another shot at my server with the user agemt "Chitika ContentHit 1.0" but this time tried a whole bunch of IPs on a single web page, one that Chitika doesn't even appear on which was most amusing.

• 67.15.219.3
• 67.15.219.11
• 67.15.219.18
• 67.15.219.14
• 67.15.219.15
• 67.15.219.10
• 67.15.219.9
• 67.15.219.16
• 67.15.219.17

So there you have it, let 'em in block their ass at your leisure.

Looksmart using Nutch?

When I was looking thru the blocked bot log today I ran across a single nutch hit that caught my attention which upon closer inspection appears to Looksmart playing with Nutch, not even identifying themselves as Looksmart.

VERY ODD.

This was the entry:

03/11/2006 07:07:16 BAD_AGENT 64.241.242.18 "NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)" "index.html"

So I looked it up and there they were:

nslookup 64.241.242.18
Server: 64.34.160.76
Address: 64.34.160.76#53 Non-authoritative answer:
18.242.241.64.in-addr.arpa name = sv-fw.looksmart.com.

Open Source replacing jobs in failing companies?

Think someone got fired in the seach dept. down there, if you can call it that.

Stupid Bots Can't Parse

The other day I posted about the idiots trying to locate "/#" and "/#top" but I didn't even notice a couple of nasty bots that aren''t handling SGML or HTML properly and are leaving things like "&" in the URIs.

Well, that just made my life a WHOLE BUNCH EASIER as a couple of the worst offenders are really sloppy like that so at the moment I've got at least 5 snares running just on their idiocy alone.

BTW, if I haven't mentioned it lately, that little bastard gnootBot is still attempting to crawl my site all these days later. Persistent little fucker, I'll give it that much.

Saturday is Scrapeday™?

Did I miss a memo?

Was Saturday designated Scrapeday™ and nobody told me?

Today's attackers should've been filmed and released onto video as "Bots Gone Wild" as there were as many as 5 at a time hitting me that were masking as a browser, well masking the user agent only, not their bad behavior.

The boys over a live-servers.net must've noticed they were being stopped and tried to send a new can of whoop ass from the following IPs:

• 88.208.193.65
• 88.208.193.67
• 88.208.200.198
• 88.208.200.236
• 88.208.200.220

I'm so done with them now that anything coming from live-servers.net is just going to get the big FUCK YOU from first contact.

The other thing that caught my attention today were several scrapers that used multiple IPs to avoid detection, big shock, but was being tracked by the cookie again which stopped the morons as they hit several landmines. It's just priceless to think someone goes to all the trouble to use what most likely was a series of proxies to avoid detection but don't dump the cookie when the IP switched to a different network.

God I love idiots, they just make my work easier.

Unipeak Privacy Scraper

Don't you just love all these so-called privacy proxy sites like these monkey spankers over at Unipeak that claim to "Filter out unwanted advertising such as banners and popups" when in fact they insert their own unwanted ads on the page AFTER TAKING MINE OFF!

They appear to be downloading your pages from 207.234.209.125 so just block their asses.

CALL TO BOT BUSTERS - ACCESS LOGS NEEDED

We're moving into the next phase of bot busting and need your help!

Anyone out there willing to open their access logs to let the Mad Scientist take a peek by letting my scraper analyzer comb a months worth of data to see what kind of shit is hitting your fan?

I would be interested in other sites that generate traffic from 100K-400K visitors a month on this first pass but won't turn down 1M+ visitors either. Your site and traffic will be kept 100% confidential with only vague statistical data comparisons used from the final analysis.

The purpose of this experiment is to see if you're being abused by the same IPs and agents, to see if there is commonality in the spider traps being used, etc. and if not, WHO is abusing you and would my current scripts stop them dead in their tracks.

Basically, this is a quest for help to build the bot blocker Rosetta Stone.

In return you'll get a report of all potentially bad activity happening on your server that you could take action against immediately.

If you wish to contact me privately and anonymously, not in blog comments, then I suggest private mail to IncrediBILL on SearchEngineWatch, WebmasterWorld and ThreadWatch.

When we have enough sample subjects we'll post a comment here that we're closed for submissions.

Be a part of internet history that's about to unfold, submit your logs today!

Matt Cutts Confirms People Are Stupid

Rarely do I write about something posted on someone else's blog but Matt's post about "How to sign up for WebmasterWorld" is absolutely priceless. According to Matt the problem arises when he references people to WebmasterWorld and then people write to him asking if there is a way to get into WebmasterWorld for free because the login screen implies you have to pay to join.

Let's give Brett Tabke kudo's for such brilliant marketing to underplay the free registration to access WebmasterWorld because there is a lot of valuable information over there and after all, he's running a business and not a charity. He once told me how much he pays a month for his servers and bandwidth and it's a shitload. I can't really say I blame Brett for making people think they need to open their wallets to get inside.

After a small amount of begging and pleading with Matt to kill the thread, I realize he's just too nice a guy to the people he's trying to help and isn't concerned that he's foisting idiots onto WebmasterWorld that can't even figure out a simple IQ test to get inside.

According to Matt:

if I wanted to post a pointer to a WMW thread, I didn’t want to get “how do I do it?” questions. And I’ve seen that from some people with high IQ.

Well just how freaking smart can they be Matt?

If you put cheese in a maze even a rat can find it eventually if they're hungry enough so why coddle these whiners that can't even help themselves to a free registration?

Not that I mind helping people, but I draw the line at holding their hands and wiping their asses when all they need to do is read and click, it's nothing mind bending.

If you want coddling then Matt's your guy, and a nice one too, no question about it.

I'll continue challenging people to think for themselves as I'm a firm believer in that teach a man to fish theory.

Googlenoia [warning, major rant]

What the hell is wrong with all you people?

Don't you have a fucking life that doesn't revolve around Google?

The last few days it's nothing but AdSense is crashing, BigDaddy is broken, Google stock is tumbling, blah blah get a fucking life blah.

First, losing some traction in the search engines thanks for some PhDs having brain farts at the 'plex doesn't mean your AdSense is broken or any of the conspiracy theories you can concoct to add to the mystique of AdSense. People using AdSense dance around it like the monkeys banging on the monolith in the opening scenes of 2001 a Space Odyssey. Come on fucknuts, unwrap the tightly wound tinfoil hats and use that steaming heap of gray shit called brains and realize that trends change, advertisers and budgets wax and wane, Google tests things now and then and on top of the list SHIT HAPPENS.

Funny, once upon a time Yahoo stock was skyrocketing and like all things that go up it came down but for some reason Google is different and held to different standards. Well too fucking bad you psuedo-religious Google freaks, it's not different. Google made a lot of people a lot of money, some would call them filthy fucking rich, and if you were too stupid to buy early or sell high then FUCK YOU for being a moron so stop whining and move on. For what it's worth you should probably sell at a huge loss before you lose everything and you're homeless living in your goddamn car because it's not going to rebound, the wild ride is over.

Last but not least, those crybabies that come out in droves every time every search engine changes, especially Google, and you or your customers go up and down from #1 to #5 or heaven forbid your sorry ass slipped to PAGE TWO and you're not in the top 10 anymore.

Well for those of you whining about your SERPs I have 2 words:
FUCK YOU!

With hundreds of thousands of pages competing for these top keywords you were lucky you got there in the first place and the fact that you couldn't survive a new way of indexing is just too damn bad. Nobody OWES you that position in the search engine and you don't have the right to demand getting it back, adapt and deal with it or just fuck off as I'm sick of hearing all your shit.

I'll bet conversely someone else is happy as shit they moved up and are dancing in the aisles getting all that free traffic that your whiny ass is bitching about as one man's tragedy is another man's blessing when it comes to search engines.

This Googlenoia is just getting so fucking old, can't we find something new to talk about?

Tell me about your latest project, found any good blogs lately?

JUST SHUT THE FUCK UP ABOUT GOOGLE ALREADY!

Enough.

Dumbest of the Dumb

Ever see a crawler look for "/#top" on your web site?

I'm still laughing over whoever wrote that shit.

UPDATE:
The dumb fucker came back today 3 times and tried to get "/#top" every time.

Appears to be some stupid fucker from The Netherlands using DHCP as each access was from something like dipshits.too.stupid.to.live.CHELLO.NL

Funny Funny Scraper Shit

Well, my scraper challenge page contains a URL in the sticky challenge loop that can vary per page that let's a human get past right away but keeps spiders looping until they try to index that particular new link, which would let them pass if they indexed it quickly, which of course is too late by the time they actually index it and are locked out for a while.

Ok, now the funny shit, these idiot spiders are now coming back looking for this link as an actual page so if you're not already in the sticky spider loop and ask for the page directly, WHAMMO!, you go DIRECTLY TO JAIL, DO NOT SCRAPE PAST GO, DO NOT SCRAPE 200 PAGES!

It's like a high-tech comedy show at times and you just sit back sipping bourbon waiting for the first asshole to set foot in the latest snare.

Ah, my side is killing me from laughing so hard at these fucking idiots.

Anonz Azz booted to infinity and beyond!

That's right, someone was on my site that came from anonz.azz.ru and I'll bet you're shocked that it has something to do with their website www.Anonymizer.ru aren't you?

Well a BOOT TO HEAD for both of them!

And one for Jenny and the wimp...

Maxwell's Silver Hammer

Most of you probably think I'm very brash and run amok implementing things on my server all willy-nilly with hardly a concern for the damage that I might be inflicting on my visitors but that's further from the truth than you can imagine. I'm actually very cautious and do a lot of testing with each new approach I phase into my bot buster by first executing the rules and giving me a preview of what would happen for a day or so before I make the rules live.

That means to date all the IPs I've been banning are being banned in software so that I could monitor their returns and activity to verify they're really a permanent source of abuse or a one-shot attack from a dynamic IP.

Well, enough of them are returning on a regular basis that I've decided it's time to start the next phase of the project which I call Maxwell's Silver Hammer where it will decide automatically that the source of abuse is bad enough and just drop them in the .htaccess file so they simply bounce off the server and don't even tie up my scripts keeping an eye on them anymore.

So here we go sweating bullets that this code won't accidentally crash some night and leave the .htaccess file all banged up and bring the site down to it's knees.

Progress, gotta love it.

Pathologically Extreme

Yep, that's what my bot busting obsession was called today in private email.

Now that I'm "pathologically extreme" I must thank the person for his bluntness as it did bring up the point that there's a lot more to this bot busting issue that someone sitting on the sidelines only casually familiar with my quest and this blog may know.

In all fairness, if you told me I'd be on this quest to abolish unauthorized access to my site 12 months ago I would've laughed in your face and said "what harm does a little crawling do anyway?" and yes, I used to hold those tightly wound content control freaks in low regard as misguided time wasting fools.

However, then I decided to get out of the consulting game and focus more attention just on my own web sites which bring in a decent revenue stream without all of the whining and hassles of customers.

That's when all hell broke loose as suddenly both the spammers and scrapers started hammering my old server so hard it was going down all the time. Not physically crashed mind you, but it was just so busy serving the needs of spammers and scrapers that my income needs weren't being met whatsoever. We're talking DOS attacks because of the sheer speed and volume of this nonsense and the server just didn't respond for 5, 10, 15 and the worst was 90 minutes at a shot. It got SO BAD at one point I had to completely get rid of server side spam filtering as that tool itself could use up all the CPU when some spammer came along doing a pump and dump of spam.

These shameless greedy bastards were impacting my site, my SERPs, my wallet and really pissing me off - the shit had to stop.

First was the easy part which was just getting rid of the spam. I blocked email coming from most of Asia and Russia which eliminated the majority of the high speed spam dumps and gave me some breathing room to work. Then I made the only way to contact me a form on the web sites, eliminating all email addresses but 2, and literally set the server not to BOUNCE emails but REJECT emails. Why I did this is bounce emails still come into your server and attempt to send a response back but most spam has a bogus reply address and thousands of bounce emails quickly fill up the queue and your email system grinds to a screaming halt processing bounce deliveries all day long. Trust me on this, just REJECT those undeliverable emails, no bandwidth or CPU wasted at all as they just bounce off your server harmlessly never to be seen again.

Guess what?

Asia and Russia are no longer blocked as REJECTing their emails stopped them from being a threat.

At this point there are never more than 10 emails sitting in my mail queue at any time and the spam that gets thru is literally a handful of emails a day, blissfully under control, I love it.

However, after solving this problem the old server was still going down like it was under a spam attack and after a while I came to the conclusion my site was probably just too busy to handle the load and my older slower server just couldn't deal with the demands of all the visitors, search engines, etc. and set out to upgrade.

Now, with a big fast shiny dual Xeon box it's back up and running faster than ever.

Two weeks later some fuckers took it offline for 90 minutes in the middle of the night and I lost my shit, that was it, the straw that broke the camels back, no more Mr. Nice Guy.

.... this was war....

Then the whole process kind of evolved into a huge eye opening adventure at this point and being a naturally curious guy and a programmer with a huge ego [yes, I am IncrediBill and I can stop these bastards] it kind of took on a life of it's own.

First, stopping the high speed scrapers was easy, totally childs play.

Next, the sheer volume of scraping became apparent once I was monitoring real-time site activity while squashing the high speed scrapers and looking for other unauthorized resource wasting bots.

Evolution just kept happening as one thing led to another, stopping more scrapers unearthed even more scrapers, that the errors I fed scrapers unveiled tons of sites with MY SHIT on them, and that many people had apparently built AdSense-incentivized businesses based on bottom feeding off my business and in the process were diluting keywords I was earning money from using my own content against me.

OK, now THAT pissed me off even more.

So while some of you may call the depths and extremes I'm taking to protect my shit as "pathologically extreme" my side of the story is self-defense for my very survival and I'll be damned if some bottom-feeding leeches are going to take me down without a good fucking fight.

Yes, that's it people, as far as I'm concerned at this point it's a fight to the death, theirs and not mine if I have anything to say about it. So far I've spent a ton of time and money addressing the issues and at this point it's paying off but for how long only time will tell but it's definitely going to be a death match for one of us.

Now, go buy my CD's and t-shirts in the lobby to help fight the cause and invite me as a motivational speaker at your next nerd conference to spread the word!

Nah, that would be WAY too extreme!

UK Scraperz in da Hood

There really aren't too many truly persistent jerks out there and I hate to rat out sites or IPs unless I'm sure they are truly rotten but this wins the award for most annoying IP address of the year. I blocked 88.208.194.241 a long time ago and it just never stops, it's relentlessly attempting to crawl, day after day, asking for thousands of pages and doesn't fucking take NO! for an answer.

A quick reverse DNS on this idiot shows it's apparently on some UK server farm, possibly run by fasthosts.co.uk, that IDs itself as server88-208-194-241.live-servers.net.

Out of curiousity I went back and ran a reverse DNS on all my blocked IPs and big shock, this server farm has a few hits in my list.

server88-208-192-204.live-servers.net
server213-171-220-120.live-servers.net
server88-208-194-241.live-servers.net
server88-208-194-252.live-servers.net
server88-208-195-4.live-servers.net

Matter of fact, I just reviewed my logs for today and all of the 88.208.19* IPs listed above hit the server so it appears to be distributed scraping.

So I'm thinking it's probably not such a bad idea just to block their whole damn neighborhood based on the non-stop abuse I'm getting from a couple of these IPs.

I would block at a minimum:

88.208.192.0/24
88.208.194.0/24
88.208.195.0/24
213.171.220.0/24

I'd keep an eye out on anything from this range as well:

inetnum: 213.171.220.0 - 213.171.223.255
netname: FASTHOSTS-UK-NETWORK

They claim to resell dial-up and broadband otherwise I'd suggest blocking the whole damn network but at this point I'm not sure 100% if we're looking just at servers or surfers but so far blocking what I've blocked doesn't appear to have any negative impact on my site except to stop their stupid bots from downloading my content.

Let me know if anyone else is seeing activity from these guys and what IPs you're seeing as this is about as bad as I've seen and they need to be stopped.

Alexa Bowling for Matt Cutts

Don't know who did it or how they did it but it was a stroke of genius to load Alexa up with a bunch of related sites for Matt Cutts that are obviously spam.

Wonder how hard this was, just write a script using the MSIE toolkit with Alexa's toolbar installed to just referer spam the crap out of his site?

Is it possible from a single IP to game Alexa or would someone need to enlist a legion of anonymous proxy servers to pull this off?

Or, could it be so simple that someone has dissected the Alexa toolbar API and just fed it a load of junk about Matt?

We may never know unless someone confesses but I'm thinking it might be worth a giggle to try and see what it takes to make something like that happen.

Good thing I'm not bored today!

Best Bot Ever - Almost!

Tonight I saw the future of sneaky bots and it did everything to look like a human so I'm thinking they used a developer toolkit to drive the crawl via MSIE. This thing downloaded images, banner ads from 3rd party servers, ran javascript and even accessed AdSense ads so it was as convincing as anything you can imagine.

Spooky and amazing in how well it cloaked itself.

I couldn't tell by looking at the log files either, very impressive.

Then after patiently crawling for a whopping 11,097 seconds or more than 3 hours for those of you that can't divide 11,097 / 3600 in your head, it exceeded my max page count which is set fairly high.

Then it proceeded to very slowly and stealthily ask for 20 more pages after being told it had exceeded it's daily limit of pages.

BUSTED!

However, the point being if it had been just a bit smarter to realize it was getting a repeated error page I'd have never known it wasn't a human.

Not a good turn of events whatsoever!

Nasty.

Every Webmasters Worst Nightmare

Here I sit working on my website late at night, we're talking LATE at 3am, so I can test some new bot blocking technology in the middle of the night when the traffic is low so impact will be minimal if I screw up and knock everyone offline.

Ok, upload some code, bug, quick patch, bug, something weird going on so hop onto SSH and look at server and something is chewing up CPU like crazy and website isn't responding properly. Whew, it's an automated nightly update that only lasts a couple of minutes and everything is back to normal.

Test some code, find another minor bug, upload some new code, run the page, and it goes and goes and goes. What the heck, did I just blow something big time? Check SSH again, staggers a little and stops, then generates a socket error and closes.

Holy Fuck! Server Down!

So I click here and there in my browser, EVERYTHING is down!

YAY! I didn't crash the server, the cable modem is offline.

30 harrowing minutes later the damn cable modem comes back online and everything on the server is running just fine except I left a debugging message that's displaying on all the pages.

Remove that message, test it again, everything seems to be OK, time for bed.

Whew!

Bracing for nightmares tonight....

Bandwidth sucking Arachmo

Some speedy little crawler from Japan named Arachmo that I've never seen before set off my speed trap tonight so you may want to just block this pest before it hits your server.

60.237.36.157 "Mozilla/4.0 (compatible; Arachmo)"

At one point this little bastard was asking for more than 10 pages a second.

Maybe Godzilla will just step on the damn server while fighting with Mothra some day!

Competition claims WE'RE NUMBER TWO!

Just about chuckled my ass off when I was snooping some of my competitors sites, not the blogs silly, my money making site, the site that keeps me in expensive booze and wide screen TVs, not this bullshit.

Back to the story as I'm DigressJacking™ it already.

Anyway, this site we've discussed before as they undersell ad prices, send out newsletters begging for advertisers to fund new projects, etc. but now it gets even more amusing. Somehow this guy has mysteriously moved up the ranks in Alexa which is now listing him as #2 in his category right below me, oh whoop de doo, I've been ranked #1 in that slot in Alexa since they first set up shop, so you bumped someone to get to #2, big fucking deal. Funny he doesn't mention anything about Google's Directory by Rank that has me still at #1 for years and he's way down the list over there, not a peep.

Then he calls out one other place where he's #2 below me for some bullshit meaningless 2 keyword search term in Google that doesn't even drive traffic.

Excuse me?

Is that smoke I feel blowing up my ass?

Hate to burst your bubble buddy boy but I'm in the top 10 for 2 keywords that are tops in the field, ONE WORD, NOT TWO, yes a SINGLE KEYWORD that drives thousands of visitors daily and it's not a BULLSHIT TERM, people actually actively fucking search this term!

So then he goes on lamenting about how his ads are a better value at his rock bottom bargain basement prices than "some other more expensive sites".

Give me a fucking break, I send people more traffic from my ads in a day than you sometimes send in a month. He knows it's true too because he used to have a page showing the traffic for all his ads and took it down, good thing too, it was embarassing.

I've been toying with making any mention to his references but so far all I did was put up a nice bold line of text on my ad page that says "We may cost more but you get what you pay for - performance" and left it at that.

I'm thinking at this point the best way to deal with this putz is just ignore him and let him dig a deeper hole until Google determines he's supplemental like they just did to an even lesser competitor and relegate them to the search engine dungheap.

Wrong Number, Hang Up Already

Well, someone with their Nokia phone just didn't take WRONG NUMBER for an answer and tried to beat the shit outta my server looking for a place that would take their call.

209.191.82.253 Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

Actually, it looks like Yahoo might've done this on their behalf as the IP address resolves to msfp02.search.mud.yahoo.com so it might not have been the phone that was so insistent.

03/04/2006 05:30:53 "/"
03/04/2006 05:30:53 "/mob"
03/04/2006 05:30:53 "/index.wml"
03/04/2006 05:30:53 "/index.xhtml"
03/04/2006 05:30:54 "/default.wml"
03/04/2006 05:30:54 "/default.xhtml"
03/04/2006 05:30:54 "/home.wml"
03/04/2006 05:30:54 "/home.xhtml"
03/04/2006 05:30:55 "/mobile"
03/04/2006 05:30:55 "/mobile/index.wml"
03/04/2006 05:30:55 "/mobile/index.xhtml"
03/04/2006 05:30:55 "/mobile/default.wml"
03/04/2006 05:30:55 "/mobile/default.xhtml"
03/04/2006 05:30:56 "/mobile/home.wml"
03/04/2006 05:30:56 "/mobile/home.xhtml"
03/04/2006 05:30:56 "/mob"
03/04/2006 05:30:56 "/mob/index.wml"
03/04/2006 05:30:56 "/mob/index.xhtml"
03/04/2006 05:30:57 "/mob/default.wml"
03/04/2006 05:30:57 "/mob/default.xhtml"
03/04/2006 05:30:57 "/mob/home.wml"
03/04/2006 05:30:57 "/mob/home.xhtml"
03/04/2006 05:30:57 "/wml/index.wml"
03/04/2006 05:30:58 "/wml/default.wml"
03/04/2006 05:30:58 "/wml/home.wml"
03/04/2006 05:30:58 "/xhtml/index.xhtml"
03/04/2006 05:30:58 "/xhtml/default.xhtml"
03/04/2006 05:30:58 "/xhtml/home.xhtml"
03/04/2006 05:30:59 "/wap/index.wml"
03/04/2006 05:30:59 "/wap/index.xhtml"
03/04/2006 05:30:59 "/wap/default.wml"
03/04/2006 05:30:59 "/wap/default.xhtml"
03/04/2006 05:31:00 "/wap/home.wml"
03/04/2006 05:31:00 "/wap/home.xhtml"

Sorry, if you'ld like to dial my website, tough shit.

Keep your hands on the fucking steering wheel.

gnootBot still going and going....

As reported the other day we seem to be the first ones reporting on gnootBot and it has never downloaded a single real page from our site but seems to somehow knows every page that's on my server and keeps slowly asking for pages day after day.

Wonder where the pages names from?

Possibly crawled me before installing the bot blocker or downloaded a list of pages from a search engine or some shit.

Doesn't matter as it's still going and there's obviously nobody at the wheel as they are getting nothing but error messages.

Hope it's worth your time when it's over asshole.

FIRST SIGHTING: Sproose Goose got Plucked

Yet another Silicon Valley startup search engine called Sproose came crawling this morning tagged as sproose/0.1-alpha using Nutch. Well, in their site it claims they have seed funding from VC's, also reported elsewhere, but you can't do any searches yet as they are currently building their Knowledge Rank™ which sure sounds a lot like PageRank, huh?

The first knowledge they got when they hit my site was that they didn't rank high enough to crawl my content and got the door automatically slammed in their faces by being an unauthorized bot. Sorry boys, robots.txt is so 90's, we use razor wire around the compound to keep people out these days.

You may have seed capitol, but I require being wined and dined before you may crawl my 40K pages, or just email mail me with a PLEASE as this entitlement mentality to crawl every site and run up our costs online just because you have been funded is BULLSHIT.

BTW, the people that wrote NUTCH should be hauled out in front of a firing squad and shot as I'm seeing more and more crawling from their little engine that couldn't constantly bouncing off my site.

POP QUIZ: Your Site Already Has a Spider Trap?

Most of you web site owners already have a spider trap on your web site and you don't even know it. There are about 3 pages that humans almost NEVER READ and spiders gobble up daily so all you have to know is which pages these are and then grep for them in your access logs and VOILA! you see a list of mostly spiders hitting your web site that can be blocked at will.

Once you get a list of who's been looking at youre spider trap pages, simply take each IP in the list and then grep for all activity for that in your access log. When you see hits to all pages and no images loaded it's a clincher you got a spider but just dont get carried away and block Google/MSN/Yahoo.

Even if an entry in your access log says Googlebot as the user agent it may not be Google, so check out where the IP resolves and make sure it's Google.com in the domain with a reverse DNS lookup, which you can do on DNS Stuff if you don't have other tools available.

Now, anyone want to guess which 3 pages or files on a web site are spider traps?

I know I give you all a lot of information but anyone should be able to figure this out by staring at any typical web site and see which links you would never click.

If nobody can figure it out MAYBE I'll tell you on Monday, if I'm in the mood and if I remember.

Come on people, POP QUIZ! post your guesses, don't be shy!

Scrapevertising

Well here's a new combination that really pissed me off with a scraper and referer spammer all rolled into one. Saw this spunk monkey crawling my site, the bot blocker was already stopping his ass, but something caught my attention in that every page crawled had the same referer as the origin. Sure enough, when I went to look at the site attempting to get notoreity from my access log it was a new directory web site that was using scrapings to get attention to the site for people submitting listings.

Holy shit, you mean to tell me people are stupid enough to click "submit url" when they can plainly see all the listings are junk that don't even have valid links out?

Apparently they are that stupid as the bottom of each so-called directory page appears to be actual submitted listings opposed to the scraped crap content without outbound links at the top trying to snare search engine love.

Now I'm steamed, this is NOT a pretty turn of events.

New AdSense Setup Wizard!

Apparently as more less technical types have joined AdSense in droves Google decided to dumb down the AdSense setup interface to reduce the tech support load and dependence on certain browser features.

The tabs across the top of the AdSense website for AdSense for Content, AdSense for Search and Referrals have all been replaced with a new tab AdSense Setup which now contains all the options for the previous tabs.

When you click on the new link for AdSense for Content you're in a 3-step wizard that drives you thru the process.

1. Choose Ad Type (ad unit/link unit)
   
2. Choose Ad Format and Colors
3. Get Ad Code

My suspicion is many people confused people were accidentally changing options and getting the wrong code, or forgot to get the code, etc. and this will simplify the process for that overloaded page. It also makes it more clear which ad type you're getting as some complaining noobs in forums weren't understanding why they only got links and not text ads, so you know those poor people in AdSense support were crying in their beer every night.

Problem is, anyone with a serious amount of channels trying to get the code will be suicidal with this back and forth nonsense between those two pages. Anyone want to join a class action for Google inducing carpal tunnel with this new page? Just kidding, well, maybe.

I can see why some of these changes were needed for them but it sure would be nice to have an "expert mode" for us AdSense old-timers that actually are power users of the page they just ripped apart.

The only thing I can't believe they didn't fix in this update is making the CHANNEL page accept multiple URLs in a textarea instead of one URL at a time. Try setting up 200 URLs with this stupid page opposed to a quick cut-n-paste from your list of page names on your web log analyzer and you'll see what I mean, it's brutal and stupid, yet it remains unchanged.

Now for the million dollar question:
"What are they about to add to AdSense that required this overhaul to make more space?"

Let's keep an eye out and see what happens.

Comment Wars in Rantville

Normally I wouldn't just blog about a stream of comments but this article about Referer Spammer Revenge is almost a month old and the shit just hit the fan yesterday. I'm thinking the person I mentioned figured out it was me posting about them from my comment at Spam Huntress and went off the deep end.

We're talking some serious flaming here, cracked me up, but you be the judge.

Let this be a lesson to any children that may be reading this shit as it proves you shouldn't stick your finger too far up your nose as you can impair your thinking if you accidentally stab your brain and you'll end up a referer spammer too.

FIRST SIGHTING - New Bot Discovered

It's very rare I run across a bot you can't find any details about in Google but last night something calling itself "gnootBot" came from 66.39.177.8 which just has a default Apache web page.

No information about this beast except it started asking for pages in the middle of my website.

Very bizarre.

Peeling the Scraper Onion with Reverse DNS

Stopping scrapers appears to be like peeling an onion in that when you peel away one layer of bad bot activity you unearth yet another whole new layer that you couldn't see before. You start with user agent filtering then put up speed bumps and honeypots to stop others and even profile other behavior to stop them and they still keep coming. Now we're several layers into this scraper onion all sorts of new things are showing up that require more sophisticated methods to detect and block as they're hiding as browsers, running low key crawls, but still obvious to anyone that it's not a human if you look at the pattern of access.

To help thwart more of this nonsense the latest tool added to the bot blocking arsensal is reverse DNS lookups to see where the IP originates from and blocking or challenging bad sources from the start. There have been a few trends to become very obvious in that many scraper IPs that were auto-blocked didn't resolve to a domain name whatsoever or come from some suspicious hosting farms, the two most notable and persistent ones have been in Taiwan and the UK.

Now they'll need to find yet another way to get around the bot blocker as the newly installed steel door now has a chain, 2 deadbolts and a mean as piss rottweiler waiting on the other side just in case.

Stay tuned for more on the next episode of As the Onion Peels.

Real Impression Tracking in Bot Buster

There was an unexpected benefit to all this bot busting and information recording in that now I can track on a daily basis all pages served to legitimate bots, blocked bots and actual people. This has allowed me to come up with some simple stats that shows a breakdown of where pages are served and I'm getting real close to almost matching exactly what you see in Google AdSense for page impressions.

This side effect alone could be an enormous benefit to people that need actual page impressions vs bot page impressions for selling advertising on their website and I've just made it a heck of a lot simpler to get that information since every single page passes thru my code in real time.

Coolio!

Cleverly Masked Bots Evolving

It would appear that the war over my content has been cranked up a level as bots masking as browsers and modifying their behavior to appear like people seems to be escalating. There are still some tell-tale signs that are easy to spot when you look at the server log but a couple of them that the bot blocker didn't catch are finding ways to game the system.

I didn't want to make the site more difficult for visitors but the only way to stop these guys would appear to be tossing in more random challenges like captchas and such after a pre-determined number of pages. To stop the typical captcha blow-thrus the challenges are very random and nobody could program a way to bypass them all as you don't know what they all are and I can add new ones daily if I wanted.

There's also something I noticed which isn't earth shattering but only humans seem to use my javascript menu which is a HUGE tell. Robots navigate the text links only but humans love those drop down lists and that's a clear sign that differentiates the two of them most of the time.

At the end of the day, it's just like trying to secure money in a bank, no matter how hard you try someone is going to rob you eventually but the best you can hope for is to make the number of times you get robbed as minimal as possible without pissing off all your customers in the process.

Fuck Your Intellectual Property

Some asshats claiming to "defend your brand" sent their little AIPBOT to crawl my pages looking for anything of their clients on my site. Listen up fucknuts, you can use my SEARCH tool and look for something being on my site but you can kiss my ass when it comes to a 40K page crawl just to see if I'm violating someone's precious brand name.

This sense of entitlement of everyone to crawl the web is really starting to piss me off.

Take a hike assholes.

Link Me Or Else!

Here we go again with a persistent raging linkaholic badgering the shit out of me to link to his crappy little directory website so he can get a few AdSense clicks.

Hi YouBigWebStudYou,

I sent you a link request for bullshit-directory.com to see
if you would be interested in exchanging links.

I realize you are probably but wanted to let you know that I will be
removing your link next Wednesday if I don't hear back from you.

You can verify your link is by going to:
http://realfuckingannoying.com

with the following details
Title- Link To Me Please
URL-www.imbeggingyou.com
Description-I'm the biggest pain in the ass link-to-my-site whining spammer you've ever seen so link to me now before I beg and plead more.

If you do add my site please use the below information and let me know
the location you added it so I don't remove the link to yours
unknowingly.

I hope to hear from you before 2nd March 2006 but if not then I'm bound
to remove the link from my site.

Well fucking remove me already and stop sending me this shit!

Didn't the dead silence after your first spam give you a fucking clue?

If I could get my hands on you they'd have a new opening homicide scene for CSI next week so just keep it up, your luck is about to run out.

Oh yeah, it's a good thing you're in India so CAN-SPAM can't be used against you and you aren't registered with GoDaddy so they can't blackmail you to get your domain back. However, if there is a god one of those nasty little bugs in your water will give you atomic diahrea and you'll shit out a vital organ and die.

Search Engines Let Scrapers Bypass Spider Traps!

Just when you thought you've seen it all the actual search engines themselves can be used by scrapers to bypass spider traps. How this is accomplished is the scrapers find all of the indexed page names from your site in Google or Yahoo and then download pages the using known page names from your site thus side-stepping spider traps as they aren't actually spidering your site at all.

Therefore, just eliminating your pages from being CACHED in the search engines doesn't stop scrapers from still using the remaining data to their advantage.

Some days it just doesn't pay to get out of bed.

Another Plug-n-Scrape Component

Yet another toolkit letting armchair programmers attempt to grab my web pages.

Yawn.

This one ID's itself as:

IP*Works! V5 HTTP/S Component - by /n software - www.nsoftware.com

And their web site claims:

The HTTP component can be used to retrieve documents from the World Wide Web.

Might want to revise that to "used to be able to retrieve documents" as it went splat against my brick wall but I found it's calling card in my auto-blocked bot log.

Chitty Content

Must be the new wave of affiliate bots as I also got hit today by the Chitika ContentHit crawler or whatever the heck it is.

No way to verify this bot as reverse DNS for this IP address just claimed to be from Charter Communications.

71.10.233.52 Chitika ContentHit 1.0

This new bot of the day chit's just getting old.

Cell Phones and PDAs Can Piss Off

All these damn cell phones and PDAs all have unique user agent strings and for the last few months the handful that hit my website are all being told to piss off.

You people making cell phones and PDAs better wake up and smell the coffee as I'll be damned if I whitelist a bazillion user agents just to let your pissy products see 5 lines of my web site.

You all better come up with some better ideas for cell phone user agents as this unique name per phone shit isn't gonna fly.

CJ Quality Bot

Well here's a new one that I've never seen before from our friends at Commission Junction.

216.34.209.23 CJNetworkQuality; http://www.cj.com/networkquality

Unforunately they bounced off the walls, think I should let them in?

They might delist my site if I don't but based on the revenues I earned with them last month it's kind of a why bother IMO.

Fine, time to whitelist CJ, sigh.

Robots.txt gives Bad Bots clues to access

In a rather lengthy debate with the owner of Majectic-12 on WebmasterWorld the issue of robots.txt came up over and over and I finally revealed that robots.txt is arcane and a real problem in the world of scrapers as it gives them clues to accessing your content.

Not only does robots.txt reveal which user agents may be blocked in the .htaccess file but it also reveals which agents are allowed into your server. Any roque bot not getting access to your site can simply examine the robots.txt file and use any allowed user agent name to get past the barracades.

My recommendation is to use a generic robots.txt file such as follows:

User-agent: *
Disallow: /stayout.html
Disallow: /keepaway.html
Disallow: /cgi-bin/
Disallow: /someotherdirectory/

Then allow and disallow robots privately in your .htaccess file only to keep that information away from prying eyes and give the lamest of the scrapers fewer clues how to penetrate your site's defences.


UPDATE: the Majectic-12 conversation at WMW is on hold pending review now so maybe I spilled to many beans on site security issues. It was a great debate, hope it comes back only slightly altered.

Cornucopia of Random User Agent Strings

When my bot buster first started operating I noticed a few gibberish user agent strings now and then as I'm sure the theory behind this is if a website is blocking known user agents then you can skirt past that technique with a string of gibberish.

The problem is that they've noticed nothing is getting thru and random user agent string usage against my site is escalating to the point it's hysterical to witness them thrashing.

Small sampling of thousands the other day:

66.148.68.37 2uigq2oecesvv2nwso rwiakBsBue Bobgw2nuB
202.125.44.200 efeSthqvkr11ticgo1iovjjrdwakbbd
66.148.68.34 emwx4cxnd pedafhfpac
66.148.68.34 ymdexin7xpebtulwnxew
202.125.44.199 pepgfu wjdjqrxckulhwiflmrdsmkc mjvldn
84.180.94.183 mairwthe Ifirpl8tiwotwyi lsu
84.180.94.183 r9Hreiynmkxmpjh ioHmmknpdmid
66.148.68.34 ewoqaohlcegoD emkdywx
66.148.68.34 obtDrqhxogxsewDfcDktb
209.190.21.100 bedmdFjkFhc4a noFjajakffieapvngdtpwxk
209.190.21.100 gdouk6Ss6nnykg66hvojc6txjsecuu
209.190.21.100 aphErvbtijj vulgctlslo
209.190.21.100 jgbhwntsdlprxcwogijI8orrw b8
209.190.21.101 DrbspcgyubxrpeikfiihxD mh
209.190.21.101 jvAhnviAjwwud8gymvewtcqhehgbAcytyqdxq
209.190.21.101 cvwkvl6kfujhqlujqblFl dffrepmrxdspmdFjq
209.190.21.101 obmJJkjtslbqreh6pwx6epruhptrpJbk

This is why I keep preaching that blocking by user agent only works for the legit crawlers that want to allow you to block them but the scrapers aren't playing by any of the old rules and I'm shocked these idiots just didn't use a browser string which has a better chance at least getting a handfull of pages of my web site until they get too greedy.

Sorry webmasters but the rules have changed in how this game is being played and you really need to block all non-browser agents and allow legit crawlers like Google, Yahoo and MSN by IP only. Any other method is just wasting your time as random user agents cannot be stopped by your old traditional techniques.

The blacklist is out, the whitelist is in.

Fuckers For Sale!

Found this little gem while playing around with AdSense testing relevant ads on dynamic search pages by using a GET vs a POST and just for shits and giggles added "?q=fuckers" to the end of the URL.

Well, much to my surprise one of the ads was extremely relevant and not only that, it appears you can compare places that sell fuckers on shopping.com and get a fucker of your choice for the best price!

I don't make this shit up, see it for yourself:



So much for family friendly AdSense ads!

Some days it seems so obvious

Today I was looking in the search engines for more signs of scrapers showing up in the SEs with my error messages and found a few more of these idiots. Then it crossed my mind that it would sure be nice to know which IP addresses that got blocked from scraping were assoicated with which web sites.

Well duh.

It's obvious I need to embed a bug in my text that ties the scraper to his web site so today starts the next wave of linking the abusers to their websites and opens up opportunities for automatic abuse reports that tie them all together.

Now this has me giddy, I can hardly wait!

Crawl Delayed RSS

Sometimes interesting solutions to problems just present themselves out of the blue as a discussion about Aaron Pratt's SEOBUZZBOX being supplemental results instead of the authoritative source gave me an idea.

What if you simply delayed updating your RSS feed until Google, Yahoo, etc. had already crawled your new content pages?

Theoretically this would improve your chances of being the first place that the new content was discovered and the news aggregators would then all become secondary sources for your information.

Would Google still make SEOBUZZBOX a supplemental result because of it's lower page rank or would fresh original content float to the top based on a first come first served basis and make the aggregator take the hit on duplicate content?

This is an experiment well worth trying, Aaron, you listening?

Anyone got any guesses what would happen?

Carnival of Scraper Sites

These circular scraper sites must be the worst sites I've encountered so far as all the scraped content is linked internally and no matter what you click on you just keep going round and round inside the scrapers web site until you click on an advertisement to escape.

These come in a couple of flavors such as the Moron Loop-de-Loop, the Loser Landing Strip and the Doorway Pages to Hell.

The Moron Loop-de-Loop sites contains snippets of your site and something that looks like it's a link to your web site but instead it links deeper into itself showing yet another page that is more thematic based on what you clicked. You can click and click and go round and round in this site because the only exits are AdSense exits. This is actually quite ingenious as you get more specific ads related to your area of interest if you keep clicking on more thematic links that seem to narrow your focus of interest but never provide anything interesting, except the ads.

The Loser Landing Strip is an interesting cloaking variation as this supposedly single page web site appears to have an infinite amount of cloaked pages behind it but only Google's IP addresses see that cloaked content, you only seel the silly landing page. No matter what snippet of page content you see in the search engine, no matter what nav links you click on in the web site, it's always that same landing page. All paths lead round and round to the same page so just click the ads already and give them what they want!

Door Way Pages to Hell is an interesting variant as it is part Moron Loop-de-Loop and Loser Landing Strip in that the first site you hit looks like a blog with links to actual web sites but the blog is actually the starting point on a trip to hell. Each link in the blog that says it's taking you to a useful web site actually has a link to one of their other scraped content sites instead, which of course link to yet other scraped content sites. It's a freaking AdSense pyramid scheme disquised like a domain park, it's absolutely frightening that someone would build such a large network just to keep a surfer going in circles until they catch a click.

Definitely a new low in scraping as the original content owner gets no value, not even a link, nothing but being used to get keywords and phrases to divert people to their sites.

Not sure it can get much worse than this but I've been wrong before!

Bullshit Gourmet

Sometimes when the little frozen food meals are on sale at the store I'll snap them up to get the 10 for $20 sale price and have a nice quick microwave lunch every now and then. Well, this weekend the fridge was bare and the usual brands weren't on sale at the market so what-the-hell let's see what these Budget Gourmet's are like since they're on sale.

Today is the moment of truth.

Opened one of these little boxes up and it has all the food, what little there is, on one side and all the sauce on the other.

Wait a fucking minute, half of this itty bitty box is reserved for a thin layer of sauce?

You must be fucking joking.

Completely unsatisfied ripped open yet another box.

Same bullshit, small pile of crap on one side, sauce on the other side.

BULLSHIT!

FUCKING BULLSHIT!

I'M STILL HUNGRY AND I'VE JUST BEEN FUCKED IN THE ASS BY A FUCKING FROZEN FOOD COMPANY AND NOW I'M FUCKING PISSED!

McDonald's anyone?

Cloaking Scrapers Busted

Yesterday was a sad day for one Russian scraper that just had a large number of websites busted for cloaking in Google and they were just reported to Yahoo today as well.

How they were busted is simple, they scraped my site!

Since my scraper stopper was deployed it's been giving all the scrapers a very unique message which they were happily scraping and including with their additional scraped web content. Wait a few weeks for the search engines to continue to crawl and update and VOILA! this message starts to appear on web sites. Clicking on the website in question and the message didn't appear but clicking on the CACHE copy of the page in Google and there it is, cloaked in all it's glory.

Report these sites and POOF! they are gone.

It's like shooting fish in a barrel and more fun than allowed by law.

Come on you cloaking scrapers, take my pages, I dare you...

Scrape Me Up Scotty

Finally I've had an out-of-this-world scraping experience when someone tried to offload about 500 pages via a satellite. Thanks to the fine people over at DirecPC for hosting this little bandit my scraper stopping efforts have now entered the realm of aerospace.

I can see the headlines now:
Earthly Bot Stopper Blocks E.T. Scrapers

It's safe to hazard a guess Ray Bradbury never, in his wildest imagination, thought people would use satellites to steal websites.

Scraper Sites are GOOD?

Usually I have a good deal of respect for Martinibuster as he's a cool dude and has some good insights into a lot of SEO and webmastering topics. However, when I read his article "Scraper Sites are Good for You - Surrender Your Content" my first thought was to print it on toilet paper so I could give it the proper respect it deserved.

Come on Martini, you can't be serious about letting people waste your bandwidth, steal your content, and spam the SERPs with your own junk just to give you links?

Martini claims "Scrapers: There is Nothing You Can do About it" which is totally wrong! I'm stopping those bots, AlexK's script snares them and BotBuster and Bandwidth Protector claim they will also stop them. Can't comment or endorse the latter products as I've never used them but they do exist as well as some others so webmasters not stopping scrapers just means they're lazy or cheap at this point.

Then Martini makes me spit soda across my keyboard with "Surrender to the Scrapers... It is Better for You". What a load of crap my friend. Just ask Aaron Pratt of SEOBUZZBOX how good surrendering to the scraper is as he's being lumped into supplemental results as the scrapers aggregators are being indexed first and Google thinks Aaron is the duplicate content which is just wrong.

So Martini says "scrapers do more good to you than harm" which gets the bullet list:

• Scrapers can put your content in supplemental results
• Scrapers can rank above you in the SERPs and get the first shot at AdSense and affiliate income using your content
  
• High speed scrapes of 100s pages/second are like DDOS attacks and knock servers offline until they stop scraping keeping visitors from clicking your ads
  
• When servers can't respond due to scrape attacks Google, Yahoo and MSN get time outs on pages and SERPs drop
• Rampant scraping can run up your bandwidth charges and you pay for their excess

Nope, no harm no foul, nothing wrong with scraping.

What I can state with certaintly is that since I've started blocking scrapers my SERPs and REVENUE are both up substantially and that's about the only major change I've made to my site recently.

Try reading one of Martini's other articles that made sense instead, he's really a nice guy, just slightly misguided on the topic of scrapers is all.

P.S. Doesn't scraper and bot stopping sound like a great session topic for PubCon Vegas this year?

Referer Spammer Revenge!

Today I caught some referer spammer bombarding my web server by hitting the same page over and over and over with only the referer changing.

To put it mildy, this pissed me off.

When I started looking up each domain I noticed something fascinating in that they all had the same AdSense account and were all registered at GoDaddy.

Recent topics on ThreadWatch about GoDaddy locking abusers domains provided a true inspiration today. Instead of wasting my time putting this asshole in my banned list of IPs and domains to keep him out as would be my normal routine, I reported him to both AdSense and GoDaddy abuse and will be waiting and watching to see if either of them take action.

If GoDaddy shuts his entire array of websites down this will be the best defensive action to take against referer spamming yet, I'll post the results if any when I see the domains go offline.

BTW, I also added referer spamming detection to my bot blocker today so I'll be snagging more of these idiots on a regular basis if this is a huge problem. I'm considering making the bot blocker automatically perform a whois on the domains when this is detected and send automatic abuse letters to the proper parties.

This could be fun ;)

Polish Robot You Can't Pronounce

Only a polish robot would be looking for polish websites on my server in Texas - sigh.

Anyway, we found Szukacz trying to snoop around but alas, it slammed into the great wall protecting my site.

Looks like it supports robots.txt according to the web page but who knows.

Burf Barf Puke

Someone in jolly old England unleashed Norbert the Spider upon my site which appears to be sent on behalf of the fine people at Burf that claim "BURF - Alternative Search Engine and Entertainment Portal"

Alternative to what, finding what I'm actually looking for?

Then I dediced to click on the "Your Ad Here" link just to see what one gets for the money to advertise with Burf and it's placement on a truckload of search sites I've never even heard about.

Well, it's cheap advertising but then again my URL would probably get about as much exposure putting it on the bottom of my shoe.

Yacy my Assy

Filed under "Who Needs Another Crappy Search Engine" we find something called Yacy that claims to be a P2P distributed search engine whatever the fuck that means. I'm translating that back into English as a free-for-all scrapefest bandwidth waster.

They claim it's "Easy to install!"

I claim it's "Easy to block!"

Toodles.

Anti-Social Bookmarking

Well here comes the new leech-of-the-week Susie came crawling and went head first into an error page. Sych2It claims that "Dead and out of date links are automatically reported" but I'm not sure how they would know for sure as they got a slap in the face instead of a web page.

Fine German Engineering? HA!

The AnonyMouse proxy slammed head first into my bot blocker.

Come on people, if you're gonna waste your fucking time writing an anonymous proxy server the least you can do is attempt to fake being a browser instead of making your user agent string your domain name.

Give me a goddamn break.

Big Decisions Time for Bot Blocker

Getting real serious about converting the bot blocker prototype into a product and all the agony that goes along with developing, launching and supporting a new product brings back fond nightmares, um, memories of products launched in the past.

Trying to determine if I should write it myself or hire someone to write it to speed it along, look for equity partners from the beginning, whether to even sell it as a product, open source it, or whatever the hell to do with it and the whole process is just maddening.

Talking to a few people the last few days, we'll see how it goes.

I knew there was a reason I've been claiming to be retired the last few years!

SuperBot Found My Kryptonite

Sorry you're not so SuperBot after all as you couldn't even leap over my index page without tripping and falling.

The author claims:

Unlike other offline browsing tools, SuperBot is fast AND powerful AND and easy to use...

Whoops!

Should now read "Just like all offline browsing tools it was stopped dead in it's tracks when it hit IncrediBILL's Bot Blocker"

Better hope your customer that tried to download my website doesn't come looking for a refund!

EUREKA! Proxy Detection Thanks to Idiots

Thanks to some sloppy work by some rank amateurs running proxy servers they pointed out a flaw that many anonymous proxy servers share that are now allowing me to automatically detect proxy usage and block the damn things in real-time.

Of course this doesn't work on all proxy servers but it caught 10 of them just today.

I should've spotted this happening weeks ago but at the time I came up with a different hypothesis for the data that presented itself which today, with additional clues, makes it obvious many of these hits are via proxy servers.

Wow, it's amazing how such simple revelations can rock your world so easily.

This is cool - blog ya later as I need to work on proxy busting now ;)

Scientific Search Halted

Here's another crawler that lost it's way called Scirus that claims to be a scientific search engine yet was trying to roam around my non-scientific web site. They claim to be powered by Fast but they were stopped dead in their tracks by my bot blocker that said not-so-Fast, heh.

Sorry boys, you need to find a new lab rat to play with.

Rufus is a Dufus

When it just couldn't get any sillier along comes something claiming to be RufusBot which claims to be a good little bot but others claim it's personal scrapeware.

I don't care either way as it's not scraping here and don't let the door hit you on the ass on your way out Rufus.

Proxy Mouse Ate the Poison Cheese

Too funny, my spider trap killed a mouse, ProxyMouse as a matter of fact.

These leeches are just like the other proxy service that strips all of my ads off the page by default and slaps their own Yahoo ads on the top of the page but NOT ANYMORE!

What's best is how I caught them thanks to MSN which somehow attempted to crawl my site thru their proxy. When MSN was crawling their proxy server just passed whatever user agent string, in thie case msnbot, thru to my server. Suddenly my server sees msnbot on an IP address that doesn't belong to Microsoft and SNAP! they are busted.

No more Yahoo income for you off my back, your filtering asses are done.

Buh bye, see ya, wouldn't want to be ya!

Yahoo Blogs Doing a Content Crawl

Not sure what good old Yahoo is up to exactly as this spider hasn't shown up in my spider trap before but Yahoo Blogs is attempting to crawl the content from my RSS Feeds directly. Not sure if they intend to display the content directly to the user or still redirect to my site so I'm not sure I'm letting them step past the RSS feed just yet.

209.191.83.13 Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )

Official Name: crc4.opn.search.mud.yahoo.com
IP address: 209.191.83.13

Maybe it's harmless and I'll let it pass, something to contemplate over a beer.

Why is WaveFire crawling?

Some Canadian consulting company called WaveFire has a bot trying to crawl for reasons not divulged on their website. Would've been nice if they posted something about what purpose they had in attempting to crawl sites.

64.141.15.109 Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com)

Official Name: search-d-02.internal.wavefire.ca
IP address: 64.141.15.109

Sorry, but your fire was put out and your spider was splattered.

Expanding Bot Blocker to More Sites

This week I'm going to install my bot blocker on 2 additional websites and see what level of abuse these sites are taking just to see what kind of an impact this technology could have on smaller less active web sites.

Don't get your tits in a twist just yet as it's still not converted to PHP and is still just a protoype no where near being a product for release.

On the more amusing side the message my site pops up when scrapers hit telling them they've been stopped has started showing up in the search engines as these automated scraping idiots haven't realized they're showing the world just how stupid they are.

What's more telling is which search engines show which sites with these messages vs. others that don't as I'm getting some insights into what some search engines are blocking as spam sites.

Blasphemers use IncrediBILL in VAIN!

Someone is using my name in vain to describe someone else in some big embittered bullshit battle about some religious horseshit.

Just what I need, now a bunch of idiots [you know who you are] will think I'm that person.

Fucking lovely.

Bend Over When Upgrading WebCeo

Finally decided to upgrade to the latest WebCeo and it did pop up a message about "losing reports" in that all existing reports should be printed before upgrading etc.

OK, big whoop, didn't need the old reports, could care less.

What that little message DIDN'T SAY was you would lose all projects, all profiles, and all that stuff and didn't even bother trying to automatically upgrade the data from the previous version leaving me to think all I was going to lose was the ranking information in the reports.

The new WebCeo version is now installed and pops up completely blank.

FUCK!

WebCeo needs to make those warnings a little more specific:

• You'll lose all reports
• You'll lose all projects
• You'll lose all profiles
• You'll lose EVERY FUCKING THING

Not that this was a complete crisis as I maintained a list of all the keywords for the ranking reports in a set of separate text files as WebCeo doesn't [didn't] have any easy way of just exporting my keyword list so I maintained it externally which worked out in the end as I just cut and paste the whole list of keywords back into the projects as I recreated them.

Back in business but the next time they release a major upgrade I may be tempted to try a different product instead as this was not amusing.

Excrutiating Back Pain

Pulled a muscle the other day which resulted in the brief hiatus from the blog as sitting in my desk chair long enough to write a rant was just too painful and I was in too much pain to use the laptop even as looking down pulled the muscle and hurt so I caught up on television watching instead.

BTW, when I'm in pain the level of cursing escalates to a new plateau so the blog may become TV-MA rated over the next few days until my back gets better as I need to release all this pent up hostility somewhere.

You've been warned so brace for impact.

The Ultimate Online Pharmaceutical

This is the topic of a bazillion spams every week which begs the question of which one of you limp dick assholes out there buy pecker pills via these mail order bullshit spams?

They wouldn't keep spamming me with this fucking garbage if one of you limp mother fuckers weren't buying these goddamn pills so just step forward so I can cut your fucking dick off and end my suffering thru these spams as a bloody stump doesn't need viagra, it needs a bandage.

Just ask your doctor for the pills you impotent bastards and leave me the fuck out of this.

Cache Hysteria Pandemic

WARNING - STRONG LANGUAGE ABOUT FUCKING MORONS

Everyone has lost their fucking mind on this topic and the fact that Yahoo and MSN have page cache doesn't matter as the only search engine being bashed over this topic is Google.

The world is not flat and the earth DOES NOT revolve around Google.

The non-stop "Google Google Google Google" mantra is getting old and you're all starting to sound like a bunch of fucking narrow minded cultists so get a life, get a clue, open your damn eyes and look around.

The best nonsense argument I keep hearing is that the less technically savvy authors out there shouldn't have to learn how to disable cache, it should be opt-out by default, blah blah blah wasting precious air spouting bullshit. OK, in an utopian world perhaps being cached would be opt-in but it isn't so just get past it and quit dwelling in fantasyland you delusional twits! Any author that can figure out how to get his content online and can set up meta tags for description and keywords can insert a fucking line to disable cache!

Holy fuck!

Have you all lost your fucking minds in that you would rather sit and whine for months and years about pages being cached instead of just inserting that one line into your template?

Got thousands of pages?

Maybe someone can get off their lazy ass and whip out a simple server side script that will update your entire httpdocs folder on the server inserting the no cache directive in all web pages.

How about fixing the tools used by the legions of technically dim-witted like Mambo/Joomla, WordPress, FrontPage, DreamWeaver, etc. and make cache opt-out the default in all your pages.

Or just drop that line in your template, all the templates should have it, you hear me template designers? Take those iPod ear buds out of your waxy ears and listen up - ADD IT TO YOUR FUCKING TEMPLATES!

Lazy whiny assed shitheads, you're really getting on my nerves.

Stop bitching and moaning and just FIX YOUR FUCKING SITES and the cache argument would go away on it's own in a few months and be a moot point.

Fucking lamos.

A Banner Exchange - How 90s

Well, I know what the competition was up to now as they just launched a banner exchange network.

Come on people!

Banner exchanges didn't work when they first came out as people with traffic didn't need them and people without traffic couldn't generates hits to bring in the traffic so it was always a waste of time.

Lame lame lame.

Should've saved that banner ad revenue and used it for beer.

C T Scans from Hell

Have you ever had the joy of a CT Scan?

Had one this morning and it's a lovely experience when they strap your ass to a small plank and zip you back and forth thru a nuclear reactor, not to mention all the radioactive dye they inject into your veins to see certain stuff.

I'll bet if I jerked off in the bathroom with the lights off right now my sperm would glow in the dark.

Today was a tale of two arms.

The nitwits in charge couldn't seem to find a vein to inject the dye into today which was probably because I downed a bunch of whiskey last night in my usual 'night-before-CT' anxiety drink-a-thon which dehydrates my dumb ass and thus makes the veins harder to spot.

Oh well, let them earn their pay to torture me.

Recently they just introduced paper underwear to put on during the scans so those of you that don't wear any crotch covers in the first place don't have your goodies waving in the wind for everyone in the room to see.

About 2 years ago I remember another CT Scan where the nurse yelled at me "close your legs, I don't need to see that" to which I replied "I'm wearing black briefs so all you're seeing is pink leg but if you really want I can whip it out for your viewing pleasure".

She shut the fuck up.

Bet she flunked anatomy and I'm sure she isn't getting laid when she can't tell thigh from ball.

Stupid stupid nurse, get your tubes tied, stop now before you pollute the gene pool.

Get a Grip You Loon!

Where does someone come off going nuts on me about a free listing on a free website?

Someone sent about a dozen emails and left a bunch of voice mails ranging from 7am-11pm while I was away for a couple of days jumping up and down wanting instant support for something that cost them nothing.

Come on people, it's FUCKING FREE!

This is the same reason I started dumping customers opting just for my own websites to get away from whiny assed people so DON'T PUSH ME ASSHOLES or the plug could be pulled!

Spammers Succeed Elsewhere but WHY?

Went looking for other sites potentially hit with the same spam garbage that my site kicked out and was shocked at just how vulernerable all these sites were.

Come on people, these spammers were trying to submit their garbage via an HTTP GET instead of a POST command so it's obvious there are some real shitty programmers out there just letting any old crap get submitted any way it can. Guess I never realized just how bad blog spam is as most places I frequent are pretty clean. However, what I saw today after reviewing a bunch of sites hit by the idiots knocking on my door is that most of the blame can be placed on shitty programming and poor validation techniques.

Case in point is this article "Some Things I Learned in 18 Years of Programming" and apparently in all those years form validation and anti-spamming techniques weren't in the list. Scroll down the page and you'll see what I mean, a real belly laugh if it wasn't so sad in the first place.

More blame could be put on the spammers but it's like locks, doors and theives. If you don't lock your doors and they loot your house then you get what you deserve for being complacent. Remove easy access simply by locking the door or installing a security system and the lazy opportunists give up quickly. That only leaves you to contend with the more sophisticated spammers but unlike bandits putting guns to your head, spammers are a usually a little easier to stop especially the greedy ones, without taking a bullet to the head.

Oh well, not my problem with the exception that the idiots running wide open spamware encourage the little fuckers to attempt it on my site.

Attempted Submit Spammer Too Stupid to Spam

Today I was looking for something in a log file just to see what was up and stumbled onto some HUGE repeated strings attempted to be submitted over and over, always different data,

Someone appears to be hellbent to muck up my directory but they went overboard and didn't make sure the submission validated so my form submission pages bounced hundreds of them.

Lucky me!

What a mess that would've been!

Dropped the IP in the "DIE VIOLENT!" database so we don't have to worry about it for a few minutes.