Giving Scrapers a Cookie

Some scrapers appear to actually use the cookies your site transmits just to make sure they don't get stopped in case you block visitors that don't use cookies. In order to turn this to your advantage you can save the original IP address in the first cookie issued. This seems to be snaring a few of them as I'm using that IP in the cookie, if present, to track these idiots so that when they come back under different IPs, assuming they use some proxy or AOL with rotating IPs, that cookie they keep transmitting allows me to link their continuing activity to the original IP address.

So much for the IP shell game you morons.

When Competitors Lose Their Minds

Sitting here minding my own business and this opt-in spam drops in my Inbox with a plea to buy advertising to generate some revenue for a new project for their web site.

Valley Girl sounds suddenly come out of my mouth:
"Like, Oh My God! Grody! Barf Me Out!"

Then the loud hyena laughter fills the air and dogs start barking blocks away.

Reading further they give the stats on exactly how many ads run maximum and that the placement costing a trivial sum of money is for a YEARLY rotation, not monthly but YEARLY.

Did some quick math and started yelling "YOU STUPID FUCKING MORONS!" as the amount being raised for this "project" is less than my AdSense pays in a few days while these numbnuts are horribly underselling the space and poisoning the well.

Hopefully my advertisers are smart enough to know that you get what you pay for and these wacky pleas for help show just what a weak player they really are.

Almost makes me want to call them and explain the facts of life but I'd hate to make them too smart if you know what I mean.

Idiots.

WebAbuse 2.0 Overnight Stats

Just to give some of you a hint that I'm not exaggerating about the amount of site crawling going on here's a list of 122 different IPs and agents automatically blocked last night that collectively attempted to access many thousands of pages. Some seem to have specific targets and only go after a few pages every time they return but others want to deep crawl the crap out of my site.

When you add it all up the bots are sometimes accessing more pages than the actual site visitors because this list doesn't include authorized bots.

FYI, don't be fooled by what you see just because the user agent looks legit means nothing as a human can't click on and read 200 pages in 150 seconds. It's possible there is an innocent or two that was snared, but considering what's at stake I don't really care anymore.

This is the future, run for cover:

12.221.77.114 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
128.2.220.167 PrivacyFinder/1.1
130.158.81.39 Wget/1.10.1
131.107.0.84 SandCrawler - Compatibility Testing
134.96.1.195 AnswerBus (http://www.answerbus.com/)
137.43.154.203 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
139.18.2.43 findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/)
142.167.88.250 internal zero-knowledge agent
144.131.251.29 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
151.24.66.200 Internet Explorer 5.5
162.40.193.253
172.169.142.20 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
172.203.82.76 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
193.165.250.22
193.42.229.3 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
193.47.80.43 Exabot/2.0
194.167.196.3 Wget/1.10.2 (Red Hat modified)
194.67.3.21 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050920 Firefox/1.0.7
195.101.0.67
195.159.130.14 ZoomSpider - wrensoft.com
195.27.247.70 ColdFusion
195.37.209.45
195.39.234.162
195.70.35.179 KummHttp/1.1 (compatible; KummClient; Linux rulez)
196.209.78.70 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5
201.230.91.192 Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
201.26.110.67
202.165.102.186 SpiderMan
203.10.224.58
203.113.238.60
203.113.238.60 Random
205.209.169.222 MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
206.188.0.11 Jakarta Commons-HttpClient/3.0-rc2
207.148.212.242 PHP/4.1.2
207.171.172.6 Java/1.5.0_04
207.58.161.116
208.185.247.74 PageBitesHyperBot/600 (http://www.pagebites.com/)
209.131.61.1 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
209.167.50.22 LinkWalker
209.178.137.175
209.18.119.138 Jakarta Commons-HttpClient/3.0-rc2
209.190.20.194 Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
209.237.238.225 ia_archiver
210.17.148.245 Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
210.173.180.156 ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
211.5.60.108 RSS_READER (mctwist@mail.dr-k.info)
212.117.84.230 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
212.117.84.230 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8) Gecko/20051111 Firefox/1.5
212.80.76.5 SeznamBot/1.1 (+http://fulltext.seznam.cz/)
213.133.123.154 libwww-perl/5.65
213.156.54.186 Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
213.176.109.234 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
213.203.184.30 InetURL/1.0
213.42.2.11
216.195.47.98 Snoopy v1.2
216.22.48.28
216.247.238.226 VSE/1.0 (vivisimolog@web121.com)
217.212.224.142 psbot/0.1 (+http://www.picsearch.com/bot.html)
220.210.177.118 RSS_READER (mctwist@mail.dr-k.info)
221.116.237.114 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
24.11.67.32 Java/1.5.0_06
24.177.134.6 aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)
24.19.240.172 Python-urllib/2.1
24.202.166.142 WebPix 1.0 (www.netwu.com)
24.216.179.135 Zeus 34366 Webster Pro V2.9 Win32
24.22.159.131 FyberSpider
24.242.26.149 Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
24.5.187.223 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
24.57.8.78 EasyDL/3.04 http://keywen.com/Encyclopedia/Bot
38.113.234.181 voyager/1.0
58.64.126.5
61.135.131.173 sohu agent
62.163.40.65 Java/1.4.1_04
63.229.208.79 NextGenSearchBot 1 (for information visit http://about.zoominfo.com/PublicSite/NextGenSearchBot.asp)
64.127.124.159 OmniExplorer_Bot/5.85a (+http://www.omni-explorer.com) WorldIndexer
64.141.15.119 Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com)
64.148.232.129 brfcaofenxv cdvP3k3xuesucrcxgPp3m
64.148.232.129 fWjnyc p ctcwmbbulcdeqw qew
64.164.63.175 Java/1.5.0_06
64.239.7.218 POE-Component-Client-HTTP/0.65 (perl; N; POE; en; rv:0.650000)
64.241.242.18 NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
64.38.240.97 Roffle/l.ol(compatible; MSIE 6.0; Windows NT 5.0;
64.40.115.34 Python-urllib/1.16
64.5.245.27 genieBot (http://64.5.245.11/faq/faq.html)
64.94.163.151 Jakarta Commons-HttpClient/3.0
65.19.150.208 OmniExplorer_Bot/5.88 (+http://www.omni-explorer.com) WorldIndexer
65.24.45.49 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5
66.117.176.20 Java/1.4.2_04
66.147.154.3 http://www.almaden.ibm.com/cs/crawler [fc14]
66.234.139.194 snap.com beta crawler v0
66.40.35.42 WWW-Mechanize/1.12
67.108.223.130 NextGenSearchBot 1 (for information visit http://about.zoominfo.com/PublicSite/NextGenSearchBot.asp)
68.127.10.143 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5
69.0.235.24 Topular/1.0
69.238.36.166
69.41.14.5
70.124.116.68 FavOrg
70.34.224.188 Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
70.49.144.182 Visual_Odyssey_Spider/3.0 (http://www.visualodyssey.com)
70.85.193.178 Poirot
71.102.140.247 envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)
71.137.197.195 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7
71.213.9.100 Lynx/2.8.5dev.7 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6b
80.219.233.222 EmailSiphon
80.255.64.42 SIE-CX70/54 UP.Browser/7.0.2.2.d.3(GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1
80.77.86.240 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
81.1.87.163 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)
81.155.34.158 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
81.19.66.38 StackRambler/2.0 (MSIE incompatible)
81.73.137.226 Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
81.83.46.233 Googlebot/2.1(+http://www.googlebot.com/bot.html) (Googlebot/2.1(+http://www.googlebot.com/bot.html); MSIE; Windows; SV1)
82.120.57.235 Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8) Gecko/20051111 Firefox/1.5
82.131.195.52 LapozzBot/1.4 (+http://robot.lapozz.com)
83.44.42.199 Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
84.148.107.62 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Larbin/2.6.3 larbin@unspecified.mail
84.148.108.134 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Larbin/2.6.3 larbin@unspecified.mail
84.81.17.28 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
85.101.47.187 Microsoft URL Control - 6.00.8169
85.108.164.241 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
85.125.153.160 larbin_2.6.3 larbin2.6.3@unspecified.mail
87.193.34.166 xyz

BTW, this was a slow night!

Zoom Zoom SPLAT!

How many fucking spiders and search engines do we need anyhow?

Apparently one more according to Wrensoft, the makers of leeches from downunder.

Obviously someone found another use for ZoomSpider and aimed it at my web site.

Even if they weren't getting in with that spider name the speed they hit my site would've blocked them anyway so Zoom my ass, you got NOTHING!

What's Microsoft doing?

Something hit my server today called SandCrawler and it appears to be coming from Microsoft.

Agent: SandCrawler - Compatibility Testing
Official Name: tide41.microsoft.com
IP address: 131.107.0.84

Anyone know anything more about this one?

Guess it doesn't matter as crawled into a brick wall.

Net Wooing

Another crawler blatantly disregarding my bandwidth and copyright is NetWu with several products to make a webmaster snarl.

My site was hit with this one:
WebPix 1.0 (www.netwu.com)

Must be ethical to write these things as long as you don't worry about the consequences of their use.

The Bus Stops Here

Another Leech 2.0 technology [some call it Web 2.0] called the AnswerBus got a flat tire when it hit my web site. Don't know what question they were asked but the answer was "HELL NO!". They were even nice enough to put a list of other potential wasteful sites that can be blocked as well.

RSS Reader Got Legs

Don't know what it is but something coming from Japan calling itself RSS_READER is trying to crawl but it can't get in.

It's polite too and checks robots.txt then hits the home page WHAMMO! tries again WHAMMO! checks robots.txt again as it's obviously confused then the home page again WHAMMO! WHAMMO! WHAMMO! banging it's head, tries one more round.

Probably someone's lame attempt to locate RSS files, so sad, too bad.

I'm sure it'll be back tomorrow for more fun and games.

Link Bait Manager

Well if you have to have an almost decent excuse to crawl my website all to hell it might as well be under the guise of being a reciprocal LinksManager which at least has some value.

Unfortunately, my web site is a directory with many thousands of pages and after observing their little bot fruitlessly flounder trying to find that reciprocal link it was obvious nothing good was going to come out of this so I blocked it.

If you really want to check links on my site I'd be more than happy to give you an XML API to do so but you'll crawl hundreds or thousands of pages looking for a single link over my dead body.

Buh bye LinksManager.com_bot, buh bye.

FYI, if you want to see a really nice but incomplete [didn't have this one] list of bots check out the database on Robots.org.

Everyone's a Critic

It's not like a I run around the internet spreading filth on other people's forums as they'd boot my ass off except on threadwatch which is a bit liberal when it comes to the four letter words. Besides, it would be just rude to misbehave on other people's websites, or at least I wouldn't do it using a traceable name and IP, I'm not THAT self-destructive. So this is my version of my "Fortress of Solitude" where I can go off on a tangent on anything I damn well please.

Now the critics are coming out in droves [ok, 2 or 3 critics] questioning my content and how I express myself. Let me say that although I do get some sort of perverse pleasure from those types of disapproving comments that's not what this site is all about as I'm not trying to shock anyone, well not too much anyway, ok maybe I try to cause a mild stroke or an occassional headache but really I mean you no harm.

When I initially started this blog I thought I would just do nothing except write clean little helpful technical articles, maybe even slap AdSense on it, and then the evil side raised it's ugly head. I haven't really seen the evil side in a long time since I ran a BBS back in the 80's, drew cartoons, and wrote all sorts of funny as hell but damning things. It crossed my mind to start up a second blog and let the evil side run rampant in it's own little safe haven and not poison the well of my good intentioned technical posts but that never happened.

Maybe it was procrastination or perhaps the need to finally integrate the good with the evil that overcame me and it was time to resolve my split online persona and I said [bet you can guess this one] "FUCK IT!" and the blog went south and never came back.

You have to understand that I'm a guy and I like to do guy things and have guy type conversations but I'm sitting at home working 24/7 with just my wife to discuss things most of the time. She's actually pretty liberal about most topics, way more so than those girly girls and metrosexual men, but there are just times when I cross that line from what she'll tolerate as a conversation and I get the "Save this one for your friends!" quip. That leaves me in a pent up state worse than a teenager in the back seat of a car with a date that won't put out.

Since this is my Fortress of Solitude there will be times that I might just scratch my balls in public and if I happen to offend someone I'm horribly sorry but the political correctness filter has left the building.

You have been warned and thanks for visiting ;)

Exploration Halted

Don't know who these guys are at Omni-Explorer but it always seemed like a legit group that claims to honor robots.txt and posts their IP range so on and so forth. I always let them crawl since they claimed to be venture backed which made me think something useful would show up eventually and being a Silicon Valley guy I'm always curious about venture backed start-ups. However, it's been YEARS now and the bot keeps crawling with no benefit for me that I can see in the near future.

Sorry guys, but your exploration has ended.

OmniExplorer_Bot is now officially arachnida non grata

Taking your Key away

In the blocked crawler du jour contest we call your attention to our latest entry from Canada called the Keyword Encyclopedia.

Load up all your little .htaccess files and drop "EasyDL" in the list of unwanted party guests.

Ta ta bandwidth waster!

V7ndotcom Elursrebmem Suspended in Gravy

Looks like the AdWords team doesn't have a sense of humor and missed the whole point of my most popular nonsense ad running on a nonsense search term.

Your disapproved ad:

Makes It's Own Gravy
Seriously, what kind of ads did
you expect on gibberish searches?
incredibill.blogspot.com

Ad Status: Suspended - Pending Revision
Ad Issue(s): Unclear/Inaccurate Ad Text

Interesting as I thought the satire was pretty clear but perhaps they couldn't get my web site to "Make It's Own Gravy" or maybe they took offense to being called a "gibberish search"?

FWIW, I guess Google really isn't in the AdWords game just to make money as that ad was the top earner racking up 1/3 of all the money I spent on that idiotic ad campaign!

Oh well, one down, 5 to go.

P.S. For you pundits out there I'm still on the fence about whether that whole ad campaign was my way of lampooning the contest or just a desperate cry for attention, you decide.

Yahoo's SpiderMan

Don't think I've ever seen Spiderman crawling before and APNIC claims it's from a registered block from Yahoo in China.

A lookup revealed:
Official Name: d24.search.cnb.yahoo.com
IP address: 202.165.102.186

APNIC claims it's them:
inetnum: 202.165.96.0 - 202.165.111.255
netname: YAHOO-ASIA-2

So now the big question is what's the advantage of letting an all English web site get crawled by a Chinese version of Yahoo?

Anyone have any insights on this?

It's a tangled Web we Copy

They call it WebCopier but it should be more appropriately called WebPirate.

The more these so-called products keep showing up the more I realize that the internet is mutating into something very ugly when copyright and bandwidth of the website owners isn't even a concern of the people building these tools.

It's starting to get depressing.

Harvest Gets Crop Failure

Caught Topular's hand in the cookie jar today.

69.0.235.24 - - [24/Jan/2006] "GET /myarticlesarenotyourarticles.html HTTP/1.1" 200 1302 "-" "Topular/1.0"

According to their website they do "Information Harvesting" without permission of course and no information posted about their robot or any other damn thing, but since it didn't check robots.txt there's no chance it's playing by the rules anyway.

Guess what Topular, your days of harvesting my shit are over.

Picture THIS!

We have yet another new bandwidth leech called PicSearch sucking the lifeblood of the internet coming to us from Sweden.

That's right boys and girls, add "psbot" to your blocked list.

I'm thinking about actually letting them index just one picture of me flipping them the bird with a caption "Pay my bandwidth fees you fuckheads".

Tools for Fools

Here's one to file in the KISS MY ASS column: Website Extractor

You may want to block anything with this user agent:
"Website Quester - www.asona.org"

Product claims to have the following benefit:

Website eXtractor saves you time and effort by downloading entire Internet sites (or the sections you stipulate) to your hard drive.

Let me help with this as I saved your customer even more time and effort as their sorry ass was blocked from downloading a single fucking page.

Assholes.

Not So Fav Icon

OK, when I pull a major blunder I do it right as minor fuck-ups are for amateurs.

Follow along carefully as this will all make complete senselessness eventually...

This problem all started sometime recently as it appears somewhere along the line my favicon.ico got zapped off my server and I never noticed nor did I bother looking at the logs close enough or I would've figured this out a few months ago.

Now imagine that my Apache configuration doesn't know the favicon.ico is an image and on a 404 error was actually displaying a 404 page for the missing icon.

Next, trying to capture more visitors instead of letting them see a 404 page and leave, possibly because of site maintenance errors, the 404 page at some point was redirected to my home page.

Last but not least, I changed my mind a few days ago and put my bot stopper code on the home page after seeing what scrapers could do with just that little amount of content.

Suddenly a small rash of people got banned with about 30 page views in 5 seconds. Looked at what was happening and these people hadn't downloaded 30 page views but had a shitload of requests to favicon.ico which looked very odd. Must be getting dense as it took a couple of days of seeing this before my little brain said "note the favicon.ico requests getting a 404 error".

Let's see what's going on here and try it:
http://www.youdumbasshole.com/favicon.ico

Up pops the home page!

Oh fuck.

So a browser or something asking for the favicon.ico about 20 times in a row loaded 20 404 pages which redirected to 20 home pages which tripped the scraper alarm and stopped them from accessing more pages temporarily.

Oooops!

Uploads favicon.ico, tucks tail between legs, hides quietly in the closet until the massive wave of embarassment passes.

Tales From the Crapped

If you're squeamish about bathroom situations, close your eyes while you read this.

So I'm sitting in the recliner and suddenly get a sharp pain in my side that feels like something large just outstreched vertically in my intestines so I stretched out in the recliner to try to relieve the pain, then it suddenly shifted horizontal causing me to spasm in the other direction. Feeling much like there's something playing X-games in my intestines this happens a few times from left to right, quite reminiscent of atomic diahrea that accompanies the stomach flu.

Great, just what I need to be getting sick.

If I were gay I would've thought it was the baby kicking but I digress.

Now comes the moment of truth, I don't know if I have to fart or shit, never a good sign.

Then without warning or fanfare comes the sudden emergence of the turtle head and the mad dash to the bathroom.

This was no ordinary trip to the bathroom, I had to get a LaMaze coach to help me with my breathing "Now PUSH!" ... "UHHHN"... "PUSH!" so I can only imagine this is somewhat similar to child birth as it feels like I have just opened up so large I could suddenly slide over the toilet bowl.

When the accompanying paperwork is done is when this traumatic trip to the bathroom reaches epic proportions.... never in my life have I seen such a thing, it's a mutant, it's HUGE! it's ENORMOUS! It's the Empire Shit Building standing entirely up the side of the bowl laughing at me as I stare at this forearm sized dung heap in horror.

What the hell did I eat?

Now the moment of truth, time to flush.

[cue theme from Jaws: bum bum bum bum bum bum...]

The water goes up, up, up, up and over the top while this big clinker just sits there mocking me without budging. [digression: Water water everywhere and not a drop to drink] I'm quickly deploying any handy towels as fast as possible so this turdnami doesn't make it to the carpet.

Anyway, you get the idea, a lot of urping and plunging came next.

I need to change my diet as that's some crazy shit that I don't need.

Pressure to Perform

Now I'm starting to get nervous as the pressure is on now that we're up to 5 whole readers and people actually think this blog is funny when it started out semi-serious. My wife is accusing me of using gratuitous cursing just to pander to my main audience (you both know who you are) and claims the blog will end up sillier than Scrubs or worse yet when NBC cancels me like they did Will & Grace.

Not to mention Sebastian is claiming I have tits and suddenly people looking for tranny porn are landing here via MSN which is probably a step up from the usual horse sex crowd. [side note: most of the horse sex requests are coming from the Middle East, camels aren't in vogue anymore?]

Then someone shockingly saw right thru my facade:

When reading his blog one word and one word only comes to mind, curmudgeon.

My wife understands that comment as my online game playing persona is CrankyBaztard which she claims was no accident that I picked it as my name since it was a such a natural fit.

So now I'm all nervous, to curse has become a curse, to not curse is worse.

Ah fuck it.

TIP: Asking your wife if you can "Play Moses and part the Red Sea" once a month does not work.

Too Stupid to Scrape

This one should be filed under "Oh My God What a Moron" as someone slammed my server today attempting to download my content with a minor twist - they got the case on all the page names wrong!

All of my pages use Upper/Lower case file names like Page1_Blah.html

Look at this shit:


0.0.0.0 - - [23/Jan/2006:14:14:48 -0600] "HEAD /page1_blah.html HTTP/1.1" 404 - "-" "-"
0.0.0.0 - - [23/Jan/2006:14:14:48 -0600] "GET /page1_blah.html HTTP/1.1" 404 1302 "-" "-"
0.0.0.0 - - [23/Jan/2006:14:14:53 -0600] "HEAD /page2_blah.html HTTP/1.1" 404 - "-" "-"
0.0.0.0 - - [23/Jan/2006:14:14:54 -0600] "GET /page2_blah.html HTTP/1.1" 404 1302 "-" "-"

The page names were all correct but lower case which sure as hell won't work on a Linux server!

Never thought I'd see a scraper too stupid to scrape!

That's one dumb asshole.

Poor Kitty, Too Funny

Just about hurt myself ROTFLMAO when I saw this entry in my webstats as some poor distressed pet owner searched MSN for "if cat barfs is there something wrong" and landed on Cat Tales of Horror as the #1 result.

Tears are still running down my face, that poor person, I almost feel bad for them.

Ya know what?

Fuck it.

They should get a dog if cat barf worries them so easily as they'll soon not have a single spot anywhere in the house that isn't somewhat slightly stained by brightly colored cat vomit.

TIP: When decorating color coordinate with your brand of dry cat food.

Bot Busting Could Have Significant Savings

The total reduction of 10's of gigabytes of bandwidth on my server alone due to bot stopping makes it easy to imagine that having such bot busting technology installed on every server in a datacenter could be an enormous savings in bandwidth second only to stopping spam. This technology could result in a small windfall for small hosting companies constantly being squeezed by service providers for more money by allowing even more customers on the same bandwidth currently being stolen without permission. Conceptually, blocking these leeches from an entire network could result in all sorts of additional savings.

The need to upgrade motherboards, especially for busy shared servers, could be significantly lessened. The older motherboards currently straining under the load, similar to how my newer dual Xeon was, would suddenly be more than adequate to continue to grow a business without any additional equipment expenditure. Being able to get a little more juice out of older equipment could allow datacenters to spend more on infrastructure instead of further lining the pockets of service providers.

Now the problem is how do you sell a product to hosting companies that could actually impact their revenues by cutting bandwidth usage which results in additional charges?

Simple.

Offer the bot busting technology as an additional paid service labelled as a content and copyright control technology that reduces the ability of scrapers and aggregators from using their content without permission.

Seems like a natural for a control panel plug-in for Plesk, CPanel, etc.

More revelations coming soon.

I'll show you PRIVACY...

Another useless excuse for a website called PrivacyFinder crawls a few of your pages looking for your p3p privacy policy and combines that oh-so-useful information with Google and Yahoo search results.

They just got a taste of MY privacy policy today as their bot couldn't get past my front door.

If this little bit of information was useful to include in the SERPs and customers demanded to see it then Yahoo and Google could just include it in the first place.

Go find your privacy elsewhere and keep your ass off my server.

Stupid.

Link Your Ass to My Foot!

More link exchange bullshit as this excerpt was one of the best lately:

Dear Dipshit,

We are pleased to inform you that your website http://www.youreanawesomewebgod.com has been listed in our site http://www.stupidfuckingpondscum.org

You can find your link at: http://www.whogivesaflyingfuck.org/index.php

Links exchange means that you need to put reciprocal link to us.

Preferrable on this page http://www.notonmywebsitenotonyourlife.com

Our robot have checked this page for 5 times but our link wasn't found.

Threats about not linking to us and losing your link blah blah I think I shit my pants etc.

Guess what?

Your robot can use that phillips head screwdriver attachment and thoroughly fuck itself.

What kind of morons are harassing me with this nonsense?

I'll link to your web site about the same time I become a rock star and my balls start slapping Pamela Anderson's ass after a concert which is NEVER!!!

Now go away, eat shit and die, leave me alone.

Block this Server Side Browser

Here's a real winner that must have a page of links somewhere that spiders crawl via their proxy server as it was looking like an attempted page hijacking in Google when it was discovered.

These slimeballs download your site via the proxy server, strip your javascript so the frame busters don't work, and slaps their ads for pecker pills on the top of the page.

Ran into a similar site from China last year but they were embedding AdSense into the page and Google took care of them in short order.

Currently you can block them both via IP and the referrer as their proxy isn't terribly clever yet and leaves their domain name in the referrer string.

Some SEOs are as DUMB as Pet Rocks

Which is an insult to Pet Rocks.

Some of you know that one of my main websites is a niche directory that has been online since before the DOT COM boom and Yahoo was still mostly used as an adjective describing people living in the south.

Anyway, some genius SEO/Web Designer/Pinhead submitted a listing for a client to my directory today and did the following:

• Put a bunch of keywords in the title, yes, nothing but keywords
  
• Put a bunch of keywords in the description, if you can call it that
• Put in HIS email address wrong with his company name butchered all to hell
• Pissed me off

I swear to god my cat can hack up hairballs with higher IQs!

Come on, it's a DIRECTORY, not a SEARCH ENGINE so pull your SEO head out of your ass and act accordingly!

Is the word TITLE too complicated?

Is a simple DESCRIPTION of what's provided at the website mind blowing?

Couldn't you just cut and paste your email address without me having to mop up after you?

Showing that my IQ is above room temperature, unlike that SEO, I was able to figure out who this guy was in 2 seconds by pasting the butchered domain name on the email address into Google which ran it's "people are stupid filter" and it figured out what it was supposed to be and landed me right on the guys web site.

Quick check with WHOIS.SC popped up the same name as owning that domain that submitted the site and the IP address was geographically similar enough that I was sure it was the same person.

How do people get work for themselves when they can't even get their own email address right?

More importantly, who in the hell even types in an email address anymore with all these auto-form filling tools built right into the browser?

What a dweeb-assed shithead, I need a drink...

Copyright Crawlers? Get the gun!

The boatload of irrational 404 errors showing up in my web stats suddenly makes a lot of sense as my spider trap snared some asshole today bombarding my server with a shitload of requests for pages that don't exist.

Well guess what?

It appears the crawler was looking for any stolen copyrighted pages and abusing my server in the process.

Who gives these copyright protection services and tools the right to fucking attack my server requesting 100s of pages a minute and suck up my bandwidth dumping 404 pages?

Now that you can't scan my site maybe I'll just locate some of the pages you were trying to find, steal the damn things, and put a big FUCK YOU on the top of each of those pages.

Not like you'll ever see it but everyone else can laugh their asses off.

Fuckers.

We Want Your Ad Space

The next deluge of emails after the link swapping morons are those greedy little make believe media companies that want my ad space.

Must get 10-20 of these a week and they all want to sell my link space.

Let's examine the potential here as they claim they'll only take 25% or 50% to "sell my space" depending on which bullshit artist company sends the email.

If they really looked at my site they would notice I also sell ads direct and take 100% of the revenue plus some AdSense which is about 60%.

So do they really think they can compete with my direct sales plus Google AdSense?

Both pay many thousands a month already and both are pretty reliable as far as getting paid, especially my direct ad sales, so what's in it for me?

On the rare occassion that I've had these types on the phone I ask two questions:

1. Do you have significant ads in my industry or 100% targetting to my topic?
2. Can you guarantee that you can replace my ad income at my current levels with your advertisers?

So far the answer has always been NO on both counts so why do they persist?

Idiots.

Hi! I'm Looking for Link Partners!

Those emails always start the same with minor variations:

My name is BLAH.
I work for http://www.blah.com/
I am looking for link partners whose sites would be of benefit to our visitors.
Your site would be an excellent fit.

Sorry pal, but you obviously didn't visit my site.

My site is about WIDGETS and not web hosting, pills to make your pecker hard, weight loss, real estate or any other damn bullshit you're selling.

You know what would be an excellent fit?

My foot up your ass.

Now go away boy, you're bothering me.

Scraping Down, Ad Revenue Up!

Somewhere in the past I rambled about my revenues getting stomped when greedy crawlers and thieving high speed scrapers hammered the crap out of my servers locking out legitimate visitors for minutes or hours during what could be considered a DOS attack based on the speed of requesting 40K pages.

Well here's a big shock, now that I've effectively stopped their asses cold my ad revenues have returned to levels they were about 3 months ago before the crawling became a near epidemic.

Makes you wonder if some of my listings were suffering in the Big 3 search engines because of this as my review of several log files showed that sometimes all 3 search engines were attempting to crawl at the same time some of the big time scrapers were tying up my server. That's when I suspected the Big 3 SEs timed out on those pages and lowered the listings in the SERPs like it did when I recently caused a problem and they've taken this long to get back where they were.

You know it was getting real bad when one night I get a wake-up call at 2am by my sister-in-law, who has sites on my server, calling to complain it went down 20 minutes ago while she was working on her web site and slightly afraid she did it. Pings to the server and various services confirmed it was up and running but unable to respond to data requests as it was just tied up, completely overloaded, and wouldn't even give me a prompt in SSH so I could kill the tasks and block the intruder. However, on that night I let it ride as I hate forced reboots that can crash a database (no sleep then) and sure enough it came back to normal an hour later as I expected.

Anyway, just thought someone that's been reading this saga might find this interesting as I certainly don't think it's a coincidence the ad revenue is bouncing back at the same time the crawlers have been beaten to a pulp.

PageBites Job/Resume Scraper

Well guess who stepped into my spider trap today but yet another robot from yet another start-up aggregator site called PageBites that thinks they have entitlement to make money sucking my bandwidth.

Like I've been telling you all for a long time now you need to raise your sheilds to OPT-OUT your site to all crawlers and whitelist just the ones you want to stop this pandemic crawling of your sites. They profit off your backs, gobble up bandwidth costing more money, then muck up your web stats to the point you can't tell your advertisers how many real impressions they really get as crawlers are becoming a pretty decent percentage of so-called visitors on a daily basis.

Search engine spam, email spam, blog spam, now crawler spam.

It's really starting to get to the point that I think turning my spider trap into a product so everyone can take control of their own sites is looking more viable by the day.

You can all take your robots and go home, party over, your ass never got past the home page.

Firefox Scraping Giggle of the Day

Just had an amusing scrape attempt from something that claims to be Firefox with HTTP_REFERER set properly as it crawled and everything but it attempted to rip 299 pages in 211 seconds which set off the alarms instantly and automatically shut them down after only a fraction of the page requests at that speed.

Maybe there's a Firefox plugin that did this and if that's the case it's just getting Firefox users blocked but I don't care as this kind of behavior isn't welcome.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; xxxx) Gecko/xxxxx Firefox/1.0.7

I was amused watching it happen in real time but you lamo scrapers didn't really think it would work did you?

What my scraper blocker used to do was just stop counting page requests after it hit a specific threshold and temporarily disabled the scraper to stop them. However, my latest modification keeps counting continued attempts so after the initial threshold trigger blocks them the page counter just keeps going to see how many pages they really wanted. Eventually the scraper will set off a second level threshold trigger that gives them a permanent ban automatically if the total page requests are too extreme.

Pretty obvious when it just keeps going that there's no human at the controls.

Too funny - got anything better to throw at me?

Boycott Bellsouth

Everyone is going on and on about the Bellsouth drama where the good old boys at BellSouth are trying to charge (blackmail) internet companies a premium for using Bellsouth's backbone.

You can all pontificate about these blowhards at BellSouth all you want but my strategy is more straightforward.

Put your money where your mouth is.

Get rid of DSL and switch to a cable modem, satellite or anything except DSL.

Toss your landline phone in the trash and use VoIP, cell or anything else as long as it isn't BellSouth.

We aren't in BellSouth territory, we're in PacBell country, but we were also sick of all their overpriced services so we virtually freed ourselves of all RBOC services except a bare-bones $25/month landline mostly for incoming calls and emergency backup for internet dialup for those rare times when the cable modem goes offline.

Besides, we have a better phone service package with our cell phone so they can take all their overpriced features and go pound sand.

Basically, if people bail from BellSouth in masses they will get the message loud and clear when those year end multi-million dollar corporate bonus packages the execs are fond of go down the crapper.

This isn't the 80's anymore and pretending to be a monopoly when you really aren't is just stupid.

So Many Rants, So Little Time

Yesterday I was all riled on up on a bazillion topics and instead of ranting all day decided "Fuck It!" and worked on my websites instead - not like you fuckers pay the bills.

Brief synopsis:

Gov't wants Google Searches:
They just want to know what we're searching for, not who's doing it, but as I see it as the government is stepping over their boundaries and Google is vying for martyrdom by turning them down. Sorry, but I don't see the church annointing St. Google anytime soon so if it doesn't contain personal information like IPs and such just turn it over Google you link baiting media hounds.

GoDaddy Shuts Down Websites:
Any moron that hosts with GoDaddy and can't abide by the terms of service gets what they fucking deserve so stop whining already, it's getting old.

German Judge Shuts Down Wikipedia.de:
Parents didn't want their son's name in the Wikipedia so they sue and now it's in 10,000 blogs instead just because some Oktoberfest liver transplant candidate masquerading as a German Judge doesn't know shit about the internet - PRICELESS

Last But Not Least:
Don't expect to get laid the rest of the week when your wife overhears you washing your hands at the sink muttering comments about "pussy fingers"

Until tomorrow...

Scrapers Don't Like Being Blocked

The last week has been getting more interesting as my banned scraper log file shows some rather interesting trends as they are squirming and thrashing trying to get around all the traps.

The most amusing is the ever changing user agent strings as they are definitely testing to see if I'm filtering based on specific user agent criteria and mostly they are right as everything is banned except http clients.

All of the legitimate search engines are being permitted based on their range of whitelisted IPs so trying to pretend to be Google, Teoma, Slurp, etc. will just instantly ban their IP for the day and repeated attempts might ban it permanently.

Almost as much fun as shooting fish in a barrel.

Courts Give Wendy's Chili Hoaxers the Finger

Sometimes you have to wonder about jurors as it's OK to kill your wife if you're O.J. or molest children if you're M.J., but mess with Wendy's Chili and your ass goes to jail for 9 years.

Guess people have their priorities.

Competitor Jumped the Shark

Nothing makes your morning like waking up to find an email from your competitors latest mass mailing explaining how he's working on his web site and all these improvements and fear sinks into your gut that you're about to be destroyed by something awesome.

You click the link with dread expecting to see COMPETITION 2.0 and as luck would have it you see HILARIOUS 2.0 instead.

I swear on a stack of religious mumbo jumbo that this guy used to have a site I considered a threat and now it looks more like some high school kid is doing his web design and things are broken all over the place.

Either he's thrashing trying to get some juice out of his site or he's lost his mind and it's about to go down in flames but either way it looks like a win-win for me based on what I'm seeing.

Google Analytics Accuracy Bullshit Challenge

Have you played with Google Analytics?

Has the happy horseshit syndrome settled in yet?

My web site shows the following stats:

1 direct access
2 http://domain1.com
3 http://www.google.com/search
4 http://domain2.com
5 http://search.yahoo.com/search
6 http://domain3.com
7 http://domain4.com
8 http://domain5.com
9 http://domain6.com
10 http://domain7.com

However, Google Analytics shows:

1 google
2 yahoo
3 (direct)
4 msn
5 aol
6 http://ga-domain1.com
7 http://ga-domain2.com
8 ask
9 aolsearch.aol.com
10 search

Best I can determine is Google combines all Google sources such as Google.com, Google.ca, Google.co.uk, etc. which makes it look more dominant as a single source but overall makes the actual weight of the individual Google sites merged into one big ass murky pile of BULLSHIT!

Worse yet is from my actual log files my #1 domain referrer shows as #62 in analytics.

Sorry Google, you can fool some of the people some of the time, and AdWords advertisers most of the time, but here at IncrediBILL's Random Rants we call this BULLSHIT!

Have a nice day.

Bot Busting Crawler Experiment Complete

Many weeks and log file combings after The Great Anti-Scrape Off started it's become quite obvious that the effort was an enormous success.

The last bit of technology was deployed a couple of nights ago to challenge robots masking as humans seems to be stopping the last of them so it would appear that my site is now reasonably safe from typical crawlers and bots.

If someone has access to 10,000 IP addresses all bets are off but most scraping and crawling operations, except those that appear to be hiding behind AOL, seem to have fairly limited resources.

The last couple of tricks deployed include:

• Multiple checkpoint profiling to identify bots masquerading as humans
  
• Randomized challenge techniques with anti-blow-thru detection so that the typical captcha defeating techniques won't work
• Adaptive time monitoring for hit and run bots that seem to think they can get small chunks at a time and come back later under the radar for the next chunk

There may be other things going on out there in the wonderful world of scraping but it would take a fairly sophisticated scraper to bust through what's now currently in place.

The technology seems to work fine so far with up to 30K page views a day but it would be interesting to see how it would perform with 100K or 1M page views a day. What might be a bit challenging is the current implementation does quite a bit of database churn but for a medium-sized sampling that's only tracking a few hundred visitors at any time which is fairly insignificant.

In the end my scraper stopper is only protecting my database of content so any crawler can access about 10 pages without question such as the home page, about us, contact us, etc. so it will be painfully obvious to them that they are being blocked from delving deeper into the site.

The benefit to this approach is that all of the hard rules being used to block access to the full content doesn't break access to other technologies like the RSS feed which appears to have all sorts of crappy homegrown readers that don't identify themselves. However, when the greedy homegrown readers try to behave like a crawler and step into the site to grab the content linked from the RSS feed they are blocked unless expressly whitelisted.

The additional benefit to allowing a handful of top level pages to be crawled is that the web site doesn't automatically drop out view of lesser search engines or up and coming technology which would happen with a more harsh approach using the .htaccess file blocking all access to any pages. Additionally, there is a nice message on every blocked page letting them know they're probably seeing that page because they are an unauthorized crawler and legitimate crawlers may contact the webmaster and petition for access.

Basically, my web site has become OPT-OUT to any aggregators, crawlers or scraping thieves and now they will need to ask for permission to be let inside and profit from my work. Assuming it's a mutually beneficial proposition then I'm sure I'll let them crawl the site.

Now comes the million dollar question of whether to convert this to PHP and attempt to find a market or just keep it under wraps and much less conspicuous so the scrapers can't study what I've done and find any loopholes in the technology to exploit.

One final thought:

Could you imagine an entire internet that is OPT-OUT from crawlers?

The ability for the next Google to crawl the web to prove their technology would be a severe challenge!

Local Radio is Dead

Now that I've had a Sirius Satellite Radio for a couple of weeks I'm hooked.

Normally when I'm driving around in the car I'm sitting there punching one button after another looking for music in a sea of commercials or looking for some talk radio worth a shit and find nothing. Prior to Sirius my little Zen Micro had become my default music player in the car so at least I could listen to something I wanted to hear and pay attention to my driving instead of the non-stop button pushing.

Heck, now I'm listening to more radio than I have in many years as we just got the home docking station so it's Sirius in the car, home and office, it's on a LOT.

Thanks to Howard taking us kicking and screaming into a subsciption radio service as I'd SIRIUS-ly be missing out on the radio revolution if I hadn't made this switch.

Scraping From AOL Users Possibly Confirmed

There has been past speculation that someone is hiding behind AOL's proxy servers doing scraping and tonight I just happened to catch it live and decided to try something.

The pages were downloading at a slow clip via AOL with a user agent like this:

Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; .NET CLR)

The minute I blocked them they tried about 10 more URIs with that user agent string and it suddenly changed to:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR ; .NET CLR ; Something I removed)

To me this proves there is someone cloaking under the bank of rotating AOL IP addresses and they just happened to download enough to catch my attention this time and tried a non-AOL browser once I stopped them.

Time to implement a little more sophistication in my bot blocker.

Sorry slick, you aren't.

BUSTED!

Cursing at Scrapers

Just for giggles I went looking to see what idiots were scraping and reposting my random rants and sure enough I found some asshole with a snippet of one of my rants in some made-for-adsense aggregator site and sure enough that page was running PSAs.

Listen up fuckwads, if you're gonna steal my shit then you better filter it to make it "family safe" for AdSense.

Fucking morons.

V7ndotcom Elursrebmem Punted by Google

Doing a few repetitive searches on Google just to see what ads were showing up resulted in this every now and then:

Your search - v7ndotcom elursrebmem - did not match any documents.

Does that mean some servers aren't updated yet with this SEO contest?

Not sure why people are having a contest to get to the top of a defective search engine.

Pay Per Crawl

With the web 2.0 aggregator craze at an all time high maybe it's time for a new business model of Pay-Per-Crawl. That's right, all the start-up leech sites wanting to waste our bandwidth should be sharing some of their VC loot with us just for the privilege.

I'm not talking about ripping anyone a new ass, but perhaps $5 per every thousand pages crawled would help cover my costs of dedicated servers and bandwidth.

Heck, if it wasn't for all the crawlers in the first place my prime site wouldn't even need a dual Xeon server to handle the load so all the bots are definitely running up my expenses so why in the hell shouldn't they share some of that cost?

Oops, they are sharing some of the cost now as I locked them all out so they can't profit from my hard work.

Too bad, so sad.

I'll take a check, money order, VISA or MASTERCARD to let your crawler back in but don't you dare ignore the crawl delay or anything else in my robots.txt or back out you go!

Who's Yer Daddy?

Did anyone notice advertising on the V7ndotcom Elursrebmem SEO contest?

So just to be a me-too I put up one serious ad:

Publishers Earn More
Free report on maximizing your ad
revenue - nothing to purchase
incredibill.blogspot.com

Then I put up one to make them go HUH?

Cat Tales of Horror
Cuddly Pets Terrorize Owners
Laugh, Cry, Run for your Lives
incredibill.blogspot.com

What possesses me to do this shit?

I'm old enough to know better, but spending my own money just to make people go "what the fuck is wrong with him?" may be a little over the top even for me ;)

v7ndotcom Elursrebmem Battle of Yahoo vs MSN

Yahoo leads the pack claiming about 151 results for V7ndotcom Elursrebmem

MSN in a close second returns117 results containing V7ndotcom Elursrebmem

With sad ass Google returning only 10 results for V7ndotcom Elursrebmem

This is nothing new as I've been reporting Yahoo and MSN regularly tear Google a new ass when it comes to new content ever since I started this blog and Google is still in slow motion. Google is behaving more and more like your slightly slower older cousin with a learning disability.

Blank User Agents

Who in the hell is using blank user agents on the web?

I can't tell if these are SpamBots, Scrapers, Trackback Pings, all of the above but none of them are getting thru so it doesn't matter.

They must naively think if people block per user agent string that if you have no user agent string you'll slip thru the cracks which might've been true until my paradigm shift switched from blacklisting to whitelisting.

You people still blacklisting, and you know who you are, are wasting your time on a no-win scenario that just doesn't work.

V7ndotcom Elursrebmem

The great race to win the SEO contest for "V7ndotcom Elursrebmem" is on and it's actually surprising to see results so early in the contest.

SEO's get your V7ndotcom Elursrebmem's in early and often and may the best V7ndotcom Elursrebmem placement win!

Yes, this is a lame attempt at V7ndotcom Elursrebmem stuffing.

Blocking Reveals Odd Traffic Hits

Now that I've tightened the noose even tighter on bots and user agents there are some really odd hits showing up in my auto-blocked log file when analyzed for any false positives.

Something very odd has surfaced with a lot of single page requests coming from seemingly random IPs with no identification whatsoever, just the page request, but doing a reverse DNS lookup can be very revealing.

One such revelation was something I've never heard of called Covenant Eye's with an anonymous spider hitting my site. Since Covenant Eyes is a paid service it's too bad since they aren't going to be collecting any more free informaion by scanning my site. If you people figure out I'm blocking you and you really want access bad, maybe we can work out some financial terms for accessing my site!

Cat Tales of Horror

When you first get your new cat you think about all the fun things like playing with the cat dancer, petting the warm fuzzy critter sitting on your lap purring, and all that pet lover crap.

What you get in reality isn't just a pet, it's a four-legged festival of bodily malfunctions designed to horrify, shock and destroy everything you own.

Don't get me wrong, I love my cat, but if you could install a zipper on his mouth and a cork in his ass life would be easier.


Strong ID

That's what it said on his papers when we bought the little fucker:
STRONG ID

We had no clue what that meant, he responded well to his name?

A few days later we figured it out.

Our cat's shit was radioactive and smelled so bad it could send people running out of the room.

One day, we're sitting in the living room over 50 feet away from the litter box and this god awful smell comes wafting down the house as people run screaming and urping in terror and gather under the kitchen ceiling fan for protection.

Can someone explain to me how a very tiny kitten dropping a very tiny tootsie roll can make it smell like the fucking sewer just exploded and overflowed into my house?

Feeling much like Indiana Jones in the Nostrils of Doom I pull my shirt up over my nose and yell "I'M GOING IN!" and run into the bathroom, scoop that one tiny fragment from the litter box and flush it down the toilet and diffuse the situation.

Gives me some respect for people that work in bomb squads right about then.

Luckily, this odiferous situation goes away when we get the little bastard neutered.


Barfy the Vomit Slayer

Twelve years later (ok, 2 days ago) my wife is running thru the house holding the cat yelling "NO NO NO NO!" heading for the bathroom after scooping up the cat from the furniture as he sat there in full blown hairball hacking mode with with his tongue hanging making that oh too familiar sound:

WHUCK! WHUCK! WHUCK!

Luckily, once again we get him into the bathroom just in time to blow up on the tile instead of the sofa, bed, carpet, etc. Just like the postman that always rings twice, the cat always barfs twice so you have to close the bathroom door until he's done with his yakfest.


That Sinking Feeling

First, let's start with the fact that my cat refuses to drink water from a bowl and stands in the bathroom sink and screams for someone to come turn on the water. It's like having a 2 year old always asking for a drink. At least the cat is nice enough to yell when he's done so you can go turn the water off, a polite pet is always a bonus.

One day a few years back, after turning on the water he just keeps going back to the sink, over and over, for maybe thirty minutes. I'm wondering what in the hell is up, too much salt in the cat food?

Suddenly, my daughter starts screaming and yelling from her room and I run in to see the cat about 7 feet up in the air, on top of the TV which is on top of the armoir, spewing buckets of water cascading down the armoir like one of those fountains in the park. Trying to get the cat off the TV makes matters worse as he goes the opposite way spewing water as he runs. Eventually he jumps off on his own running in a typical cat panic to hide under the nearest bed after the waterfalls have stopped.


Who needs an alarm?

One morning I was sound asleep and woke up to some odd noise in the room:

WHUH WHUH WHUH WHUH

I look around the room and much to my horror I see the cat sitting atop the armoir catapulting balls of cat food across the room. His head turns back and forth and "WHUH!" another ball of brightly colored cat food shoots about 3 feet onto the carpet.

Nothing I'm going to do about this until it's over as I'm not taking a cat bomb in the face trying to get him down - the horror of it all.


Follow the Bouncing Ball

The cat comes running out of the bathroom one afternoon with something bouncing along behind him and he just keeps running around from room to room with this little thing bouncing around behind him.

I know this is going to be bad and I should probably turn and run in the other direction but instead I catch the cat to see what in the hell is going on.

It appears kitty has somehow consumed a few of my wife's hairs and now has a bouncing ball of shit chasing him attached to 3 hairs still hanging out of his ass.

Just lovely, you know someone has to pull these hairs out of his ass, and you know who that someone is going to be.


Carpet Crawler

Ever see a dog or a cat sitting ass down on the carpet pulling themselves along by their front feet?

That's what my cat does when he hasn't been drinking enough and his shit gets stuck half way out his ass - runs out of the litter box and scoots down the hall until suddenly a little tootsie roll pops out on the carpet.

Have I mentioned I love my cat?

Computer Monitor

The cat is always on top of my computer monitor, as a matter of fact he's there snoozing right now as I'm typing this. Good thing the little fucker can't read or I'd probably lose a layer of epidermis for this rant.

Anyway, one day he comes flying out of the bathroom and jumps on the desk and monitor with a vengeance and I'm startled even more when he suddenly goes ass down on the top of the monitor vents and starts going in circles.

Fucking lovely - BOING! Tootsie roll.

Fine, now it's time to get the disinfectant and the paper towels to clean off the brown streaks on top of the monitor.

Did I mention I REALLY REALLY LOOOOVE my cat?


It's 3AM, do you know what that smell is?

My wife wakes up at 3AM yelling about the horrific smell in the room, then she notices the smell is coming from the cat laying next to her in bed. Then she notices some stinky smelly goo all over the bed near his tail.

Did you know cats have skunk like glands in their ass that need periodic cleaning?

Neither did we until it just unloaded in our bed!

Nothing more fun than changing sheets and deodorizing a room at 3AM


What's Cooking?

One fine Saturday we're having lunch in the other room and suddenly a smell starts wafting across the house so I ask "Honey, you leave something on a burner?" and she replies "Nope, and it's not coming from the kitchen either".

Suddenly the cat looking scared shitless bolts into the living room - never a good sign.

So I start walking around the house to see where the odor is coming from and as I get closer to my computer room the smell of burning circuit boards fills the room.

One look at the monitor and I see the problem as smoke billows out the top of the vents which are covered in cat spew and my HP M90 is officially toast.

ISN'T LOVE FUCKING GRAND?

OK, maybe love is only half-grand as that monitor cost $500 when it was new.

Fucking cat.

Incremental Genius and Scraping Epiphany

Some times ideas strike and you just go EUREKA! when you know you've hit a home run.

Earlier I posted about sending trash data back to bad bots instead of just blocking them and that idea turned into a genius brainstorm.

SEND THEM ADSENSE BLOCKING WORDS AND STUFF KEYWORDS!

Sending scrapers piles and piles of nasty crap as a result of their unwanted crawl of your site could disable AdSense from their page, get them banned from AdSense, Google or BOTH.

How about a nice page of this:

Don't scrape my site or I'll get even with guns guns guns bazookas firearms guns guns guns nude erotic suicidal dead corpse etc. repeated over and over.

NOW we're going to have a little fun.

Some days a nice ephiphany just puts a little bounce in your step doesn't it?

Forbidden or Trashed Response

From what comments are posted on various forums the most common thinking is telling bad bots they've been forbidden from the server with a 403 error to make them go away.

However, if the scraper already has your content from previous scrapes the best method may be giving the bots placebo data so the idiots will just let it run until they've trashed all previous copies of your content.

Gonna give that a whirl for a few weeks and see what happens, should have some angry scrapers soon ;)

Gotta Stop Blogging

Wasn't paying attention today and my inbox was overflowing with people trying to give me money to advertise on my sites.

Nobody appreciates this shit anyway and it certainly doesn't pay the bills!

Who am I kidding, I'll have 4 posts before you know it...

Scraper Thoughts

Do lazy scrapers just download other scrapers sites?

Q: Why doesn't Google ban scrapers?
A: Professional courtesy

Q: How many scrapers does it take to change a lightbulb?
A: None, they'll just download your lightbulb

Q: What do you call 50 scrapers going over a cliff in a bus?
A: A good start

Q: How do you kill a scrapers?
A: Slam the lid on their head when they try to take a drink

If you like this stupid shit let me know and we'll post more.

If you don't like this stupid shit, tough shit, write something better

AdSense Running Low or Slow?

Something I've been noticing the last few days is my AdSense ad units seem to either be low on inventory or slow as they are inserting a lot more PSAs than usual.

When I noticed this I reloaded the page a few times to see what was going on and it's hard for me to fathom that reloading a page with 3 ad units suddenly has 2, then 3, then 1, then 2, then none, then 3 and so on and so forth.

Would either indicate there is a lack of ads available or perhaps AdSense just can't connect to the ad server fast enough and they drop a PSA when it panics during a time delay.

Brings up many questions:

• Could AdWords be running low on inventory?
• Are advertisers just broke after Xmas?
• Perhaps a new experiment in showing less ads for more revenue?
• Maybe the network is just overloaded?
• Could YPN be stealing that many advertisers?

Whatever the reason, it doesn't seem to be impacting my revenues so far.

Yahoo Labs?

Never seen this Yahoo Address before, wonder what they were doing?

Name: rlx-2-4-10.labs.corp.yahoo.com
IP address: 66.228.182.210
OrgName: Overture Services (formerly Goto.com)
NetRange: 66.228.160.0 - 66.228.191.255

Accessed just a single page in the middle of my web site somewhere, very very odd.

So I expanded my log search based on the NetRange:

66.228.182.203
66.228.165.141
66.228.165.140
66.228.182.202
66.228.182.210

Just a handful of pages, very odd indeed, but one of the direct hits attempted to load an AdSense PSA alternate ad so was it a human or a new bot that executes Javascript?

Handful of hits, didn't load any graphics whatsoever or things a normal browser would do, and hit one page twice.

Very curious

Bot Busting Primer vs Security Concerns

Now that I'm pretty sure my bot busting techniques are working like a charm the big dilemma is at hand.

A. - Do I completely disclose all the bot busting techniques so that others can bust these scraping assholes too?

or

B. - Do I keep some of the secrets to myself so the scrapers can't adapt and appear invisible to the naked algorithm?

It's a real catch-22 in that disclosing what appears to be solid scraper stopping techniques could unwittingly let me get scraped all over again. Most likely my web site with 40K pages would still be safe from a complete scrape as you just can't hide that kind of activity but more subtle "update" scrapes just culling the most recent content additions would be easier to slide under the radar.

What to do, what to do...

At the moment, I think I'll do nothing except document it for my own purposes.

What happens after that is anyone's guess.

SNAP.COM? You Must Be Joking!

Wow, haven't heard of Snap in ages and saw some silly little bot claiming to be a beta crawler from Snap so my curiosity was riled up and I pulled up their web site. If you haven't been there lately you need to go just to see it's kinda cool but so slow you'll get a giggle.

The search results has a preview window that shows a snapshot of the web site and what's odd is one of my web site previews shows fine in Internet Explorer but in Firefox gets a generic screen with the message "The website http://www.snapsucksbigtime.com may not preview properly".

What the fuck?

They also frame people's sites which my javascript doesn't permit, maybe that's the issue.

Don't know, but why they can show my preview in one browser and not the other yet everyone else seems to work in both is just bizarre.

Free Anti-Virus and then some

What was shocking to me wasn't that the Google Pack contained mostly free rehashed stuff, it was the 6-month Norton Anti-Virus trial that raised my eyebrows. They must get some kick back from Norton to have that placement in the Pack otherwise why wouldn't they just include the free home version of AVG anti-virus?

Been using AVG to protect my laptop for a long time and it seems to work just fine.

Heck, where was the Open Office Bundle or Putty the free SSH client?

Now if someone just could build a free version of PhotoShop then there would be no reason to pay corporate America for software ever again.

We Will Block You

Sung to the tune of We Will Rock You with apologies to Queen

Scraper with a bot made a big web crawl
Scrapeins’ from my site be on Google for all
You got snips on yo’ site
You big web plight
Blockin’ your bot all thru the nite

We will we will BLOCK YOU!
We will we will BLOCK YOU!

Scraper you’re a pond scum low life ass
Postin’ in the forums gonna kick my ass
You got my stolen content
You better repent
Clickin’ your ads til the money is spent

We will we will BLOCK YOU!
We will we will BLOCK YOU!

Scraper you’re an moron not too bright
Bleedin’ from your eyes would just make my night
You got my crap on your site
You big web plight
Maybe AdSense will ban account tonight

We will we will BLOCK YOU!
We will we will BLOCK YOU!

NOFOLLOW Straight To Jail, Do Not Pass Go

The next trick up my sleeve in the war against scrapers and badly behaved bots is to cloak NOFOLLOW tags into the pages when it's not a major search engine requesting the page..

Basically the concept is by changing the home page, site map and site wide navigation links to contain NOFOLLOW on all major links it will stop legitimate bots at the front door and they will be allowed to crawl a few pages by default. However, any crawler that follows links flagged with NOFOLLOW will get the hammer dropped on their ass.

One added touch is there will be a few spider-trap pages that always have NOFOLLOW on them, never allowed to be crawled by any bot, just like the current robots.txt file has some traps in it as well.

Building a better spider trap, how far will this obsession go?

Hysterical User Agent

Someone that obviously doesn't know their ass from a hole in the ground tested my bot trap with a user agent name of "MSIE" claiming to be coming from "www.av.com", or AltaVista if you didn't know, and I just about fell off my chair laughing.

Sure don't know what they're giving customers to smoke over on Cox cable but I sure wish Comcast would get some of it and share.

Word and Internet Explorer are Loathesome Beasts

Wrote an article in MS Word, posted it to Blogger, looked just perfect in Firefox.

For some reason today I took a look in Internet Explorer and there is all sorts of shit in the page like this you don't see in Firefox:

<!--[if !supportEmptyParas]-->

<!--[endif]-->

Now I'll go spend time stripping some garbage out of the HTML thanks to you stupid MS jackoffs.

Microsoft just take your bullshit HTML conversion from Word and your stupid Internet Explorer and hunker down in a corner somewhere and go fuck yourself.

WHAT DO YOU WANT?

I'm perplexed as this blog has a duality of users thanks to MSN thinking we're a porn site and the rest of the world thinking I'm a techie dude, so I have no clue what my readers want to read.

Do you come for the latest research data I'm posting or do you come for the full tilt rants that I tend to make when I'm pissed?

So much to share, so little time, so let me know if you want more technical garbage or prefer it when I get all riled up and step off the deep end and make a total jackass out of myself as there's only so much blogging to go around.

Maximizing Advertising Exposure

Many web sites make their money strictly on third party advertising programs, which are becoming more of a challenge as various disruptive technologies are thwarting webmasters from capitalizing on their content. To get advertisements in front of all visitors you need to deploy a battery of techniques that can maximize the likelihood of advertisements being seen by everyone including selling direct advertising.

Typical advertising methods and technologies

Various advertising models exist to meet the needs of both merchants and publishers of varying technical ability.

• Pay Per Click (PPC) ads such as AdSense or Yahoo Publisher Network which rely on javascript to function
  
• Cost Per Impression (CPM) ads typically served by 3rd party networks, such as FastClick, showing as banner or text ads rely somewhat on javascript but may work without.
• Affiliate Programs, such as Amazon or Commission Junction, are typically just 3rd party banner servers but rely on cookies to track sales for the website.
  
• Direct Advertising consisting of sponsored links, banners, and text ads being served by 3rd parties like AdBrite, hosted on third party services or locally on the web server.

Weaknesses of advertising technology

Each type of advertising has fundamental flaws in that it only works well assuming that the visitor to the web site has that technology enabled and ads are blindly inserted into web pages. This blind technique may work somewhat for the majority of the website’s visitors but the odds are that a significant amount of ad technologies may be failing.

• Blocked cookies render affiliate programs useless
• Disabled javascript stops PPC programs from working
• Blocked 3^rd party images stop banners from displaying although the links are still present
• Ad blocking technologies like Norton Firewall typically disable all 3^rd party and local CGI technology ad servers.
• Contextual ads may display Public Service Ads (PSAs) due to stop words in your text, 3^rd parties framing your site or a visitor using an anonymous proxy server which alters the page name originally indexed.

Simple Method to Maximize Ad Exposure

Considering many visitors to your website may only see a single web page then your best bet of getting any advertising exposure whatsoever is to use a “buffet” approach with a little taste of all types of ads on a single page. What this combination of ads does is improve the odds that your web page will display at least one ad since each type of advertising relies on different technology. This is not to condone putting a ton of ads on your web page to the point it looks spammy so design the ads into the site intelligently.

• Include a PPC ad which assume javascript is enabled
• Include an affiliate banner or box ad which assumes 3^rd party ads and images aren’t blocked and cookies are enabled
• Include affiliate text links which won’t track impressions if 3^rd party ads are blocked but will track sales if cookies are enabled
• Include a CPM banner ad if you want, assumes 3^rd party ads aren’t blocked.
• Embed direct advertising or banners such as a sponsored link, and these should be directly included in the page using SSI (server side includes), PHP or some other technology that cannot be detected and disabled in the visitors browser.

The best way to describe this technique is “slinging mud at the wall and pray that something sticks”.

Simple Fallback Ad Strategy

Any advertising method that uses javascript that doesn’t automatically supply a <noscript> alternative, such as AdSense, should have one added so ads display when javascript is disabled and it doesn’t violate your AdSense terms of service either.

Example:

<script>
… AdSense Code…
</script>
<noscript>
… insert non-javascript banner or text ads here, affiliate program, etc.
</noscript>

Alternative Ads for PSAs

To further maximize your advertising exposure you need to make sure that all contextual advertising, or any other that provides this option, have alternative ads defined so that the ad space is being fully monetized and not displaying free Public Service Ads (PSAs).

Implementation will vary depending on which service you’re using, and may even be able to be daisy chained such as AdSense loading an alternate ad that also can’t be displayed thus loading another alternate ad, and so on and so forth. Recommendations are that alternate ad chains shouldn’t be longer than 3-4 advertising networks in depth otherwise the time to display the ad could take too long and the visitor will be gone from the page or have scrolled and missed it entirely.

Dynamic Adaptive Advertising

Truly maximizing the revenues on a website requires some server side programming that can detect disabled javascript, disabled cookies and banner blocking. How this works is that a series of probes is inserted into the first page displayed to a visitor and the results of this probing cause the appropriate adjustments to the advertising being served in subsequent web pages.

Cookie Detection

Setting a cookie can be done a couple of ways such as sending the cookie from the server and setting a cookie in javascript which will indicate to the server if both cookies and javascript are enabled in one shot. When cookies are disabled any form of affiliate tracking won’t work and should be avoided in subsequent web pages.

Javascript Detection

Another method of detecting javascript being disabled is to embed an image being served and tracked by a PHP script in the NOSCRIPT tag of a javascript probe. When javascript is disabled any form of PPC advertising (AdSense) should not be in subsequent web pages.

Example:

<script>
… Set a cookie here to prove javascript is enabled
</script>
<noscript>
… Load a dynamic tracked image to verify javascript is disabled
</noscript>

Banner Blocking Detection

Something as simple as a cgi-bin based page counter can be used to detect banner blocking software as Norton’s Firewall tends to block images being served by a cgi-bin application as it’s assumed to be an ad server. When banner blocking is detected most 3^rd party advertising networks and some local ad servers won’t work and should be avoided in subsequent web pages. However, if cookies are enabled affiliate text links will work but impression tracking for these links won’t function.

Dynamic Advertising Matrix

	Affiliate Ads	CPC Ads	CPM Ads	Embedded Direct Ads
Cookies Disabled	NO	YES	YES	YES
Javascript Disabled	YES	NO	YES	YES
Banner Blocking	NO	MAYBE	NO	YES

Summary

By using the various techniques outlined above anyone from a novice webmaster to an experienced web professional should be able to maximize their revenue potential.

Looking forward to your success stories!

Free Internet Could Slay Telephony

Everywhere you look people are talking about free wifi and free internet as if this commodity pays for itself. This must be leftover thinking from the dot bomb days where everything is free but you make it up on volume.

What these naive sacks of shit don't realize is that services such as Skype, or worse yet something FREE like VoIP on Yahoo Messenger, will also run on any free internet infrastructure. Unless these free internet connections are throttled to speeds to slow for VoIP these well intentioned public works projects could end up undermining the very backbone of the telephony companies we depend on for a significant amount of the internet.

Can you imagine that telcos could lose so much money that landlines, DSL and internet connection fees could skyrocket for the rest of us not in a free internet area or the companies simply collapse and go bust leaving the internet in shambles?

Tread lightly with all this hippy inspired communal free shit as your free internet could ultimately cost way more than you ever imagined.

Bad Bots Bad Bots, What Ya Gonna Do?

Looks like more of them hiding now trying to slither thru my site under the radar and it's just not working because my new visitor control panel, aka radar detector (hehe), let's me see this activity at a glance without breaking a sweat.

Today's idiot bot-du-jour was much like yesterday's, a more well behaved variety of slow crawling scraper masked as a human, but stepped in the spider trap and >SNAP!< lost an IP address.

What I'm starting to think is this is just too much fuss and a better idea might be to just break all the navigation for spiders by converting everything to javascript navigation and then supplying authorized robots the normal version of navigation in <noscript> tags.

The only issue that needs to be resolved is whether or not the search engines would deem this cloaking as the content both the search engine and the end user sees would be identical, only the technology of the navigation would change based on who requested the page.

The other idea I'm tossing about is to simply insert a captcha randomly after so many pages views so that the bot would just be stopped dead in it's tracks opposed to a human that would type in the text and continue on their path. Humans rarely ever get into hundreds of page views and interjecting captcha's after about every 40 pages over and over would pretty much bring bots to a screaming halt.

Of course there are blow-thru captcha tricks where scrapers ask humans on other sites to enter the captcha data needed to get past these traps, but setting a series of random traps with random code on random page names might just make it too hard for the scrapers to accurately identify a captcha and they'll simply download hundreds of captcha pages instead of content.

More crazy ideas coming as this episode of web warfare evolves.

Bad Bots Trying Harder

Today some stupid bot came slow crawling like a human and masked itself as Internet Explorer but left telltale signs of a robot.

You ready for the best tell-tale?

Shockingly it looked at my robots.txt file as the first file of my site, probably trying to avoid my spider-traps.

Talk about stupid, you scrapers need a life as looking at robots.txt *IS* a spider-trap in itself!

MUAHAHAHA!

Ooops, let a secret out ;)

AdSense PSAs Per Browser?

Well Google AdSense is really pissing me off today as my Firefox browser is showing PSA ads on my own websites home page yet the same page on the same computer is showing ads in Internet Explorer.

Other pages show ads in Firefox, just not my own home page.

Some days this bullshit really gets old and at this point I don't even care why I'm seeing PSAs on certain pages in just one browser as I'm thinking I should be seeing Yahoo ads there instead.

Sirius-ly Folks, Stern Rulez

Since Howard Stern has left the airwaves my wife is having withdrawl like a junkie as she finally realized that not only is Howard missing during her morning commute but the Best of Stern is also not on and after listening to Stern for more than 15 years she decided she needed her morning fix for Stern since she doesn't drink coffee.

Today, 2 days before he premieres on Jan 9th, we run around from store to store like assholes trying to buy a last minute Xmas present because everyone is sold out of all the good Sirius models and the home docking stations. Nobody had anything but the crappiest cheap Sirius models my wife didn't want, not Circuit City (more coming Tuesday!), not Best Buy, not even our neighborhood Target that said they had them when we called and we go there to find out the pimple headed freak didn't now shit from shinola as all they had in stock was XM.

Whoever said the public is over Stern and nobody would follow him to satellite is out of their fucking minds as these stores told us everyone was making a run on these Sirius radios just to hear Howard next week.

Finally, we find that Fry's Electronics in Palo Alto still has a couple of limited models in stock and only the car model, not the home docking station, so we make a mad dash down 101 to get to Fry's before they're sold out as well and walk out semi-satisfied with a unit for the car.

Now for the fun part!

It's night and quite dark outside when wifey heads out to install/activate her new Sirius unit while I'm sitting here banging away on the keyboard. About 10 minutes later she comes fuming back into the house with the Sirius and all the wires and shit going "This unit is dead, doesn't even work out of the box".

For some reason [self preservation?] it seemed like it was time to get involved so I go over and examine everything she's brought with her back into the house and something odd catches my eye in that there are TWO car adapter jacks and one of them ends with the connector for her cell phone.

Tempting fate "Um, honey, did you try to actually plug this in or were you trying to charge your phone?" and with that "OK, let me try it." grabbing all the unit and cables, sans the phone charger, and heading to my car.

Plugged everything in and it powered up first time!

Amazing how it works when you actually have the right adaptor plugged into the car.

What was even more amazing is the Sirius really did what they claim and worked first time with no problems transmitting over the FM radio in the car or directly via a cassette adapter.

We went out for a quick 30 minute drive up in the hills (mountains for most of you) and down by the bay and the reception was flawless in all terrains we tested in that short jaunt so we were impressed.

Now we'll stay tuned for the Monday broadcast of Stern and see what happens!

Bot Blocking Blacklist vs Whitelist

Since launching my scraper stopper all sorts of little bots have been caught and categorized with the number of them simply staggering and it doesn't seem to stop with new ones popping up daily.

My initial pass at blocking bots was the automated snare to stop them in real time, then let me review it and install a permanent IP block if I wanted or simply add the crawlers' user agent string to my blacklist and block any occurance from any IP, or all of the above..

Then I had an epiphany as this blacklist approach is simply too much work as the number of bots out there is just too much for any one person to deal with sorting out and banning, even with the assistance of automatation to stop them in real-time.

My new approach which was sweet and simple was just the opposite using the whitelist approach and now all bots are initially banned with only the ones I deem worthy being added to the whitelist after the fact.

Compare the difference in the approach:

• Previously blacklisted bots in the hundreds and growing
• Current whitelisted bots less than 10

So it seems to me that the best policy to apply is block all, log the user agent strings requesting access, and only let in the ones you want and not vice versa or you'll just be spinning your wheels messing with every new scraper or Beta search engine (and there are a lot) to hit the internet.

A Tad Pompous

Here's a little office mind fuck I did about 15 years ago that is still somewhat amusing.

One of the software engineers I'll call "John Jingleheimer" [not his real name] just got his new name placard for his cube.

I walked up while he was sitting at his desk and read it out loud for all to hear "JOHN J. JINGLEHEIMER, PHD" then just to mess with his mind muttered "Well, isn't that just a little pompous" and walked off.

A week or so later I walked by his cube and the name plate just said "JOHN JINGLEHEIMER" and I stopped dead in my tracks, barely able to contain myself, and said "John, why did you change your name plate?"

John says "I overheard your comment and thought you were right that brandishing my PhD on my name plate was a bit pompous"

To which I replied "Hell John, you earned that PhD and should proudly display it. I just thought that middle initial 'J' sounded a bit snooty!" and walked off leaving John gape mouthed realizing that he'd been set up.

Sometimes I'm just a bad boy.

Massive Search Engine Bounce Back!

Mind blowing results that my positions and traffic in the search engines seem better than ever since fixing that little BUG that almost de-listed my site entirely. Very odd behavior seeing traffic levels rise above their previous levels as my pages rapidly return to all their former positions and some new bonus positions.

More importantly, for some reason this morning AdSense seems to be giving me $1/click so it's possible smart pricing was reset when the Google Mediabot couldn't crawl the site for a few days.

Kind of a web based version of a high colonic seems to have cleansed the search engines.

Maybe de-listing a few thousand pages every now and then is good for a web site?

Don't think we'll be testing that theory soon!