Sirius-ly Folks, Stern Rulez

Since Howard Stern has left the airwaves my wife is having withdrawl like a junkie as she finally realized that not only is Howard missing during her morning commute but the Best of Stern is also not on and after listening to Stern for more than 15 years she decided she needed her morning fix for Stern since she doesn't drink coffee.

Today, 2 days before he premieres on Jan 9th, we run around from store to store like assholes trying to buy a last minute Xmas present because everyone is sold out of all the good Sirius models and the home docking stations. Nobody had anything but the crappiest cheap Sirius models my wife didn't want, not Circuit City (more coming Tuesday!), not Best Buy, not even our neighborhood Target that said they had them when we called and we go there to find out the pimple headed freak didn't now shit from shinola as all they had in stock was XM.

Whoever said the public is over Stern and nobody would follow him to satellite is out of their fucking minds as these stores told us everyone was making a run on these Sirius radios just to hear Howard next week.

Finally, we find that Fry's Electronics in Palo Alto still has a couple of limited models in stock and only the car model, not the home docking station, so we make a mad dash down 101 to get to Fry's before they're sold out as well and walk out semi-satisfied with a unit for the car.

Now for the fun part!

It's night and quite dark outside when wifey heads out to install/activate her new Sirius unit while I'm sitting here banging away on the keyboard. About 10 minutes later she comes fuming back into the house with the Sirius and all the wires and shit going "This unit is dead, doesn't even work out of the box".

For some reason [self preservation?] it seemed like it was time to get involved so I go over and examine everything she's brought with her back into the house and something odd catches my eye in that there are TWO car adapter jacks and one of them ends with the connector for her cell phone.

Tempting fate "Um, honey, did you try to actually plug this in or were you trying to charge your phone?" and with that "OK, let me try it." grabbing all the unit and cables, sans the phone charger, and heading to my car.

Plugged everything in and it powered up first time!

Amazing how it works when you actually have the right adaptor plugged into the car.

What was even more amazing is the Sirius really did what they claim and worked first time with no problems transmitting over the FM radio in the car or directly via a cassette adapter.

We went out for a quick 30 minute drive up in the hills (mountains for most of you) and down by the bay and the reception was flawless in all terrains we tested in that short jaunt so we were impressed.

Now we'll stay tuned for the Monday broadcast of Stern and see what happens!

Bot Blocking Blacklist vs Whitelist

Since launching my scraper stopper all sorts of little bots have been caught and categorized with the number of them simply staggering and it doesn't seem to stop with new ones popping up daily.

My initial pass at blocking bots was the automated snare to stop them in real time, then let me review it and install a permanent IP block if I wanted or simply add the crawlers' user agent string to my blacklist and block any occurance from any IP, or all of the above..

Then I had an epiphany as this blacklist approach is simply too much work as the number of bots out there is just too much for any one person to deal with sorting out and banning, even with the assistance of automatation to stop them in real-time.

My new approach which was sweet and simple was just the opposite using the whitelist approach and now all bots are initially banned with only the ones I deem worthy being added to the whitelist after the fact.

Compare the difference in the approach:

• Previously blacklisted bots in the hundreds and growing
• Current whitelisted bots less than 10

So it seems to me that the best policy to apply is block all, log the user agent strings requesting access, and only let in the ones you want and not vice versa or you'll just be spinning your wheels messing with every new scraper or Beta search engine (and there are a lot) to hit the internet.

A Tad Pompous

Here's a little office mind fuck I did about 15 years ago that is still somewhat amusing.

One of the software engineers I'll call "John Jingleheimer" [not his real name] just got his new name placard for his cube.

I walked up while he was sitting at his desk and read it out loud for all to hear "JOHN J. JINGLEHEIMER, PHD" then just to mess with his mind muttered "Well, isn't that just a little pompous" and walked off.

A week or so later I walked by his cube and the name plate just said "JOHN JINGLEHEIMER" and I stopped dead in my tracks, barely able to contain myself, and said "John, why did you change your name plate?"

John says "I overheard your comment and thought you were right that brandishing my PhD on my name plate was a bit pompous"

To which I replied "Hell John, you earned that PhD and should proudly display it. I just thought that middle initial 'J' sounded a bit snooty!" and walked off leaving John gape mouthed realizing that he'd been set up.

Sometimes I'm just a bad boy.

Massive Search Engine Bounce Back!

Mind blowing results that my positions and traffic in the search engines seem better than ever since fixing that little BUG that almost de-listed my site entirely. Very odd behavior seeing traffic levels rise above their previous levels as my pages rapidly return to all their former positions and some new bonus positions.

More importantly, for some reason this morning AdSense seems to be giving me $1/click so it's possible smart pricing was reset when the Google Mediabot couldn't crawl the site for a few days.

Kind of a web based version of a high colonic seems to have cleansed the search engines.

Maybe de-listing a few thousand pages every now and then is good for a web site?

Don't think we'll be testing that theory soon!

Competitor Pulls Better Fiasco

Sending blank pages to the Big 3 search engines that I did was admittedly bad, but what I saw on a so-called competitor site tonight is priceless:

Error

Sorry, you have reached either a non-existent site or the site has been suspended (or deactivated) due to Disk Space Violation / Exceeded.

Site Owner: Please refer to this help page for information.

Makes me smile as this crap-of-the-month-club web site is the only one I've ever complained about to Matt Cutts as I don't mind others being ahead of me in the SERPs, it happens, all fair in love and war but this crap site appears to only be at the top of the heap by gaming DMOZ.

Just what I wanted to hear that someone got to the top by putting spamming ad-ridden gargage at the top of a major keyword using that corrupt so-called authority site.

That's ok, they can't be making nearly as much money or they wouldn't appear to be on shared web hosting with a disabled site.

Looks like I get to do some of the giggling this time ;)

Bug Eats Website, Film @ 11

File this little post under the category "So Embarrassing He Should Keep His Mouth Shut" or maybe more correctly titled "IncrediBILL makes IncrediBLUNDER".

A very minor bug in my scraper stopper just about sent me to the poor house as it was inadvertently blocking Google, Yahoo and MSN because of a very minor programming flaw.

I had cut/paste some code and it said:

if( isGoogle() )
exit();

Instead of:

if( isGoogle() )
return();

Meaning that my entire database of pages were being returned BLANK instead of having any content when search engines crawled and sure enough all three search engines have already gobbled up all these blank pages.

This would've gone unnoticed for a long time except I went looking for a page reference to compare with something and it was blank, so I checked all page references from the site and all the static pages still had text and all the database pages were blank.

OOOOPS!

Looked at the code, fixed it real quick and we'll see if everything is back to normal in a week after they've all been crawled again.

In the meantime, I bought a box of Depend as a prophylactic measure to save my chair in the event the worst happens.

Dogpile of Shit

Dogpile was never terribly useful in my opinion but I somehow ended up on their site today for some odd reason and just about fell off my chair when their top results included ads as if they were actual search results.

Stunned and shocked there I sat staring at the top search results showed with the source listed as "[Found on Ads by Yahoo!]" like these paid results had a damn thing to do with what was actually relevant except that the advertiser targeted that particular keyword.

It's one thing when search engines put ads at the top of the results, as it's clearly identified a sponsored link, but when meta-search engines just combine the ads as matter of fact part of their results then the concept of meta-search has completely jumped the shark.

Sorry Dogpile, but you've fetched your last bone for me.

Slow Scrape Via AOL?

Something eluded to in a previous post, or maybe on someone else's site as it's all becoming one long blur, but it appears there is slow crawling running across an entire block of 256 IP addresses.

What seems to be happening is this scraping is coming from a couple of sources and one of them is someone using AOL as the IP resolved to an AOL proxy cache server. The implications are fairly disturbing in that blocking the scraper might also block a bunch of AOLers.

Many aren't aware that AOLers get issued a new IP address while surfing on the internet, typically around every 15 minutes, so if that person isn't using cookies they are quite difficult to track when the IP changes.

It could be multiple visitors but probably not due to the sequential nature of the pages being accessed and a couple of other factors that won't be mentioned so that the scrapers can't fake the behavior being targeted.

So now comes the ultimate question, as this activity is quite obviously scraping, to block or not to block, THAT is the question!

If a block is put in place it would obviously have to be some adaptive technology to analyze and temporarily block and only those IPs being used until the suspicious activity subsides.

Just what I need, to start out '06 chasing uber-scrapers.

Happy New Year scrapers, it might be your last.

Remember Me?

Don't you love all these web services that have the login with the checkbox "[ ] Remember Me On This Computer" under the password?

Then 30 minutes later you click something on the page and get "Session has timed out, click here to login again" or better yet, you go to their home page which has no clue who you are.

Did someone forget to click the checkbox on their website developer tool to "[ ] Implement Remember Me On This Site" or what?

This is further proof that the idiots of the world are taking over and the Future 2.0 is not going to be pretty.

War on Arrogant AdBlocking Assholes

Enough is enough already, this means WAR!

It's bad enough we have people disabling Javascript and running Norton Firewall blocking ads but now some Firefox extension lets you just right click on any ad and POOF! it's gone.

Once installed, it's a snap to filter elements at their source-address.
Just right-click: Adblock: done.

The only upside at this writing is Firefox, Norton Firewall and javascript disabling is all collectively under the 10% mark of all pages being displayed based on web log and advertisement impression comparisons on a couple of fairly popular web sites.

Why do they do this?

If you're tired of all the intrusive adverts that are increasingly taking over the Internet, Adblock is for you.

Well I got a better idea, if you're tired of the adverts that pay for the websites thus allowing people to read the content for free, then stay the fuck off those sites, maybe the entire internet or better yet, PAY the website directly to access their content!

That's right, you heard me you banner-blocking ad-stopping javascript-disabling assholes:

PAY TO PLAY or get the fuck off my bandwidth!

To counter-attack this ad blocking mentality let's try taking a page from the porn industry and create a new service like AdultCheck that would allow all the ad blockers access to all the member sites for an annual fee.

It's a perfect solution as the webmaster gets paid, the ad-blocking assholes get to see the content, and a new middleman web service gets rich off their ass.

Watch out blockers, your free ride is almost over.

Webmasters Protect Your Own Damn Content!

What the hell is wrong with all these cry baby writers and webmasters crying foul that Google should protect their content?

• Google didn't write your content, nor did MSN or Yahoo
• Google didn't steal your content, nor did MSN or Yahoo
• Google isn't a law enforcement agency, nor is MSN or Yahoo
• Protect your own shit and grow the fuck up!

Putting the blame on Blogger and AdSense is just pandering to the masses when all sorts of blogs not on Blogger steal content as well and use all sorts of ways to monetize their sites other than AdSense but of course if you aren't Google bashing then nobody gives a shit and won't read your flea bitten bullshit blogs.

Hell, many scrapers aren't even using blogs, they just scrape crap into some automatically generated pages on some free web server hosted in BFE (Bum Fuck Egypt) where your complaints to abuse@bfe.com will be ignored because your email isn't written in Swahili.

So what's a webmaster to do?

• File a DMCA (Digital Millennium Copyright Act) complaint to the ISP hosting the site and Google, Yahoo and MSN. Since the search engines are all in the US they have to comply with US law.
  
• Install some robot blocker/scraper stopper scripts (see link at page bottom)
  
• Fix your damn blog software to make scraping harder instead of putting a months worth of shit on one page which is imperceptible for scraper stopper scripts to catch someone downloading a page every now and then vs. many pages.
  

How do you file a DMCA complaint?

You should read the DMCA law first (pay special attention to the safe harbor provision) as it will help you get it right the first time as there are some specific things you must do, such as provide exact samples or the infringement, URLs, etc. and swear on a stack of religious reading material that you're the original author.

• File Google DMCA complaints
• File MSN DMCA complaints
• File Yahoo DMCA complaints - Not sure if this is the right place on Yahoo, messy web site
• Send a copy to the webmaster and ISP

Typically 1-2 days and POOF! they are gone.

They will now be offline, or at a minimum locked out of the 3 search engines which is what they use to drive traffic in the first place. If they are using AdWords (or something similar) for PPC traffic, you may have to take that up with Google's AdWords dept. or whoever they use.

Now, to make sure you are fully legal, take your lazy ass to the US Copyright website and spend about $30 getting a legal copyright so you can drag US-based thieving assholes into court and take advantage of what's called "statutory damages" which can hit 6 figures.

TIP: What I do on my primary web site is set a "usage" fee on the policies page, just like photographers do for images, and set the page content licensing fee at $2,000 per page. Whether this holds up in court is another matter but it sure works for photographers with stolen copyrighted images so I'm willing to give it a whirl when someone steals my shit.

High Speed Scrapers Steal More Than Pages

The common thinking about scrapers is they just steal your content and make money off your hard work. However, the damages can be worse and more immediate if they overload your web server and stop other traffic from accessing your site while they scrape. Some of them are so aggressive it's practically a DOS (denial of service) attack until they get what they want.

The problem for dynamic database driven websites (like mine that I'm trying to protect) is they tend to need more CPU resources than the normal garden variety static web sites so a high speed scraper, and sometimes even a regular search engine bot, can easily overload a server's CPU. This can quickly escalate to the point that the web server is queued up with so many page requests that it is unable to respond to new requests and appears "offline" for seconds, minutes or even hours depending on just how aggressive a spider gets with it's page requests.

Worse case, it may crash your server altogether under the strain.

The net result is that sites like mine that survive on advertising revenue, such as Google AdSense, suffer total income loss during temporary bot induced service outages. Visitors that would normally be clicking on ads are sitting there waiting for pages that will never display. Therefore, these bots do more direct damage to your pocket than just bandwidth wasting and stealing content for their own sites as the monetary losses can be quite immediate and left unchecked potentially devastating.

Prior to launching my new spider-trap/scraper blocker there were days where these scrapers had impacted the site income as much as 20%-30% in a single day. How this was possible is they were pummeling the server at night while nobody was watching and significant amounts of high income producing traffic coming from other time zones was lost.

This may be more of an issue to some webmasters than others but website scrapers, other than legitimate search engines, just need to be put out of business as they provide zero value and do nothing but steal.

Protect your site today with this nifty PHP scraper blocker Alex Kemp has written.

Many of you might find Alex's tool very useful to integrate into your web sites and stop copyright theft and income loss today.

The Great Anti-Scrape-Off

Previous posts have mumbled about my new spider-trap anti-scraper tool that I added to my web site and I must say it's working so well I'm contemplating converting it into PHP so the masses can play with it.

Don't hold your breath as I'm fundamentally lazy.

The features are as follows:

• Fast crawl auto-block
• Slow crawl detection and optional blocking
• Spider trap auto-block
• Webmaster control panel that shows last 15 minutes of live activity by visitor
  
• Manual ban/block from the control panel
• Allowed spider pass thru with built in passes for Google, Yahoo and MSN

Overall it has put the skids on 99.9% of the scrapers and off topic bots crawling my site within 2 days of being fully deployed. Previously I was just monitoring what it was doing, or going to do, in order to make sure it wasn't zapping legitimate spiders and visitors and now it seems to be ready for prime time so it was set to LIVE a couple of days ago with fingers crossed.

New enhancements I'm working on for next week aren't spider related but visitor related. These new 'sensors' will detect additional information per visitor such as cookies, javascript and banner blocking being used so I can remove old hacks that do some of this and make a centralized visitor knowledge base.

The final features in the visitor knowledge base will allow me to dynamically deploy the appropriate advertising model that each visitor can view when the second page is loaded, or theoretically allow me to start interjecting intermission pages to become a "subscriber".

So far so good, will keep everyone posted as to the effectiveness of the anti-scraper tools.

Voyager the Cosmix Crawler

Found another super slow web crawler slowly downloading pages for Cosmix's Kosmix health search which was completely off topic for my site. It was only moving at a snails pace looking for pages with medical information but being that far off topic I decided to block them just on principle alone.

What all these crawlers and scrapers do is inflate my page views and for my direct advertisers which are embedded in every page, inflating their page impressions and make the click thru rates for the ads look worse than they really are.

Sorry unwanted spiders, take a hike.

Sensis Web Crawler

Maybe I'm leading a sheltered life but I've never heard of the Australian based Sensis Web Crawler before this week when I noticed it slow crawling my biggest web site over a period of days now. Just before I almost blocked this bot I noticed that it seemed to be a legit service and Alexa claims is reasonably popular in their top 10,000 sites so I let it crawl.

Probably one of the best behaved but persistent little bots I've seen in a while.

Sensis has only crawled about 1300 pages in 24 hours which is about 54 pages an hour so Sensis will probably be on my site 30 days depending on how deeply they index based on their current rate of crawl.

This is much less abusive than Yahoo, MSN or Google that want whatever they want whenever they want and sometimes as fast as they can get it just to be competitive with each other.

Maybe the Big 3 can take a clue from this little player and be a bit more web server and bandwidth friendly. Then again, maybe Sensis just doesn't have the bandwidth and computing horsepower to beat the crap out of my server like the Big 3.

Good luck Sensis, will keep an eye on my results down under.

Google Peeks on Purpose

Google must peek at pages they are expressly told not to index in robots.txt on purpose just to see what in the heck is being hidden. There is no technical reason imaginable that they can't read the same robots.txt everyone else is reading as neither Yahoo, MSN or Teoma have ever crawled the pages marked off limits yet Google just can't seem to control themselves and keep their damned bots off those pages.

So which is it Google?

• Everything at Google is still in BETA so what do you expect
  
• Our engineering dept. just can't get all the bugs out, get over it
  
• We peek regardless because we're Google and we can

I'd really like to know which it is as the competition doesn't seem to break those rules.

Google's Blog Search Doesn't

Noticed some poor hapless bastard looking for a little holiday cheer came here from Google for the search phrase "good will toward men" but noticed that it was from the main Google index and not the Google Blog Search.

Upon closer examination this site is definitely indexed by the blog search but oddly enough you can't find that quoted phrase for this site in the 700 returned results but it's about in the top 20 in the main Google index unquoted, about #10 quoted.

No wonder I'm losing my hair as Google keeps making me scratch my head all the time wondering what in the heck is going on over there but the blog search does say BETA so we'll give them the benefit of the doubt for the moment.

UPDATE:
Someone must've complained to Google as my "good will toward men" rankings got flushed.

AdSense Makes Americans Fat

Time for a quick game of twisted logic that's about as scientific as some of the stupid surveys and studies you'll read but more amusing in the end.

Reading this piece about The New American: Fat, weak and slow caused me to attribute this to Americans sitting in chairs all day glued to the internet and video games. which would ironically make the article about Americans being fat and weak itself contributing to the problem it claims to expose as it's also on the internet but we're sidetracking.

So let's draw some conclusions from my hypothesis that everyone's increasing tendency to be sedentary is caused by the internet.

Many webmasters are creating tons of content to get people to visit their monetized sites more often in order to make them more money, thus contributing to these sedentary lifestyles.

Therefore, since most webmasters monetize their websites with AdSense, we can conclude that ultimately it's AdSense that makes you fat.

How can this be?

Because AdSense drives the money grubbing webmasters to write more and more content that your addicted asses just can't stop reading all day on the internet which is why your ass has now spread to consume the complete lateral surface area of your chair cushion and perhaps overflow it somewhat.

Notice no AdSense on this blog?

That's right!

IncrediBILL's Random Rants are DIETETIC because we care!

AdSense Vigilanteism

Here's a neat trick if you want to help maintain the integrity of the AdSense network when you run across some ignorant blogger or some other idiotic web site with calls to "click the ads" on their web page.

Posting the following snippet below from the AdSense Policies page into their online comments has resulted in the "click the ads" requests being yanked sometimes instantly when the webmaster shit their pants thinking big brother must've done it.

Try it, just cut and past the following in someone's page as comments:

Google AdSense Program Policies

https://www.google.com/adsense/policies

Incentives

Web pages may not include incentives of any kind for users to click on ads. This includes encouraging users to click on the ads or to visit the advertisers' sites as well as drawing any undue attention to the ads. For example, your site cannot contain phrases such as "click here," "support us," "visit these links," or other similar language that could apply to any ad, regardless of content. These activities are strictly prohibited in order to avoid potential inflation of advertiser costs. In addition, publishers may not bring unnatural attention to sites displaying ads or referral buttons through unsolicited mass emails or unwanted advertisements on third-party websites. Publishers are also not permitted to use deceptive or unnatural means to draw attention to or incite clicks on referral buttons.

Funniest amusing shit ever and it's FREE!

Not to mention you may be saving your own source of income by stemming the tide of advertiser withdrawl of the AdSense content network so it's a win-win.

Firefox Prefetch is Crap

Now that I've been monitoring what good old Firefox is prefetching since first watching it step right into my new spider trap it seems to have one more disturbing trend in that it tries to prefetch the same pages over and over and over.

What seems to be happening is that every page a visitor goes to the Firefox prefetch appears to keep asking the web server about the same navigation pages repeatedly.

Well that's just fucking stupid!

Oh well, what do you expect for free.