Stop DNS Site Hijacking With Apache Redirect

There have been several instances of DNS hijacking reported lately so I thought it might be useful to briefly address this problem and it's easy resolution.

DNS hijacking occurs when someone using a different domain name aims their domain name at your server and it hijacks a domain on your server. Both domains, the hijackers and yours, now serve up the same exact content. The search engines aren't sure which one to believe so the hijackers domain, if it gets crawled, may suddenly inherit all your site pages and your domain goes away completely in the search engine index.

This has actually happened a couple of times to various people I know or I wouldn't bother sounding the alarm that this could also happen to you. Some cases appear to be accidental leftovers from people using 3rd party DNS services that still point to servers they no longer use. Others appear to possibly be deliberate, trying to hijack someone's site in Google or other search engines, and sometimes it works too!

If all your sites are using virtual hosting and share a single IP then it's a crap shoot which domain they'll hijack.

However, if you have a dedicated IP for a site that uses a dedicated SSL server then they can aim their DNS entry directly at your site and potentially take it for a search engine joy ride.

The easiest way to stop this is to add the following lines into your .htaccess file on every site in your server and replace example.com with your own domain name.

RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]

This also canonicalizes your domain to the non-www form of the domain name.

If you want your canonical domain to start with www. then you'll need to add that to the script.

Remember, an ounce of prevention is worth a pound of cure.

Updating Apache Zaps Plesk 8 Suexec

Boy did I hit the panic button tonight when I did a server software update and suddenly everything running in a CGI-BIN folder started generating 500 errors.

I remembered this problem with earlier versions of the Plesk control panel I use, but it seems that back then it was a simple CHMOD problem, took a look and the SUEXEC group, owner and flags were all still proper.

Lots of stuff down, clock ticking, trying this and that, mild panic setting in...

Turns out the stock Apache SUEXEC isn't compatible with Plesk 8 whatsoever which is why everything was coughing up 500 errors.

Luckily for me there is a second copy of it laying around in a Plesk folder so a quick copy and everything is back running just fine.

cp /usr/local/psa/suexec/psa-suexec /usr/sbin/suexec

Most of you won't have a clue what any of this Linux gibberish is about so don't worry about it but if this happens to some other nerd using Plesk out there perhaps in his moment of nerd panic he'll find my post and know how to solve the problem quickly.

Bankrupt Silicon Valley Crybabies - Try Living Within Your Means

Reading this post on WebGuild about "Down and Out in Silicon Valley" really pisses me off.

The author is trying to make me feel bad about someone making $12K/month ($144K/year) that's now grabbing grub at the food pantry and using food stamps.

Give me a fucking break, why the hell did you spend it all?

What part of "bank" didn't you understand?

Back in the 90's someone I used to work with was a multi-millionaire just from his stock options in one company.

What did he do?

You probably already guessed it, he bought a million dollar house, hundred thousand dollar cars, all of it on credit of course and kept the money in the stock market.

Ujita just arranged for a new food pantry in Los Gatos where the median income is $175,000 and where many home owners have been wiped out. Their mortgages have overtaken the value of their homes and a layoff quickly depletes theirs saving and soon they find themselves in soup kitchens.

Classic mistake, when you can afford to pay cash, pay cash and keep your ass debt free within reason and have some of those things called assets, which you can sell when the shit hits the fan and you need money.

Well, of course my friend lost it all when the dot-com bubble burst and was sitting there with massive payments needing to be made and millions gone in a flash, POOF!

So ask yourself, if you can't afford to pay cash for a new Cadillac Escalade but you could pay cash for a Buick, maybe you stick to a Buick budget, or perhaps get a used car a couple of years old at nearly half the price.

People are out of their fucking minds trying to impress people with bigger better faster newer cooler and then when it hits the fan expect everyone to feel sorry for their plight, loan them money, cut them slack, bummer dude.

For instance when I worked in the corporate world of Silicon Valley almost everyone on my team was driving Mercedes and Porches while I was driving a Ford or a Buick and pressuring me to get with the program and get a car that went with the status of the job.

Guess what, that's $50K price difference between what I paid for my car vs. theirs and my cash went into the bank while their cash depreciated every time they drove that car.

I'm not jealous, if driving that $100K car makes you think your dick is bigger or harder than mine, then enjoy your big fucking expensive car but don't expect sympathy from me when the economy blows a gasket and you have no money because you spent it all.

Wah fucking wah.

I'll admit I blew a wad on a 50" plasma TV but of course I could afford it just off the interest earned from the money I saved not buying the overpriced German car.

If you can't afford it, don't buy it and the next time the economy takes a dump you'll survive with something called "savings", one of those magical things banks help you accomplish if you don't fucking spend it all trying to impress your goddamn friends and family.

Nobody will be impressed when you're homeless.

I can't blog anymore!

My blogging bone appears to be broken.

Got a few good rants backed up, will see what I can do about those.

Until then, read this post about Rich Snippets and Microformats

Credit Card Fraud on Unused Card!

Here's one for the record books.

I just got a wake up call at 8am from the credit card fraud dept. at a major bank, who's name will go unmentioned, but you can purchase their stock for less than the price of a Happy Meal.

Anyway, one of my credit cards that's been quietly and securely sitting in my desk drawer and hasn't been used in over 5 years is out making ridiculously small purchases. If this happened to any of my other cards I would probably gloss over such a small charge until they zapped me for a much bigger item, but the fraud dept. noticed this sudden use of an idle card.

It would appear someone managed to steal an ancient list of credit cards and is merely guessing at the new expiration date!

They certainly didn't get this from my trash as all credit card data is shredded, and I mean minuscule confetti kind of shredding, so unless you have infinite time on your hands and a lot of tape and glue, you're not getting anything from me freely.

So there you have it, my fun Friday wake up call from the friendly credit card dept.

Hope this day gets better as it goes along.

Florida Drinks Cat Piss Picture Book

A couple of years ago I was faced with the horrors of the cat piss they try to pass off for beer in Florida. Last week I was back in Florida and found myself face to face with the same situation, a state full of cat piss beer, but I was determined to find something drink worthy.

A few places had something called Yuengling which by it's very name sent shivers down my spine sounding like some rice beer shit from Japan.

I decided to give it a go, and here's how it went...

Challenge: IncrediBill vs. Yuengling Ale

How could anything called Yuengling taste good?

This shit better be fucking good!

I'm pleasantly surprised.

Iterasi Archives Sites Without Permission

Guess what boys and girls?

There's another wonderful new site that allows people to copy your shit without your permission!

Iterasi allows their members to "archive" individual web pages.

The pages on my site have a meta tag "NOARCHIVE" which tells everyone DO NOT ARCHIVE this page yet they archived it anyway. They also stripped out my frame busting javascript so they are seriously thwarting sites at every turn that don't want to participate in their tool.

Being that Iterasi is in Beta maybe I'll cut them a little slack, very little, but just a bit.

On their web site it says:

At iterasi, we love the Web. So much so, that we want to keep it. Forever.

If you really love the web you would follow standard web protocols and if the webmaster gives you permission, fine, do whatever you want.

For those of us that don't allow it, back the fuck off.

Here's the IP and user agent details:

198.145.117.78

"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; .NET CLR 2.0.50727)"

They operate out of this IP range:

OrgName: Infinity Internet, Inc.
NetRange: 198.145.0.0 - 198.145.255.255
CIDR: 198.145.0.0/16

Infinity Internet is a mixed service with both hosting and business/residential DSL services so blocking the whole range probably isn't safe.

The reverse DNS shows:

pointer ip78.117.colo.iinet.com.

For the time being, you can opt-out of Iterasi by blocking anything with an RDNS containing ".colo.iinet.com" which seems to stop them dead in their archiving tracks.

Here's a few things Iterasi could do so webmasters don't get hostile:

• Honor robots.txt
• Honor meta tags like NOARCHIVE
  
• Provide a user agent string that identifies Iterasi accessing a site
  
• Provide reverse DNS so we can tell it's your company and not a spoof
  

Until that time, I have you permanently blocked and I'm sure others will soon .

MSNBOT Crawled Thru Javascript!

Today I caught MSNBOT-MEDIA crawling thousands of links that were only accessible thru javascript.

These links were only intended for human use, only accessible via javascript, therefore never added to robots.txt...

... until today

Here's the MSNBOT specifics used for this crawl:

65.55.235.202 "GET /feedback.html?id=1010101234 HTTP/1.0"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"

I have a page used for site feedback for various page elements and each link on the page has an OnClick command like this:

a href="#" OnClick="OpenFeedback(1010101234)

Elsewhere in the code is the actual function:

function OpenFeedback(id) {
window.open('feedback.html?id=' + id,.....')
}

MSNBOT appears to have assembled it together and was crawling thousands of links such as "/feedback.html?id=1010101234" and so on, page after page.

I have no clue if this was a handjob done just for the site in question or some new pet project testing their ability to crawl javascript, but the game has definitely changed.

To put it bluntly, javascript itself is no longer sufficient to curtail crawlers on the web, at least not simple javascript.

Design With Screen Shots in Mind

With the proliferation of screen shots everywhere you would think that site designers would make sure that their sites make good screen shots, right?

Unfortunately this is not always the case.

When a surfer uses a visual search engine or a directory that has screen shots of the site the visual appeal often dictates which site is chosen and which site is left behind, not the SEO value of the text that got the site there in the first place.

Download a copy of WebShot and see how your site looks as a thumbnail.

If the resulting thumbnail doesn't convey an eye catching concept of the site you've failed.

More importantly, if the thumbnail comes up as a solid color because your Flash file is too slow to load and play in 30 seconds or for some other technical reasons, you've failed even worse.

One of my sites has almost 40K screen shots online and trust me when I tell you that the crap screen shots aren't getting the lion's share of the clicks, people just assume it's broken or something and click elsewhere.

Hope this helps a few of you revise how you think of your home pages.

How to Block SpyderMate SEO Tool

Trapped another SEO tool called SpyderMate that crawls your site analyzing data.

Nothing wrong with using it but don't let competitors get a free ride analyzing your site.

The spider details:

208.78.98.236 [208-78-98-236.slicehost.net.]
"MentorMate Spider"

Their host:

OrgName: Slicehost LLC
OrgID: SLICE
NetRange: 208.78.96.0 - 208.78.103.255
CIDR: 208.78.96.0/21

They just keep coming and eventually they'll run out of data centers we haven't blocked!

Top 20 Forum Spammers for 2008

So far this year I've been tracking 9,694 unique IPs attempting to spam 49,492 links on my sites and they all bounced into a spam tracking log file.

Using this log file I can track where all the crap is coming from and (not so) surprisingly it's primarily coming from Europe.

The #1 spam host is blueconnex.net which wins the prize with 4 IPs in the top 20 and their #1 spammer at 7,798 posts is still trying today!

The runner up is giga-hosting.biz which has a few prolific IPs and took the #2 and #10 positions.

7798 : 92.48.122.3
2327 : 193.37.152.242
2174 : 94.100.29.250
2156 : 94.102.60.11
1889 : 85.255.120.210
1603 : 85.255.120.74
1551 : 195.190.13.242
1541 : 78.129.202.15
1218 : 92.48.122.2
1085 : 193.34.144.72
1008 : 77.92.88.13
1003 : 77.92.88.27
599 : 94.102.49.34
560 : 193.34.144.83
459 : 92.48.127.97
437 : 92.48.127.96
434 : 77.92.88.5
413 : 85.12.25.66
379 : 77.92.88.6
194 : 85.255.118.50

Not a single spam got thru but they just keep trying because they aren't that fucking smart.

Temporarily Block HotLinking To Find Copyright Abusers

Blocking hotlinks is usually considered a method used to conserve bandwidth and stop leeching of images off your server. However, you can also use hotlink blocking to quickly and easily find all those sites using your content.

The most common solution for Linux servers is to add the following hotlink blocking code into your .htaccess file.

RewriteEngine On
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?yourserver.com [NC]
RewriteRule \.(jpeg|jpg|gif|png)$ - [F]

Obviously you want to change yourserver.com to the domain name of your site before adding this to .htaccess on your site.

Now once you've added this code the fun begins as you sit back a few hours and wait for all the "403 forbidden" codes to start filling up your access log file.

Now using a simple grep on your log file will generate a nice list of sites in the referrer field that are hotlinking your images, or much worse which is often the case.

grep "\.jpg" access_log | grep " 403 "

grep "\.gif" access_log | grep " 403 "

The first part of the grep locates all ".jpg" files then the second part filters out all but the " 403 " forbidden errors.

After a day or 2 you'll have a nice list of sites to send C&D's, DMCAs, and all sorts of fun stuff.

Now disable your hotlink blocking script or remove it from your .htaccess file.

Why disable hotlink blocking?

Because hotlink blocking encourages people to actually download your images making the process of finding stolen images way more difficult. Therefore, a temporary hotlink block shows you everyone doing this just long enough to take corrective measures, then let your site wide open again and wait for the next batch of idiots to start hotlinking.

Hope a few of you find this little tip handy!

Pubcon '08 and Other Announcements

I'll be presenting at PubCon '08 on the topic of Competitive Intelligence. The only difference is the other panelists will be discussing how to find competitive intelligence while I'm telling people how to protect themselves from such research.

Also, keep your eye on this space:

http://twitter.com/CrawlWall

All shortly upcoming announcements will be made via Twitter and there's a bunch coming up soon.

It's what you've been waiting for...

JadynAve Bot Wants Your Local Data

If you have a bunch of local data like I do then you better protect it because JadynAve's Local Business Search appears to be coming after your site with their JadynAveBot!

Didn't ask for robots.txt, has no data whatsoever on their robot page except to email them if you have any questions, big whoop.

Here's the IP and user agent:

38.99.186.40
"Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot"

I wouldn't bother trying to add them in robots.txt since they didn't ask for robots.txt.

This is a job for .htaccess!

A little research revealed they have also crawled without the "bot" in their user agent so you'll just want to block anything with "jadynave" in it.

Suspected Copyright Offenses

Something amusing hit my site from from .t-dialin.net which appears to be .t-online.de or the German version of T-Mobile.

I see the following IP and user agent:

84.153.98.95 "Verdacht Vergehen nach UrhG"

Which Google translates into:

Suspected offenses under the Copyright Act

Well isn't that just the cutest little user agent to get caught in a bot blocker?

Now I've had my chuckle for the evening, back to work...

Why Does Copyscape/GoogleAlert Hide?

Never really played around with Copyscape/GoogleAlert much but I noticed it tries to completely hide it's presence when accessing a server which isn't cool.

Not that I'm a fan of plagiarism as my copy of the DMCA is almost worn out from use, but I'm even a less fan of sneaky web crawlers that pretend to be shit they aren't.

The IP that Copyscape uses:

212.100.254.105 -> www.googlealert.com

The Copyscape user agent:

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

This is located in a Rackspace so if you're already blocking Rackspace then you probably won't be bothered with Copyscape in the first place:

inetnum: 212.100.254.64 - 212.100.254.127
descr: Rackspace Managed Hosting

Of course you might not want to block this if you actually use Copyscape as it will become quite useless.

Viewzi's Meta Search Engine Taking Screenshots Without Permission

Here we go again with yet another visual search engine called Viewzi taking a bunch of screen shots without asking for permission from robots.txt.

In this case it's a meta search engine and technically the search engines Viewzi culls from has been given permission to crawl, but Viewzi itself was never given access permission.

Here's the Viewzi user agent:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b4pre) Gecko/2008022910 Viewzi/0.1

They appear to have just replaced the word Firefox with their user agent name Viewzi instead of just adding Viewzi to the agent which is kind of crappy to not even give Firefox attribution for their code being used to make screen shots.

Viewzi currently crawls from the "compute-1.amazonaws.com" range of IPs so if you're already blocking amazonaws.com then you've blocked Viewzi already.

Sorry, but you won't get any Viewzi of my sites until you learn to play nice.

Google Analytics Finds Bandits and Proxies!

Google Analytics has a Hostnames feature that most overlook which normally displays the hostname of the site your visitor landed on, like example.com. However, you'll probably notice a bunch of IP addresses and other interesting information in this list including sites that may have stolen your content!

To see what I'm referring to go into your Google Analytics account and go to Visitors -> Network Properties -> Hostnames.

Many of the IPs listed will be for Google or Yahoo translator services or such and you wouldn't want to block any of these. Other IPs and host names will be proxy servers in data centers you probably never heard of and possibly host names to places that have your stolen content posted!

Now expand the date range for your report to show all your Hostname data as far back as Google has been tracking your site and see what people have been doing with your site all this time.

Probably not worth trying to just block old single proxy IPs as proxy sites come and go all the time, but most likely you'll find these IPs are associated with data centers which host lots of servers and perhaps that proxy is just on a new IP so now you have another data center you can block.

Fun fun fun!

The list of actual host domain names, not the IPs, is what I found most useful as a few of those turned out to be idiots that managed to scrape a page or two from my site and still had my Google Analytics tracking codes on their pages!

Enjoy this new toy while I start sending C&Ds to the idiots with my tracking codes still on their sites.

Possibly Slowest Scraper Ever

I've seen slow scrapers before but this is fucking ridiculous.

80.197.39.182 has been automatically challenged to answer now 227 times since 09/23/2008 and it just keeps plugging along slow enough to be off the radar of most webmasters but just fast enough it keeps nudging my bot blocker once a day to keep tracking it.

The user agent claims to be Firefox 3:

Mozilla/5.0 (Windows; U; Windows NT 5.1; da; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

Which could indicate someone making screen shots.

No clue what they're really doing but it's going really slow and they're getting pages of garbage instead of what they want, so I hope they're having a real good time fucking with my bot blocker.

SEOMoz's New Linkscape Creates Webmaster Backlash

10/06/08 - A day that will live in internet infamy when a prominent internet company caught millions of webmasters off guard and sent shockwaves around cyberspace.

The event that caused this uproar was the launch of Linkscape with supposedly 30 billion pages indexed that stunned even the most savvy webmasters because they didn't see it crawling and were totally taken by surprise.

This new snooping SEO tool is billed as "An Index of the World Wide Web – 30 billion pages (and growing!), refreshed monthly" which has left webmasters that are already battered and abused by a massive onslaught of automated bots more angry than ever.

The internet entitlement mentality thinks that all webmasters have unlimited bandwidth and CPU and that anything that's online should just be taken without regard to the consequences.

Webmasters will no longer tolerate Indexation Without Representation and are moving to regain control over their sites, their content, and their competitive intelligence. Many webmasters that previously called bot blockers paranoid draconian control freaks are now crying for solutions to high profile marauders raiding their sites and reaping large profits. Now that the tide has turned the webmasters are preparing for the revolution with new sites such as the NoArchive Initiative, better bot blocking scripts, honey pots and much more.

Even a competitive site called MajesticSEO which provides a similar product actually gives Free multiple page reports on your domains if you register and prove you own the site which is at least a symbiotic relationship and not completely parasitic.

However, not only doesn't Linkscape give anything back to the webmaster for allowing your site to be crawled, or mined for competitive intelligence, they actually increased the price $30 to access their tools so you actually have to pay more for the privilege of being crawled to see your own data!

New Pricing, featuring three levels of PRO membership depending on the size and needs of our members. Current PRO members need not worry - you'll be grandfathered in at the current price level. We're just creating two new echelons for those who need access to more. If you'd like to lock in at the current price level ($49/month), I won't stop you :-)

So what benefit do we all get from all this?

For a monthly fee our competitors can see everything we do in infinite detail.

Wow! That's a benefit?

Sorry, but that just doesn't play homey.