I can't blog anymore!

My blogging bone appears to be broken.

Got a few good rants backed up, will see what I can do about those.

Until then, read this post about Rich Snippets and Microformats

Credit Card Fraud on Unused Card!

Here's one for the record books.

I just got a wake up call at 8am from the credit card fraud dept. at a major bank, who's name will go unmentioned, but you can purchase their stock for less than the price of a Happy Meal.

Anyway, one of my credit cards that's been quietly and securely sitting in my desk drawer and hasn't been used in over 5 years is out making ridiculously small purchases. If this happened to any of my other cards I would probably gloss over such a small charge until they zapped me for a much bigger item, but the fraud dept. noticed this sudden use of an idle card.

It would appear someone managed to steal an ancient list of credit cards and is merely guessing at the new expiration date!

They certainly didn't get this from my trash as all credit card data is shredded, and I mean minuscule confetti kind of shredding, so unless you have infinite time on your hands and a lot of tape and glue, you're not getting anything from me freely.

So there you have it, my fun Friday wake up call from the friendly credit card dept.

Hope this day gets better as it goes along.

Florida Drinks Cat Piss Picture Book

A couple of years ago I was faced with the horrors of the cat piss they try to pass off for beer in Florida. Last week I was back in Florida and found myself face to face with the same situation, a state full of cat piss beer, but I was determined to find something drink worthy.

A few places had something called Yuengling which by it's very name sent shivers down my spine sounding like some rice beer shit from Japan.

I decided to give it a go, and here's how it went...

Challenge: IncrediBill vs. Yuengling Ale

How could anything called Yuengling taste good?

This shit better be fucking good!

I'm pleasantly surprised.

Iterasi Archives Sites Without Permission

Guess what boys and girls?

There's another wonderful new site that allows people to copy your shit without your permission!

Iterasi allows their members to "archive" individual web pages.

The pages on my site have a meta tag "NOARCHIVE" which tells everyone DO NOT ARCHIVE this page yet they archived it anyway. They also stripped out my frame busting javascript so they are seriously thwarting sites at every turn that don't want to participate in their tool.

Being that Iterasi is in Beta maybe I'll cut them a little slack, very little, but just a bit.

On their web site it says:

At iterasi, we love the Web. So much so, that we want to keep it. Forever.

If you really love the web you would follow standard web protocols and if the webmaster gives you permission, fine, do whatever you want.

For those of us that don't allow it, back the fuck off.

Here's the IP and user agent details:

198.145.117.78

"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; .NET CLR 2.0.50727)"

They operate out of this IP range:

OrgName: Infinity Internet, Inc.
NetRange: 198.145.0.0 - 198.145.255.255
CIDR: 198.145.0.0/16

Infinity Internet is a mixed service with both hosting and business/residential DSL services so blocking the whole range probably isn't safe.

The reverse DNS shows:

pointer ip78.117.colo.iinet.com.

For the time being, you can opt-out of Iterasi by blocking anything with an RDNS containing ".colo.iinet.com" which seems to stop them dead in their archiving tracks.

Here's a few things Iterasi could do so webmasters don't get hostile:

• Honor robots.txt
• Honor meta tags like NOARCHIVE
  
• Provide a user agent string that identifies Iterasi accessing a site
  
• Provide reverse DNS so we can tell it's your company and not a spoof
  

Until that time, I have you permanently blocked and I'm sure others will soon .

MSNBOT Crawled Thru Javascript!

Today I caught MSNBOT-MEDIA crawling thousands of links that were only accessible thru javascript.

These links were only intended for human use, only accessible via javascript, therefore never added to robots.txt...

... until today

Here's the MSNBOT specifics used for this crawl:

65.55.235.202 "GET /feedback.html?id=1010101234 HTTP/1.0"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"

I have a page used for site feedback for various page elements and each link on the page has an OnClick command like this:

a href="#" OnClick="OpenFeedback(1010101234)

Elsewhere in the code is the actual function:

function OpenFeedback(id) {
window.open('feedback.html?id=' + id,.....')
}

MSNBOT appears to have assembled it together and was crawling thousands of links such as "/feedback.html?id=1010101234" and so on, page after page.

I have no clue if this was a handjob done just for the site in question or some new pet project testing their ability to crawl javascript, but the game has definitely changed.

To put it bluntly, javascript itself is no longer sufficient to curtail crawlers on the web, at least not simple javascript.

Design With Screen Shots in Mind

With the proliferation of screen shots everywhere you would think that site designers would make sure that their sites make good screen shots, right?

Unfortunately this is not always the case.

When a surfer uses a visual search engine or a directory that has screen shots of the site the visual appeal often dictates which site is chosen and which site is left behind, not the SEO value of the text that got the site there in the first place.

Download a copy of WebShot and see how your site looks as a thumbnail.

If the resulting thumbnail doesn't convey an eye catching concept of the site you've failed.

More importantly, if the thumbnail comes up as a solid color because your Flash file is too slow to load and play in 30 seconds or for some other technical reasons, you've failed even worse.

One of my sites has almost 40K screen shots online and trust me when I tell you that the crap screen shots aren't getting the lion's share of the clicks, people just assume it's broken or something and click elsewhere.

Hope this helps a few of you revise how you think of your home pages.

How to Block SpyderMate SEO Tool

Trapped another SEO tool called SpyderMate that crawls your site analyzing data.

Nothing wrong with using it but don't let competitors get a free ride analyzing your site.

The spider details:

208.78.98.236 [208-78-98-236.slicehost.net.]
"MentorMate Spider"

Their host:

OrgName: Slicehost LLC
OrgID: SLICE
NetRange: 208.78.96.0 - 208.78.103.255
CIDR: 208.78.96.0/21

They just keep coming and eventually they'll run out of data centers we haven't blocked!

Top 20 Forum Spammers for 2008

So far this year I've been tracking 9,694 unique IPs attempting to spam 49,492 links on my sites and they all bounced into a spam tracking log file.

Using this log file I can track where all the crap is coming from and (not so) surprisingly it's primarily coming from Europe.

The #1 spam host is blueconnex.net which wins the prize with 4 IPs in the top 20 and their #1 spammer at 7,798 posts is still trying today!

The runner up is giga-hosting.biz which has a few prolific IPs and took the #2 and #10 positions.

7798 : 92.48.122.3
2327 : 193.37.152.242
2174 : 94.100.29.250
2156 : 94.102.60.11
1889 : 85.255.120.210
1603 : 85.255.120.74
1551 : 195.190.13.242
1541 : 78.129.202.15
1218 : 92.48.122.2
1085 : 193.34.144.72
1008 : 77.92.88.13
1003 : 77.92.88.27
599 : 94.102.49.34
560 : 193.34.144.83
459 : 92.48.127.97
437 : 92.48.127.96
434 : 77.92.88.5
413 : 85.12.25.66
379 : 77.92.88.6
194 : 85.255.118.50

Not a single spam got thru but they just keep trying because they aren't that fucking smart.

Temporarily Block HotLinking To Find Copyright Abusers

Blocking hotlinks is usually considered a method used to conserve bandwidth and stop leeching of images off your server. However, you can also use hotlink blocking to quickly and easily find all those sites using your content.

The most common solution for Linux servers is to add the following hotlink blocking code into your .htaccess file.

RewriteEngine On
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?yourserver.com [NC]
RewriteRule \.(jpeg|jpg|gif|png)$ - [F]

Obviously you want to change yourserver.com to the domain name of your site before adding this to .htaccess on your site.

Now once you've added this code the fun begins as you sit back a few hours and wait for all the "403 forbidden" codes to start filling up your access log file.

Now using a simple grep on your log file will generate a nice list of sites in the referrer field that are hotlinking your images, or much worse which is often the case.

grep "\.jpg" access_log | grep " 403 "

grep "\.gif" access_log | grep " 403 "

The first part of the grep locates all ".jpg" files then the second part filters out all but the " 403 " forbidden errors.

After a day or 2 you'll have a nice list of sites to send C&D's, DMCAs, and all sorts of fun stuff.

Now disable your hotlink blocking script or remove it from your .htaccess file.

Why disable hotlink blocking?

Because hotlink blocking encourages people to actually download your images making the process of finding stolen images way more difficult. Therefore, a temporary hotlink block shows you everyone doing this just long enough to take corrective measures, then let your site wide open again and wait for the next batch of idiots to start hotlinking.

Hope a few of you find this little tip handy!

Pubcon '08 and Other Announcements

I'll be presenting at PubCon '08 on the topic of Competitive Intelligence. The only difference is the other panelists will be discussing how to find competitive intelligence while I'm telling people how to protect themselves from such research.

Also, keep your eye on this space:

http://twitter.com/CrawlWall

All shortly upcoming announcements will be made via Twitter and there's a bunch coming up soon.

It's what you've been waiting for...

JadynAve Bot Wants Your Local Data

If you have a bunch of local data like I do then you better protect it because JadynAve's Local Business Search appears to be coming after your site with their JadynAveBot!

Didn't ask for robots.txt, has no data whatsoever on their robot page except to email them if you have any questions, big whoop.

Here's the IP and user agent:

38.99.186.40
"Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot"

I wouldn't bother trying to add them in robots.txt since they didn't ask for robots.txt.

This is a job for .htaccess!

A little research revealed they have also crawled without the "bot" in their user agent so you'll just want to block anything with "jadynave" in it.

Suspected Copyright Offenses

Something amusing hit my site from from .t-dialin.net which appears to be .t-online.de or the German version of T-Mobile.

I see the following IP and user agent:

84.153.98.95 "Verdacht Vergehen nach UrhG"

Which Google translates into:

Suspected offenses under the Copyright Act

Well isn't that just the cutest little user agent to get caught in a bot blocker?

Now I've had my chuckle for the evening, back to work...

Why Does Copyscape/GoogleAlert Hide?

Never really played around with Copyscape/GoogleAlert much but I noticed it tries to completely hide it's presence when accessing a server which isn't cool.

Not that I'm a fan of plagiarism as my copy of the DMCA is almost worn out from use, but I'm even a less fan of sneaky web crawlers that pretend to be shit they aren't.

The IP that Copyscape uses:

212.100.254.105 -> www.googlealert.com

The Copyscape user agent:

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

This is located in a Rackspace so if you're already blocking Rackspace then you probably won't be bothered with Copyscape in the first place:

inetnum: 212.100.254.64 - 212.100.254.127
descr: Rackspace Managed Hosting

Of course you might not want to block this if you actually use Copyscape as it will become quite useless.

Viewzi's Meta Search Engine Taking Screenshots Without Permission

Here we go again with yet another visual search engine called Viewzi taking a bunch of screen shots without asking for permission from robots.txt.

In this case it's a meta search engine and technically the search engines Viewzi culls from has been given permission to crawl, but Viewzi itself was never given access permission.

Here's the Viewzi user agent:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b4pre) Gecko/2008022910 Viewzi/0.1

They appear to have just replaced the word Firefox with their user agent name Viewzi instead of just adding Viewzi to the agent which is kind of crappy to not even give Firefox attribution for their code being used to make screen shots.

Viewzi currently crawls from the "compute-1.amazonaws.com" range of IPs so if you're already blocking amazonaws.com then you've blocked Viewzi already.

Sorry, but you won't get any Viewzi of my sites until you learn to play nice.

Google Analytics Finds Bandits and Proxies!

Google Analytics has a Hostnames feature that most overlook which normally displays the hostname of the site your visitor landed on, like example.com. However, you'll probably notice a bunch of IP addresses and other interesting information in this list including sites that may have stolen your content!

To see what I'm referring to go into your Google Analytics account and go to Visitors -> Network Properties -> Hostnames.

Many of the IPs listed will be for Google or Yahoo translator services or such and you wouldn't want to block any of these. Other IPs and host names will be proxy servers in data centers you probably never heard of and possibly host names to places that have your stolen content posted!

Now expand the date range for your report to show all your Hostname data as far back as Google has been tracking your site and see what people have been doing with your site all this time.

Probably not worth trying to just block old single proxy IPs as proxy sites come and go all the time, but most likely you'll find these IPs are associated with data centers which host lots of servers and perhaps that proxy is just on a new IP so now you have another data center you can block.

Fun fun fun!

The list of actual host domain names, not the IPs, is what I found most useful as a few of those turned out to be idiots that managed to scrape a page or two from my site and still had my Google Analytics tracking codes on their pages!

Enjoy this new toy while I start sending C&Ds to the idiots with my tracking codes still on their sites.

Possibly Slowest Scraper Ever

I've seen slow scrapers before but this is fucking ridiculous.

80.197.39.182 has been automatically challenged to answer now 227 times since 09/23/2008 and it just keeps plugging along slow enough to be off the radar of most webmasters but just fast enough it keeps nudging my bot blocker once a day to keep tracking it.

The user agent claims to be Firefox 3:

Mozilla/5.0 (Windows; U; Windows NT 5.1; da; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

Which could indicate someone making screen shots.

No clue what they're really doing but it's going really slow and they're getting pages of garbage instead of what they want, so I hope they're having a real good time fucking with my bot blocker.

SEOMoz's New Linkscape Creates Webmaster Backlash

10/06/08 - A day that will live in internet infamy when a prominent internet company caught millions of webmasters off guard and sent shockwaves around cyberspace.

The event that caused this uproar was the launch of Linkscape with supposedly 30 billion pages indexed that stunned even the most savvy webmasters because they didn't see it crawling and were totally taken by surprise.

This new snooping SEO tool is billed as "An Index of the World Wide Web – 30 billion pages (and growing!), refreshed monthly" which has left webmasters that are already battered and abused by a massive onslaught of automated bots more angry than ever.

The internet entitlement mentality thinks that all webmasters have unlimited bandwidth and CPU and that anything that's online should just be taken without regard to the consequences.

Webmasters will no longer tolerate Indexation Without Representation and are moving to regain control over their sites, their content, and their competitive intelligence. Many webmasters that previously called bot blockers paranoid draconian control freaks are now crying for solutions to high profile marauders raiding their sites and reaping large profits. Now that the tide has turned the webmasters are preparing for the revolution with new sites such as the NoArchive Initiative, better bot blocking scripts, honey pots and much more.

Even a competitive site called MajesticSEO which provides a similar product actually gives Free multiple page reports on your domains if you register and prove you own the site which is at least a symbiotic relationship and not completely parasitic.

However, not only doesn't Linkscape give anything back to the webmaster for allowing your site to be crawled, or mined for competitive intelligence, they actually increased the price $30 to access their tools so you actually have to pay more for the privilege of being crawled to see your own data!

New Pricing, featuring three levels of PRO membership depending on the size and needs of our members. Current PRO members need not worry - you'll be grandfathered in at the current price level. We're just creating two new echelons for those who need access to more. If you'd like to lock in at the current price level ($49/month), I won't stop you :-)

So what benefit do we all get from all this?

For a monthly fee our competitors can see everything we do in infinite detail.

Wow! That's a benefit?

Sorry, but that just doesn't play homey.

Vote for the Best Tattoo - Neil Patel vs. IncrediBILL

Neil Patel has been making some noise about a lady friend getting his name tattooed her lower back as the ultimate sign of success.

Not to be outdone, one of my bot blocking fan girls stepped up to the ink for this IncrediBILL asstravaganza tattoo.

So which is the best tattoo?

You be the judge!

Exploring The Tynted Web

Here we go with Tynt, yet another startup trying to socialize the web.

Joy.

Yesterday there was a little bit of a flap about Tynt as they're running a wide open proxy on tynted.net that allows page hijacking in the search engines.

Today Tynt responded on their blog to the SEO community.

First, we understand that Tynt has the potential to impact the major search engines in ways that were detrimental to the sites being Tynted. Our community recommended blocking spiders from crawling Tynts through the use of a Global Robots Exclusion file (robots.txt) as well as other techniques to minimize the problem. We have already implemented the ROBOTS.TXT file and are working on additional solutions.

That's a noble effort but now anyone running AdSense, YPN or any other contextual network ads are shit out of luck because their bot has to visit the page to serve up the ads. This turns out to be a moot point because allowing the bot also makes the ads go to shit for other reasons explored below.

I think Tynt is missing the point that their proxy server is wide open and can be used by scrapers and other online vermin to access your site although they might be blocked by other means.

Here, try it with Google or Yahoo or whatever you want, wide open and works for anyone and is ripe for abuse including phishing expeditions, very nice.

Shouldn't that proxy only work for registered members that are currently logged in?

Just a thought, I know it's beta and you want to demo some pages, but put a unique key in the URI so that only pages requested from actual Tynt members works with the proxy and it can't be randomly exploited.

For instance, instead of allowing the raw path "google.com.tynted.net" maybe it should be "google.com.someuniquekey.tynted.net".

BTW, pass through the actual end user in your HTTP_X_FORWARDED_FOR field as we don't really find the NAT addresses of your internal servers all that useful.

Moving right along...

On their Twitter bio it says:

Tynt lets you put contextual relevance and dialog on web pages for sharing and interaction.

Listen, if I want dialog on my web pages, I'll put a comments section on the bottom of the pages, I don't need or want your help in this matter.

They clarify this further on the Tynt FAQ page:

Q. What kind of 'stuff' can I put on top of a web page?
A. There is a bunch of different things that you can do to a web page ranging from tools for research like sticky notes and highlighting text, to more fun stuff like text, speech bubbles, graphics and animations. Tynt is a fabulous tool for in context editorializing; in other words, Tynt lets you say what you think right on top of the topic you are talking about.

Lets' see, what does putting things on top of a web page mean? Vandalism, grafitti and lampooning quickly come to mind, something every business online welcomes. Several examples on their site actually show exactly this so no thanks, I think I'll keep my site out of your "stuff".

Wait, it gets even better, we get to foot the extra bandwidth bill for the privilege of letting the Tynt users download our pages just to draw horns and funny faces on our site. Fuck that.

Besides, if you Tynt a blog, forum, or twitter you're actually taking away from the value of those social mediums by breaking up and disjointing the conversation into multiple places which adds no value to the original discussion.

Here's another precious gem from the Tynt FAQ page:

Q. Does Tynt steal my traffic and therefore my revenue?
A. All Tynted web pages, including images, ads and other media all load live from the originating web server so every time a Tynt is viewed the Tynted web site get the ad revenue and traffic.

Holy misconception and major bullshit alert Batman!

Anyone using a context sensitive advertising medium like AdSense and YPN will be in trouble. This is because AdSense and YPN doesn't know their context in relationship to the tynted.net domain name and it shows a bunch of off topic garbage which won't interest the person viewing the page whatsoever.

Want proof?

Let's use a Google AdSense case study site called CoolChaser just to see how well the ads work before and after you run the site through Tynt.

The results we saw were priceless:

Gay Bears Chat anyone?

That's just what my visitors crave, they love big hairy men that like other big hairy men and just can't keep their hands off of them - NOT.

As a matter of fact, something like this happening on a family friendly website could cause a huge problem but we won't delve into those issues at the moment.

Sorry, nothing about your Beta causes this problem as this is how AdSense responds over most any proxy and even in their own cache pages so anyone relying on AdSense or YPN revenue that has traffic redirected through Tynt's proxy will probably just lose out.

One of the Tynters tweeted me:

iancheung @IncrediBILL Tynt drives more people to sites and since people make money by ad views, it actually increases revenue.

I don't see how increased revenue is possible since you can't even see the ads because they're covered up with all those goddamn sticky notes!

From the blog:

Second, site owners have requested the ability to opt-out of having their sites publicly Tynted. We’ve given this a great deal of thought...

I gave it 2 seconds of thought and blocked your IP ranges:

Tynt Multimedia Inc. (TYNTM)
204.244.109.240 - 204.244.109.247
204.244.120.176 - 204.244.120.183

Out of site, out of mind, not a problem.

But wait, they have more in store:

The reason for the gateway (and the different looking URL) is to allow us to insert the JavaScript which loads the Tynt engine for the in-context comments and conversations (and hey, if everyone installs the plug in, then there is no need for our gateway and we can save ourselves the bandwidth and effort there too!).

The end users will be doing the page loading and we'll be unable to see them or stop them from fucking with our pages.

Many webmasters take their livelihoods and reputations very seriously and don't like being fucked with so there needs to be a way to detect the use of Tynt and or a way to opt-out of Tynt before this happens or it could get very ugly.

Last but not least, Tynt has made no mention of how they plan to make money.

Do you ultimately plan on using our sites to trigger your ads?

That's when the shit will really hit the fan.

Have a nice week.

2008 a Chrome Odyssey

Did Stanley Kubrick join Google marketing?

Ever since Chrome released it's been like watching the online monkeys beat the monolith with bones like the intro of 2001 a Space Odyssey.

Sometimes people just make me want to smack them upside the head the way Gibbs dishes it out to DiNozzo on NCIS.

First, Google releases a beta product and everyone starts reviewing it like it's a finely polished shipping product. For those of you not in the software business there are various levels of beta which typically evolve into final beta, meaning that all features are frozen for that release, Then comes a beta gold candidate which appears to be as bug-free as possible and is about to become a shipping product. The initial release of Chrome is an obvious real beta that they want feedback on so get over it, it's an early beta, it's no where close to a final beta IMO but it's a damn good first release.

Second, a whole bunch of people are trying to steal Google's thunder comparing it to MSIE 8 primarily, which is also still in beta. If Google didn't already have those features coded then they wouldn't be in their initial beta either. It's not like someone at Google woke up on a Monday, read the MSIE 8 feature list, and wrote a whole bunch of new features for Chrome that just showed up in the release on Tuesday, get real. Most of those features were obviously in the works for quite some time but MSIE 8 was publicly known opposed to the closely held secrets of Chrome.

Thirdly, it's mostly just desperate cries for attention and link bait for people that have nothing better to do than bash Google on a good day and how there's more fuel for the fire.

Just remember, the first version of Firefox wasn't much to write home about which was initially distressing considering it's Netscape heritage, but it got better in a hurry until they bloated up Firefox 3, what a slug.

Additionally, Chrome is built on the same rendering engine that runs Safari so it has a lot of history already and should be pretty solid except for it's handling of some plug-ins which will be fixed.

Overall, I'm hopeful that Chrome gets the bugs patched quickly because I love the speed and snappy page displays like you get with Opera without all the javascript quirks.

I just hope it ships with add-on capability or more java script control like Firefox's NoScript and I'll be a happy boy.

Remember people, it's JUST BETA, give them a chance to polish it up.