IncrediBILL's Random Rants

CrawlWall LIVES! Signup Now!

2011-12-27T10:33:00.000-08:00

Some of you probably never thought you would see it happen, but CrawlWall LIVES!Even though I've been sick as a dog for months, I started plugging away at the damn thing and doing some unit testing of various components on IncrediBILL.net, and now it's close to going into alpha testing.Go signup if you want to be a CrawlWall alpha tester, I think you'll like it.BTW, approved alpha testers, and it

Bot Blacklisting vs. Whitelisting, are you a convert yet?

2011-11-10T19:56:00.000-08:00

I'm still shocked that after all these years people not only are still practicing the ancient black art of blacklisting, but I'm even still more shocked to see several so-called website content security products being recently released that rely on blacklisting as their primary defense.Are they fucking kidding?Do people really pay good money to chase an endless supply of bots?Let's explore the

Tracking Domain Intel Site Bots

2011-11-08T02:48:00.000-08:00

I've taken a recent interest in tracking down the shitload crawlers of the domain sites out there that scrape your homepage, keywords, etc. and even display your AdSense and Google Analytics IDs.Fucking asshats.Posted a bunch of them on WebmasterWorld so drop in there to get details about domainsoutlook.com, statshow.com, urbandata.com, zitetrendz.com, hostnology.in, dawhois.com,

Where's Bill Been Hiding?

2011-10-15T03:10:00.000-07:00

I actually haven't been hiding, I was on the endangered species list.I have been sliding down a slippery slope for about a year now and seem to be finally recovering.Hopefully.It all started around the beginning of the year when my doctor gave me a new drug that after about a month appeared to be causing a mild rash. We didn't stop the drug at the time because it also seemed to be working

Looking for LinkScrubber Alpha Testers!

2011-06-30T06:52:00.000-07:00

After years of running an internal link checker that rocks, I'm taking it public!Why LinkScrubber?The answer was simple, the other link checkers sucked and just reported "200 OK" for many bad sites. They didn't catch domain parks, prize pages, SEO iujection hacks, soft 404 pages, so on and so forth, so it had to be written. Now LinkScrubber has a very extensive set of fingerprints for all of

Anyone still listening?

2011-05-26T06:11:00.000-07:00

I haven't blogged much for a variety of reasons this year, sorry about that.Got a lot going on, anyone even still listening?Probably not, don't blame ya.

Whitelisting, Not Blacklisting to stop bots!

2011-05-24T11:44:00.000-07:00

Really getting sick of repeating myself as people just don't seem to get it when it comes to blocking bots that blacklisting doesn't fucking work.Blacklisting requires wasting time chasing bots in access logs, huge ass .htaccess files that slow Apache and impact server performance, and are easily bypassed with changing a single character in the user agent name.Whitelisting on the other hand only

When Webmastering Should Become A Capital Crime

2011-04-27T21:15:00.000-07:00

A little project I've been working on has helped me unearth some of the sheer horrors that the search engines have to deal with on a daily basis and some web designers and webmasters should literally be shot.How about a fully functional web page that has no anchors on the HTML page, none whatsoever, yet you can obviously see a fully functional navigation menu on the page?That's right, some dumb

GlueText Crawlers Identified and Blocked

2011-01-28T11:04:00.000-08:00

Started noticing some leeched content showing up on a site called GlueText so it got my curiosity up to see how they were gathering their content.Turns out initially they were using the default libwww-perl user agent back in '0999.231.221.217 "libwww-perl/5.820" Looks like they got a little smarter after being bounced by sites to switch to the old Netscape Navigator user agent for the Win 98

Having Netflix or Hulu Buffering Problems?

2010-12-21T10:06:00.000-08:00

Several times now I've had buffering problems with Netflix or Hulu when being played over wifi via the Wii or Playstation 3 on the big screen. To quickly diagnose whether or not it was a possible "throttling" situation, the first thing I did was verify if my desktop computer, which is hardwired to the internet, not wifi, was having the same problem or not. The desktop computer was running full

Domain Parks Hijacking Previously Hacked Sites?

2010-09-26T15:41:00.000-07:00

Lately I've been seeing an increase in some domain parks using frame buster scripts and it really didn't make any sense to me.Why would a domain park need a frame buster script?Then I discovered a potential secret, these domain parks, whether intentional or not, are inadvertently hijacking sites that have been previously victims of the iframe injection hack!Let's examine how this works.The iframe

Final American Airlines Flight - Never Again!

2010-09-21T16:28:00.000-07:00

Just took 4 flights on American Airlines and I'm done, finished, never doing it again.On the first flight out the entire cabin aisle was jammed with assholes in line for the crapper. Nobody told them to sit down and wait, there they were bouncing back and forth in the aisle bouncing off passengers sitting in the aisle. Then some crazy bitch with a hot cup of coffee joined the line holding this

Spam and Trojans Exist Because People are STUPID!

2010-09-20T07:35:00.001-07:00

That's right, don't blame the spammer, don't blame the guy writing the Trojan horse attachment in your email, blame yourself and your stupid friends.Why blame yourself and your stupid friends?Because if people weren't so fucking stupid to buy shit advertised in spam or open those fucking Trojan horse email attachments then the authors of that shit would simply go away.That's right, all you idiots

How to Upgrade HP-MINI 210 Netbook Memory: FACTS NOT FICTION

2010-09-03T16:43:00.000-07:00

We've got a couple of HP-mini's and they're nice little netbooks, but when I wanted to upgrade them to 2GB, that's when the bullshit started. Trying to find any resources online about the specific makes and models and which chips they require is truly maddening. Making it even more maddening are the journalists that add to the steaming pile of misinformation. However, after taking the time to

Are Joomla! Users Lazy or Just Stupid?

2010-07-08T14:23:00.000-07:00

Let's start by looking in Google for the default Joomla! title "Welcome to the Frontpage" and Google claims there's 5,000,000 results with all those words in the title.Absofuckingtively unfuckingbelievable.However, many Joomla! users seem to somehow manage to change their title but leave the meta keywords and description as follows:

Android Phones Make Men Better Lovers than iPhone

2009-11-07T20:10:00.000-08:00

Women, pay attention because this blog post could change your life!Your sex life that is.Does your man have an iPhone or iPod?Throw those iFuckers away and get your man an Android phone for his birthday, Christmas, or just because you want the wettest wildest screaming orgasms you've ever had.How is this possible?Well the Android phones have a little clitoris sized trackball on the front of the

Poker, Charity and Big Glands

2009-11-06T20:50:00.000-08:00

Last year it was easy to get into Purpose Inc's 2008 Poker Tournament, you just had to pay the entry fee and they picked the charity, easy peasy, no fuss no muss, just how lazy shits like myself prefer it.Not this year, for Purpose Inc's 2009 Poker Tournament they're making me think about stuff and blog it, then post images, links and all sorts of crazy shit, too much like work.Considering I

Can Open Source Topple The Microsoft Monopoly?

2009-11-04T14:18:00.000-08:00

I've been using Microsoft Software since 1980 and this is the first time in the history of computers that I think Microsoft is in a position to be easily toppled.Who will topple Microsoft, will it be the next big computer company with a "Windows Killer"?Nope.It will be FREE software.It will be OPEN software.It's already here today, there's no waiting, you can download it all for free!Free Desktop

PayWave Credit Card Cloned - Was My Ass Scanned?

2009-11-03T07:17:00.000-08:00

In all the years I've been using credit cards I've not had a single incidence of credit card fraud until now.Someone actually CLONED my Visa PayWave card and was using an actual physical copy of the card last weekend in person in Illinois, which is a long way from California.The credit card fraud department caught the activity instantly, and shipped me a new card via UPS in 2 days.I'm very

Why buy MS Office when Open Office is FREE

2009-10-21T11:46:00.000-07:00

I've used MS Office products such as Word and Excel for about 20 years now and I've always upgraded to the latest version as needed.When I got my new computer a few months ago it came preloaded with Office 2007 for a limited trial and guess what?The trial expired and I didn't purchase.Nope, I decided to bail on MS Office and try the free OpenOffice and see how it worked before forking over real

Unleash HTC Hero Android's Bar Code Scanner

2009-10-20T15:39:00.000-07:00

The purpose of this post is to show people how to improve their marketing efforts to the mobile market and speeding up basic information acquisition such as the URL to your site or your contact data is a major step forward.How?The Barcode Scanner!Many of you are carrying phones around that have bar code scanners built into them and you probably have no clue about the true potential sitting in the

Sprint HTC Hero Android Phone Custom Notification

2009-10-15T06:33:00.000-07:00

The only thing I found real difficulty with in it's initial intuitiveness on the Android phone was setting the custom notification sounds for various things to sound unique.For instance, I have specific inbound emails that come from my alarm monitoring company when a server goes down and I need it to wake up the dead when the alarm goes off not just go "BING!" and go away.So here's the dirt on

HTC Hero Android Phone Music Sync

2009-10-11T20:29:00.000-07:00

I'm not sure what all the fuss is about not having any software to sync music for the Android.ADDED: MOUNT ANDROID PHONE FIRSTMany first time users don't seem to know how to make the Android Visible to the PC. Pull down your notifications window on the Android, you should see an entry at the top "USB Connected", touch that then touch "MOUNT" and your PC should be able to see your phone. An

Sprint HTC Hero : Android Phone or Tricorder?

2009-10-11T14:11:00.000-07:00

Just picked up my new HTC Hero phone from Sprint yesterday and it's pretty damn cool all on it's own but after installing a few super cool apps, it's damn near a Star Trek Tricorder!Why do I equate the HTC Hero to a Tricorder?Because it interacts with the real world in some amazing ways that are just the beginning of what's to come.It can hear and analyze voice and musicIt can see barcodes, do

Screwed of Prize by IMCharity Party

2009-08-13T19:24:00.000-07:00

Yeah, that's right, I'm going to rant about a charity party.However, before I rant, I'm going to 1st say it was REAL FUN and I had a REAL GOOD TIME and I'm wondering why the place wasn't packed way more than it was and where all you cheap SEM/SEO bastards were that could've contributed to the IMCharity?Anyway, on with the rant...After shelling out $50 to get inside which is nothing, it's for a

Alexa The Texting Sewer Diving Moron

2009-07-14T20:45:00.000-07:00

A manhole cover left off and a stupid girl texting without watching where she's going meet their shared destiny in one smelly misstep.I'll agree that the city employee that left the manhole cover off needs to be taken out behind the woodshed and let his Grandmother beat the shit out of him with a belt for doing something so stupid.However, Alexa bears some blame here too walking around not paying

Men Not Working - Slow Blog Ahead

2009-06-30T21:19:00.000-07:00

After a few weeks it might seem like I've given up on the blog but that's hardly the case.Just been using my blogging efforts in other places, stirring the pot here and there, and taking some heat for stupid comments.If you missed some of the fun try these threads for gigglesSomething AdSense:Google Bugs May Contribute to Lower AdSense IncomeSomething Linux:Making easy graph XML output from log

Stop DNS Site Hijacking With Apache Redirect

2009-06-07T19:18:00.001-07:00

There have been several instances of DNS hijacking reported lately so I thought it might be useful to briefly address this problem and it's easy resolution.DNS hijacking occurs when someone using a different domain name aims their domain name at your server and it hijacks a domain on your server. Both domains, the hijackers and yours, now serve up the same exact content. The search engines aren't

Updating Apache Zaps Plesk 8 Suexec

2009-05-28T00:36:00.000-07:00

Boy did I hit the panic button tonight when I did a server software update and suddenly everything running in a CGI-BIN folder started generating 500 errors.I remembered this problem with earlier versions of the Plesk control panel I use, but it seems that back then it was a simple CHMOD problem, took a look and the SUEXEC group, owner and flags were all still proper.Lots of stuff down, clock

Bankrupt Silicon Valley Crybabies - Try Living Within Your Means

2009-05-26T17:08:00.000-07:00

Reading this post on WebGuild about "Down and Out in Silicon Valley" really pisses me off.The author is trying to make me feel bad about someone making $12K/month ($144K/year) that's now grabbing grub at the food pantry and using food stamps.Give me a fucking break, why the hell did you spend it all?What part of "bank" didn't you understand?Back in the 90's someone I used to work with was a

I can't blog anymore!

2009-05-13T17:30:00.000-07:00

My blogging bone appears to be broken.Got a few good rants backed up, will see what I can do about those.Until then, read this post about Rich Snippets and Microformats

Credit Card Fraud on Unused Card!

2009-03-13T08:47:00.000-07:00

Here's one for the record books.I just got a wake up call at 8am from the credit card fraud dept. at a major bank, who's name will go unmentioned, but you can purchase their stock for less than the price of a Happy Meal.Anyway, one of my credit cards that's been quietly and securely sitting in my desk drawer and hasn't been used in over 5 years is out making ridiculously small purchases. If this

Florida Drinks Cat Piss Picture Book

2009-02-22T13:07:00.000-08:00

A couple of years ago I was faced with the horrors of the cat piss they try to pass off for beer in Florida. Last week I was back in Florida and found myself face to face with the same situation, a state full of cat piss beer, but I was determined to find something drink worthy.A few places had something called Yuengling which by it's very name sent shivers down my spine sounding like some rice

Iterasi Archives Sites Without Permission

2009-01-31T11:39:00.000-08:00

Guess what boys and girls?There's another wonderful new site that allows people to copy your shit without your permission!Iterasi allows their members to "archive" individual web pages.The pages on my site have a meta tag "NOARCHIVE" which tells everyone DO NOT ARCHIVE this page yet they archived it anyway. They also stripped out my frame busting javascript so they are seriously thwarting sites

MSNBOT Crawled Thru Javascript!

2009-01-01T15:48:00.000-08:00

Today I caught MSNBOT-MEDIA crawling thousands of links that were only accessible thru javascript.These links were only intended for human use, only accessible via javascript, therefore never added to robots.txt...... until todayHere's the MSNBOT specifics used for this crawl:65.55.235.202 "GET /feedback.html?id=1010101234 HTTP/1.0""msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"I have a

Design With Screen Shots in Mind

2008-12-24T13:31:00.000-08:00

With the proliferation of screen shots everywhere you would think that site designers would make sure that their sites make good screen shots, right?Unfortunately this is not always the case.When a surfer uses a visual search engine or a directory that has screen shots of the site the visual appeal often dictates which site is chosen and which site is left behind, not the SEO value of the text

How to Block SpyderMate SEO Tool

2008-12-07T15:03:00.000-08:00

Trapped another SEO tool called SpyderMate that crawls your site analyzing data.Nothing wrong with using it but don't let competitors get a free ride analyzing your site.The spider details:208.78.98.236 [208-78-98-236.slicehost.net.]"MentorMate Spider"Their host:OrgName: Slicehost LLCOrgID: SLICENetRange: 208.78.96.0 - 208.78.103.255CIDR: 208.78.96.0/21They just keep coming and eventually they'll

Top 20 Forum Spammers for 2008

2008-12-05T09:13:00.000-08:00

So far this year I've been tracking 9,694 unique IPs attempting to spam 49,492 links on my sites and they all bounced into a spam tracking log file.Using this log file I can track where all the crap is coming from and (not so) surprisingly it's primarily coming from Europe.The #1 spam host is blueconnex.net which wins the prize with 4 IPs in the top 20 and their #1 spammer at 7,798 posts is still

Temporarily Block HotLinking To Find Copyright Abusers

2008-11-05T12:44:00.000-08:00

Blocking hotlinks is usually considered a method used to conserve bandwidth and stop leeching of images off your server. However, you can also use hotlink blocking to quickly and easily find all those sites using your content.The most common solution for Linux servers is to add the following hotlink blocking code into your .htaccess file.RewriteEngine OnRewriteCond %{HTTP_REFERER} !^$RewriteCond

Pubcon '08 and Other Announcements

2008-11-03T00:42:00.000-08:00

I'll be presenting at PubCon '08 on the topic of Competitive Intelligence. The only difference is the other panelists will be discussing how to find competitive intelligence while I'm telling people how to protect themselves from such research.Also, keep your eye on this space:http://twitter.com/CrawlWallAll shortly

JadynAve Bot Wants Your Local Data

2008-10-30T11:35:00.000-07:00

If you have a bunch of local data like I do then you better protect it because JadynAve's Local Business Search appears to be coming after your site with their JadynAveBot!Didn't ask for robots.txt, has no data whatsoever on their robot page except to email them if you have any questions, big whoop.Here's the IP and user agent:38.99.186.40"Mozilla/5.0 (compatible; JadynAveBot; +http://

Suspected Copyright Offenses

2008-10-28T20:17:00.000-07:00

Something amusing hit my site from from .t-dialin.net which appears to be .t-online.de or the German version of T-Mobile.I see the following IP and user agent:84.153.98.95 "Verdacht Vergehen nach UrhG"Which Google translates into:Suspected offenses under the Copyright ActWell isn't that just the cutest little user agent to get caught in a bot blocker?Now I've had my chuckle for the evening, back

Why Does Copyscape/GoogleAlert Hide?

2008-10-28T12:45:00.000-07:00

Never really played around with Copyscape/GoogleAlert much but I noticed it tries to completely hide it's presence when accessing a server which isn't cool.Not that I'm a fan of plagiarism as my copy of the DMCA is almost worn out from use, but I'm even a less fan of sneaky web crawlers that pretend to be shit they aren't.The IP that Copyscape uses:212.100.254.105 -> www.googlealert.comThe

Viewzi's Meta Search Engine Taking Screenshots Without Permission

2008-10-27T23:31:00.000-07:00

Here we go again with yet another visual search engine called Viewzi taking a bunch of screen shots without asking for permission from robots.txt.In this case it's a meta search engine and technically the search engines Viewzi culls from has been given permission to crawl, but Viewzi itself was never given access permission.Here's the Viewzi user agent:Mozilla/5.0 (X11; U; Linux i686; en-US; rv:

Google Analytics Finds Bandits and Proxies!

2008-10-25T15:46:00.000-07:00

Google Analytics has a Hostnames feature that most overlook which normally displays the hostname of the site your visitor landed on, like example.com. However, you'll probably notice a bunch of IP addresses and other interesting information in this list including sites that may have stolen your content!To see what I'm referring to go into your Google Analytics account and go to Visitors ->

Possibly Slowest Scraper Ever

2008-10-13T00:36:00.000-07:00

I've seen slow scrapers before but this is fucking ridiculous. 80.197.39.182 has been automatically challenged to answer now 227 times since 09/23/2008 and it just keeps plugging along slow enough to be off the radar of most webmasters but just fast enough it keeps nudging my bot blocker once a day to keep tracking it.The user agent claims to be Firefox 3:Mozilla/

SEOMoz's New Linkscape Creates Webmaster Backlash

2008-10-09T03:30:00.000-07:00

10/06/08 - A day that will live in internet infamy when a prominent internet company caught millions of webmasters off guard and sent shockwaves around cyberspace.The event that caused this uproar was the launch of Linkscape with supposedly 30 billion pages indexed that stunned even the most savvy webmasters because they didn't see it crawling and were totally taken by surprise.This new snooping

Vote for the Best Tattoo - Neil Patel vs. IncrediBILL

2008-10-08T20:01:00.000-07:00

Neil Patel has been making some noise about a lady friend getting his name tattooed her lower back as the ultimate sign of success.Not to be outdone, one of my bot blocking fan girls stepped up to the ink for this IncrediBILL asstravaganza tattoo.So which is the best tattoo?You be the judge!

Exploring The Tynted Web

2008-09-11T14:58:00.000-07:00

Here we go with Tynt, yet another startup trying to socialize the web.Joy.Yesterday there was a little bit of a flap about Tynt as they're running a wide open proxy on tynted.net that allows page hijacking in the search engines.Today Tynt responded on their blog to the SEO community.First, we understand that Tynt has the potential to impact the major search engines in ways that were detrimental

2008 a Chrome Odyssey

2008-09-07T14:29:00.000-07:00

Did Stanley Kubrick join Google marketing?Ever since Chrome released it's been like watching the online monkeys beat the monolith with bones like the intro of 2001 a Space Odyssey.Sometimes people just make me want to smack them upside the head the way Gibbs dishes it out to DiNozzo on NCIS.First, Google releases a beta product and everyone starts reviewing it like it's a finely polished shipping

Chrome Shines While Fat Lady Sings for Opera, Firefox and MSIE

2008-09-02T16:58:00.000-07:00

I'm very hard to impress, an old school hard core programmer that detests software bloat from lazy assed programmers using poorly implemented cross-platform development environments.I like fast and lean code, shit that makes an old laptop look all shiny and new and maybe that's why Google named their new browser chrome cause this fucker shines.It was only yesterday that I threw my hands up in the

Opera 9.5 Smokes Firefox 3

2008-09-01T14:30:00.000-07:00

The latest Firefox 3 upgrade was just pitiful and the memory footprint was fatter, it seemed to be leaking memory, page loads just crawled at a snails pace, it was an overall hog.The options were limited:- Go back to MSIE 7, which isn't much better, or- Give Opera one last try as I've always found something that stopped me from using Opera in the past.Well, the first thing I noticed is Opera 9.5

MJ12BOT's Dirty Little Secret

2008-08-31T14:03:00.000-07:00

Many of you have seen MJ12bot hammering your site from IPs all over the world, both the legit crawler and the fake virus version 1.0.8 as well."Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)"Everyone likes an underdog and we tolerate these crawlers when they claim such a noble purpose:We do spider the Web for the purpose of building a distributed search engine

FireFox 3 Bloated Leaky Pile of Shit

2008-08-31T13:05:00.000-07:00

Firefox 3 with all it's much anticipated upgrades turned out to be a big fat flop that's annoying me to the point I'm thinking of switching back to Internet Explorer or giving Opera a try.Some of the new features seem interesting but overall it's slower, bloated, and appears to be leaking faster than the Titanic after a close encounter with an ice cube.The damn thing starts at about 40MB and just

Woman Weather Channel

2008-08-24T15:06:00.000-07:00

During a discussion with a few of my male friends today it became obvious that some of us could benefit from such an online service.You could have a widget, perhaps a Google Gadget even, sitting on your desktop that declares your woman's current menstrual state such as:Today: Slightly SpottyThen you could click on the widget and look at the 5 day forecast and see what's in store for the week:Mon:

How Flawed is Your Anti-Virus?

2008-08-14T10:46:00.000-07:00

Some of the anti-virus web surfing protection products are permitting some very risky behavior due to flaws in their basic design. For instance, some of them allow your browser to willingly go to known bad locations they have in their database until something catastrophic gets downloaded. Once the file is downloaded it might be too late so there's the real problem.Here's a quick for instance, the

SEO Community in TailSphinn

2008-07-24T11:06:00.000-07:00

I tried to support Sphinn's efforts by putting the SphinnIt button on my site to help raise awareness of what they were trying to do with something unique for the SEO community.Unfortunately, Sphinn devolved into a bunch of Sphamm and when one of their members pointed out how widespread the problem was they banned Edward. OK, Edward (pageoneresults) can push the envelope a little but it wasn't

iPowerWeb Hacking Continues

2008-07-06T16:55:00.000-07:00

Over a year ago I wrote about a bunch of iPowerWeb's shared servers being hacked, and it looked like they were trying to clean it up, but now it's time for round two of hacking.The latest batch of hacked sites may have a DNS hack as well, I'm not sure that's the case but Alex seems to think it is.All these sites have the following Whois Name Server entries:Name Server: NS1.IPOWERDNS.COMName

AVG 8 LinkScanner Fiasco Recap

2008-06-17T12:16:00.000-07:00

For those of you that might've missed the whole AVG 8 LinkScanner disaster and ensuing AVG reputation nightmare, here's a quick recap and links to places to read all the details.Webmasters started noticing a rash of distributed IP's with the same user agent, no referrer, and a few other technical issues I won't go into now, that suddenly started pounding their sites:Mozilla/4.0 (compatible; MSIE

Did CSC's Spybot Get Caught?

2008-05-22T02:00:00.001-07:00

Looks like yet another corporate compliance spybot is hitting our servers, not like we need yet another spybot.There's only one IP out of this entire range that consistently hits my servers.OrgName: Corporation Service CompanyOrgID: CORPO-9-ZNetRange: 165.160.0.0 - 165.160.255.255They claim to crawl the web:Our proprietary technology scans and digests web pages

Impact On Your Bandwidth Will Be Minimal My Ass

2008-05-12T15:24:00.000-07:00

How often do we see that happy line of horse shit spread by every new startup that crawls the web about how minimal it's impact will be?Every fucking one of them claim it but when you add them all together the bot traffic is quickly exceeding the human traffic.Who the fuck am I kidding, on most sites the bots clearly out number the humans in pages read on a daily basis.First we put the big search

Comparing Effectiveness of Anti-Virus Web Protection Methods

2008-05-12T11:46:00.000-07:00

There's three basic methods being used at the moment to protect web surfers from potential dangers which are static (stale), active and passive.Static Web ProtectionVarious companies use the static method which relies on crawling the web in advance to find vulnerabilities and then attempt to warn visitors about these problems as they are about to visit a web site. McAfee's SiteAdvisor and Google

Off By More Than One

2008-04-27T13:42:00.000-07:00

Can you believe that someone is actually surfing the web using some free browser called Off By One that doesn't appear to have been updated in the last 2 years?The user agent is as follows:"Mozilla/6.0(compatible;OffByOne;Windows 2000)"The irregular formatting convention triggered the bot trap with the lack of spaces alone.Then it claims to be Mozilla 6.0 when it's probably Mozilla 3.0 at

Reciprocal Link Exchange? Let's Swap!

2008-04-20T15:32:00.000-07:00

For years I've been deleting all those emails asking me to exchange links and I won't swap links with any of that crap.Suddenly I've had an epiphany and YES!, now I'll swap links with you, no problem!I'm only agreeing to swap links as requested.I'm not using NOFOLLOW on those links as requested.You can see my links when you visit, online and visible as agreed.Unfortunately my link swapping page

Kaushik, What Freaking Experiments?

2008-04-20T14:24:00.000-07:00

I found this user agent coming out of Microsoft's Area 131 requesting that people "contact kaushik for these experiments" that kept hitting one of my servers.131.107.0.96 "contact kaushik for these experiments" So I did a little data mining of my own and searched Microsoft and couldn't decide if this experiment was from Kaushik #1 or Kaushik #2.Both Kaushik's appear to be working for the Data

DNS Right But User Agent Wrong

2008-04-20T14:11:00.000-07:00

Ran into a user agent from DNSRight today that claimed to be some link check tool that doesn't appear on their site.66.240.236.220 "GET / ""http://www.dnsright.com/" "DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com"So I ran some of their other tools that don't identify themselves at all.66.240.236.220 "GET / HTTP/1.1" "-" "-"They host this mess at cari.net so just block '

Picmole, Yet Another Spybot!

2008-04-17T15:59:00.000-07:00

There must be good money spying on everyone because it seems a new company springs up almost weekly trying to claim their stake in this new gold rush.How many fucking spybots do we need?Today on the spybot circuit the we're serving up a helping of Picmole that's using heritrix to do it's crawling. Surprisingly it still checks robots.txt but who knows if they'll honor it down the road because

Favcollector Bandwidth Waster

2008-04-17T10:46:00.000-07:00

Here's another product of Canada doing the stupidest shit ever seen, collecting favicons.It came and grabbed my icon, then hit the home page which the bot blocker promptly stopped, so who the knows what else it would've done beyond that.66.207.217.138 [gaspra.crazylogic.net.] "Favcollector/2.0 (info@favcollector.com http://www.favcollector.com/)"From their FAQ:Favcollector is a spider that

Canasasearchbot For Canasians, Oh Canasa!

2008-04-17T10:13:00.000-07:00

It's hard to resist commenting on a bot that can't even spell it's own name or it's country name correctly.206.248.137.34 [mycanadasearch.ca.] "canasasearchbot(http://www.mycanadasearch.ca/robots.html)"However they got it right on their robots page:User-agent: canadasearchbotIt did ask for robots.txt but who knows if it was looking for "canasasearchbot" or "canadasearchbot", total crap shoot.I

Mozshot Tries Taking a Screenshot

2008-04-14T17:46:00.000-07:00

Yet another Firefox-based screen shot tool hit my other site today just in time to take a screen shot of an error message telling them they weren't allowed to take screen shots without permission.Details:61.206.125.245 [tempest.nemui.org.]"Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; http://mozshot.nemui.org/)"This thing appears to be open source, oh joy...

RTGI - The French Social Media Spybot

2008-04-11T10:43:00.000-07:00

Yet another social media mining operation designed to track every bit of intel said about brands, people, politics and more.From a translation of their site:Our solutions simplify the identification of influential communities and monitoring of their conversations, to the benefit of businesses, communication agencies or research institutes.RTGI's approach allows the analysis of the links and

Project Rialto's PRCrawler Is Data Mining?

2008-04-11T09:56:00.000-07:00

Since I whitelist allowed bots I've had Project Rialto blocked since the beginning but I was curious what they were doing since they first showed up on my radar on 01/23/2008 and kept coming back over and over.From one of their job ads:We are designing high-performance algorithms and developing reliable, fault-tolerant and scalable real-time systems that can handle

Radian6's R6_FeedFetcher Fetching More Than Feeds

2008-04-09T17:10:00.000-07:00

For those of you unfamiliar with Radian6 it's a "social media monitoring tool" because apparently everyone with an opinion on the internet needs someone to spy on their ass since we're disruptive.Well bummer.Isn't it a shame the good old days are gone where companies told you everything you needed to know about their brand and you had to be a journalist just to get your opinion heard?Of course

Discovery Engine's Discobot Discovered My Bot Blocker

2008-04-04T22:50:00.000-07:00

I found this little Discobot from Discovery Engine trying to dance around on my server but the bot blocker bouncer at the door was already keeping him behind the velvet ropes.Here's a sample of what I saw on my site:208.96.54.74 "GET /robots.txt""Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)"208.96.54.68"Mozilla/5.0 (compatible; discobot/1.0; +http://

Persaibot - The Rude Crawler

2008-04-04T21:45:00.000-07:00

I saw this little Persaibot hit my site today without even looking at robots.txt and their website has the balls to say:Persai uses this bot to crawl the web. It's probably the nicest bot with the greatest personality in the world. Seriously, give it some attention. Exactly how nice can a bot be that doesn't read robots.txt?Did you read it and cache it some other day?Doesn't matter, that was more

DART Agent - Another Annoying Distributed Tool

2008-04-04T11:36:00.000-07:00

This little annoying DART thing that keeps bouncing off my web site appears to be written by CRS4, the Center for Advanced Studies, Research and Development in Sardinia.It would appear DART stands for "Distributed Agent-based Retrieval Tools" and they even have a workshop in '06 about this damn thing touted as "The Future of Search Engines' Technologies" that had people from Yahoo!, Google,

WHO is Scraping My Site!

2008-03-29T11:31:00.000-07:00

Note the lack of a question mark in the title because this wasn't a question about "WHO?" but an actual statement about "WHO!" and by that I mean the WHO as in an office of the World Health Organization.It registered 411 page requests from 203.94.76.59 which is a non-portable address assigned to the WHO Representative Office in Sri Lanka.Here's the IP and UA:

REBI-Shoveler Digging for Korean Search Engine

2008-03-28T14:22:00.000-07:00

REBI-Shoveler must be easily overlooked as it's very unusual to go to a search engine and type in the user agent and get no authoritative hit from any bot hunter whatsoever. There were tons of hits from various web stat pages but nothing I could easily find that gave me any clue what in the hell this thing was.With this little information all I knew was it came from Korea, otherwise I was stumped

We'll Have Anon Of That, John Doe Must Go

2008-03-28T13:43:00.000-07:00

Looks like JonDonym - the internet anonymisation service is actively operating as those little anonymous hits are coming from their servers.I have a couple of actual scrapes happening from their IPs, who would suspect abuse of anon proxies, right?Here's a couple of examples of activity:141.76.45.34 [proxy1.anon-online.org.]Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.13) Gecko/20080311

Please Install Flash - Idiots Guide To Flash Web Stupidity

2008-03-24T16:42:00.000-07:00

Time to rant about a big pet peeve of mine, that little line of javascript that detects whether or not Flash is installed and the stupid shit developers do when it fails.For a little introduction to the problem, I run Firefox with NoScript enabled globally for security purposes. However, I can easily enable javascript with a click except some developers do some really stupid shit that's costing

SearchMe Demos Wicked Cool Visual Search Engine

2008-03-14T14:55:00.000-07:00

Looks like I was right on the money back in Oct '07 when I announced that I had spotted SearchMe taking screen shots on one of my sites and I knew this was a hot news item but couldn't get the Sphinners to bite on it.Here we are 6 months later and the story broke a couple of days ago on the Silicon Valley WebGuild:Searchme is a new search engine that captures images of web pages and allows users

Welcome to Opt-In Web 3.0 Politeness

2008-03-12T23:47:00.000-07:00

Here's a fine example of how the internet may soon look with an email I got recently that actually asked permission to do something because they couldn't just take what they wanted without asking!The following is slightly edited, but you get the idea:We use a service called xxxxxxxxx.com to provide xxxxxxxx of sites that we have links to on our site.It appears that the xxxxxxxxx.com is being

Gone Fishkin With More SEOMoz Tool Activity

2008-03-09T17:16:00.000-07:00

In my continue series of exposing SEO tools we find this little SEOmoz-bot over at SEOmoz.I'll give SEOmoz some credit where credit is due in they at least identify their tool as a bot so it can be blocked if you want. However, they don't check robots.txt to see if the bot is allowed as I think they assume it's always going to be used by the site owner but it could just as easily be used on some

Smack the SMILE SEO TOOLS Off Your Face

2008-03-09T15:18:00.000-07:00

Some spamming assholes in Russia think automatic directory submission is the same as SEO and added one of my sites to their so called SMILE SEO TOOLS.Here's a list of the various user agents I've seen claiming to be this tool:"SMILESEOTools""SMILE SEO Tools""SMILESEOTools(Windows;compatible;MSIE6.0;I;WindowsNT5.0)"The last user agent with an extremely lame ass attempt to mimic MSIE 6 gave me a

More Pesky SEO Tools To Block

2008-03-09T15:09:00.000-07:00

Seems there is something in Germany called SEO.AG that has been pestering my site for quite some time.The IP and User Agent it uses is:85.214.35.2 "SEO[.AG] - Search Engine Optimizer Bot [http://www.seo.ag]"However, they also run a web proxy on 85.214.35.2 so you have to block the IP to stop all the nonsense.I'm not sure which is worse, the scrapers, proxies, aggrators, or the SEOs and their

You Know You Drink Too Much When...

2008-03-09T14:28:00.000-07:00

When you wake up face down in a pizza you know you got mad drinking skills, especially when you went face down in mid-bite of the pizza.When you wake up and your pillow is covered in pizza vomit, that's madder skills cause you didn't die in your sleep aspirating on pizza vomit. Having to shave your beard off because you can't seem to wash out all the partially digested bits of pizza is a bit

What Time Is It Anyway?

2008-03-09T11:30:00.000-07:00

Got up this morning and all the computers and TV's said it was 9:00am but the phones and alarm clocks said it was 8:00am.Obviously this was the daylight savings bullshit gone bad but how in the hell could someone fuck up the atomic time clock which the alarms and phones feed from?Had this been an actual day when I really needed to get up and be somewhere by 8am I would've been fucked since both

There Goes the Bad Neighborhood

2008-03-09T09:37:00.000-07:00

Isn't it ironic that a day after I wrote about stopping snooping SEO tool's here comes one of them trying to crawl one of my websites.The user agent and IP address are: 208.77.208.198 [emeraldarborvitae.viviotech.net.]"Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)"They were automatically blocked on my site because I white list only allowed user agents and they use an

Jayde NicheBot Crawls for iEntry's Web of Sites

2008-03-08T18:43:00.000-08:00

Who out there remembers the Jayde directory?Some of us submitted our sites to Jayde way back in '96 or '97, who knows exactly, and now our sites are being hit by something called the "Jayde NicheBot"."Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) Jayde NicheBot"I was curious why some site I submitted to about 10 years ago was pinging my server all these years later so I did a little research to