Billed as a RoadBlock to the Semantic Web

Got a sudden burst of traffic from Greg Yardley's site today and noticed the topic was about "The coming semantic web roadblock" which I find amusing as I loathe the onslaught of data miners that hit my site and block their asses automatically on a daily basis.

Greg raised a couple of issues that I've heard a few times from other people that my technology will block everything and prevent new search technology from becoming established, and potentially block things that are currently providing value for your site and that's not entirely true nor my intent at all.

Remember, my primary goal is to make the websites using my product OPT-IN or whitelist things that want access instead of OPT-OUT or blacklist which doesn't work at all.

When you first install this bot blocking tool, it's in a PREVIEW mode by default which means you can see what it would be blocking but no action is being taken. It's completely passive when it's in PREVIEW mode and doesn't even challenge possible stealth scrapers, so it may not know if they're human or not but will take a guess. That means you can observe what's going on with your website for days or even weeks and then authorize anything that's providing value before turning the product LIVE and blocking the rest.

Now the next thing that's important to know is that the product records and reports new user agents that appear, so you will see in REAL TIME when something new, never before seen, hits the site. Remember, since we're OPT-IN, we haven't decided if these new things are good or bad yet so the first time they visit the site they'll get bounced off robots.txt assuming they honor it or not. The next time they visit, if the webmaster decided to let them in, they'll be allowed to crawl without issue.

To summarize, it's up to the decision of each webmaster whether or not the Semantic Web will become a reality or not, not me, my tool or service.

I prefer to think of Web 3.0 as the Democratic Web so if the majority decides to vote the Semantic Web out, who am I to argue?