Well guess who stepped into my spider trap today but yet another robot from yet another start-up aggregator site called PageBites that thinks they have entitlement to make money sucking my bandwidth.
Like I've been telling you all for a long time now you need to raise your sheilds to OPT-OUT your site to all crawlers and whitelist just the ones you want to stop this pandemic crawling of your sites. They profit off your backs, gobble up bandwidth costing more money, then muck up your web stats to the point you can't tell your advertisers how many real impressions they really get as crawlers are becoming a pretty decent percentage of so-called visitors on a daily basis.
Search engine spam, email spam, blog spam, now crawler spam.
It's really starting to get to the point that I think turning my spider trap into a product so everyone can take control of their own sites is looking more viable by the day.
You can all take your robots and go home, party over, your ass never got past the home page.
Friday, January 20, 2006
PageBites Job/Resume Scraper
Subscribe to:
Post Comments (Atom)
1 comment:
Maybe I need to hire a small village in india to create a PHP version, or maybe start a non-profit foundation like the Joomla crowd....
Let it be a group effort off my specs.
... or maybe just get off my lazy ass and get all the glory (and any money) for myself
Dammit, that sounds like work!
BTW, have I mentioned that I absolutely loathe (ok hate, well maybe despise, nah - just really annoyed by) customers, deadlines, schedules and support which is why I'm doing my own thing in the first place?
Sigh.
Post a Comment