With the web 2.0 aggregator craze at an all time high maybe it's time for a new business model of Pay-Per-Crawl. That's right, all the start-up leech sites wanting to waste our bandwidth should be sharing some of their VC loot with us just for the privilege.
I'm not talking about ripping anyone a new ass, but perhaps $5 per every thousand pages crawled would help cover my costs of dedicated servers and bandwidth.
Heck, if it wasn't for all the crawlers in the first place my prime site wouldn't even need a dual Xeon server to handle the load so all the bots are definitely running up my expenses so why in the hell shouldn't they share some of that cost?
Oops, they are sharing some of the cost now as I locked them all out so they can't profit from my hard work.
Too bad, so sad.
I'll take a check, money order, VISA or MASTERCARD to let your crawler back in but don't you dare ignore the crawl delay or anything else in my robots.txt or back out you go!
Monday, January 16, 2006
Pay Per Crawl
Subscribe to:
Post Comments (Atom)
1 comment:
Damn I knew you would be hard on the *wipes, but really Bill, give'em a break will you?
You know an arm here and a leg there and soon you could make your own monster ;-).
PSA's to the rescue, we now know what to send the scraper crowd.
Post a Comment