Now that I'm pretty sure my bot busting techniques are working like a charm the big dilemma is at hand.
A. - Do I completely disclose all the bot busting techniques so that others can bust these scraping assholes too?
or
B. - Do I keep some of the secrets to myself so the scrapers can't adapt and appear invisible to the naked algorithm?
It's a real catch-22 in that disclosing what appears to be solid scraper stopping techniques could unwittingly let me get scraped all over again. Most likely my web site with 40K pages would still be safe from a complete scrape as you just can't hide that kind of activity but more subtle "update" scrapes just culling the most recent content additions would be easier to slide under the radar.
What to do, what to do...
At the moment, I think I'll do nothing except document it for my own purposes.
What happens after that is anyone's guess.
Thursday, January 12, 2006
Bot Busting Primer vs Security Concerns
Subscribe to:
Post Comments (Atom)
4 comments:
Or, be a pal and send the plans to your three loyal readers, eh?
EVO
How do I know my 3 loyal readers (that's up 1 from last count) aren't scrapers themselves?
Your loyal readers are more likely to try and start a side-business with it than they are to scrape your content. You know I think there's money an effective, easy-to-install 'bot blocker. :)
Just tell me and I will blog about it! :)
Aaron
seobuzzbox
Post a Comment