Previous posts have mumbled about my new spider-trap anti-scraper tool that I added to my web site and I must say it's working so well I'm contemplating converting it into PHP so the masses can play with it.
Don't hold your breath as I'm fundamentally lazy.
The features are as follows:
- Fast crawl auto-block
- Slow crawl detection and optional blocking
- Spider trap auto-block
- Webmaster control panel that shows last 15 minutes of live activity by visitor
- Manual ban/block from the control panel
- Allowed spider pass thru with built in passes for Google, Yahoo and MSN
The final features in the visitor knowledge base will allow me to dynamically deploy the appropriate advertising model that each visitor can view when the second page is loaded, or theoretically allow me to start interjecting intermission pages to become a "subscriber".
So far so good, will keep everyone posted as to the effectiveness of the anti-scraper tools.