Scrapers Don't Like Being Blocked

The last week has been getting more interesting as my banned scraper log file shows some rather interesting trends as they are squirming and thrashing trying to get around all the traps.

The most amusing is the ever changing user agent strings as they are definitely testing to see if I'm filtering based on specific user agent criteria and mostly they are right as everything is banned except http clients.

All of the legitimate search engines are being permitted based on their range of whitelisted IPs so trying to pretend to be Google, Teoma, Slurp, etc. will just instantly ban their IP for the day and repeated attempts might ban it permanently.

Almost as much fun as shooting fish in a barrel.