Monday, May 08, 2006

Scrape-a-thon endurance test

Just for giggles I tweaked the parameters of the bot blocker a few weeks ago to see if I could snare any super slow crawlers trying to fly under the radar.

So far, the grand prize winner has been scraping a few pages daily for 13 days now for a grand total of 1567 pages.

What a loser, sheesh.

Unfortunately, it looks like this latest change has snared about 20-30 innocents so we'll be retooling it today, but I learned a lot about slow scrapers during this process and it was quite an eye opener.

No comments: