Just for giggles I tweaked the parameters of the bot blocker a few weeks ago to see if I could snare any super slow crawlers trying to fly under the radar.
So far, the grand prize winner has been scraping a few pages daily for 13 days now for a grand total of 1567 pages.
What a loser, sheesh.
Unfortunately, it looks like this latest change has snared about 20-30 innocents so we'll be retooling it today, but I learned a lot about slow scrapers during this process and it was quite an eye opener.
Monday, May 08, 2006
Scrape-a-thon endurance test
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment