Not sure what's going on with some of the scrapers exactly but it seems my "CHALLENGE" page, which has about 5 links on the page, is sending them in circles asking for the same pages over and over.
Lately I'm seeing a lot of lost scraper bots doing something like this:
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
... and so on and so forth...
Sometimes they'll ask for hundreds of pages like this before breaking out of that loop.
Quite amusing that I seem to have put a kink in their gears.
Friday, May 12, 2006
Circle Jerk Scraping
Posted by IncrediBILL at 5/12/2006 12:57:00 PM
Labels: Scrapers
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment