Sunday, January 08, 2006

Bad Bots Trying Harder

Today some stupid bot came slow crawling like a human and masked itself as Internet Explorer but left telltale signs of a robot.

You ready for the best tell-tale?

Shockingly it looked at my robots.txt file as the first file of my site, probably trying to avoid my spider-traps.

Talk about stupid, you scrapers need a life as looking at robots.txt *IS* a spider-trap in itself!


Ooops, let a secret out ;)

No comments: