EmeraldPantyShield

Nothing pisses me off more than some company like EmeralShield sending a bot that masks who they are when they request robots.txt files and then proceeds to crawl with the actual user agent name.

Look at this shit:

82.165.186.202 - "GET /robots.txt" "-" "-"

82.165.186.202 - "GET /" "EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx)"

I was curious who these dumb fucks were so I checked and found a thread on WebmasterWorld and then some bigger horseshit in their forum.

We also use the webbot with our web filter service. Customers visit sites that we don't know about and we use the webbot to go and dig the site. In this case we are looking primarily to filter porn for our customers. The site pages that are downloaded are fed into a scan engine that attempts to determine if the site is objectionable or not.

Well dig this, your customers can grow the fuck up and be adults about the 'net as you aren't digging my fucking website as I have too many little piss ants like your crawler all trying to get a piece of my website so you get ... NOTHING! NOT A SINGLE FUCKING PAGE!

Dig that?

I bet not.