Well, color me stunned shocked and appalled as I ran into an actual real live corporation with a legitimate product that is deploying a crawler that sets the user agent as MSIE ""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; ....)".
Yeah, that's right, forget robots.txt, forget letting you block them by normal user agent filtering means, they're getting into your website whether you like it or not because they have MANIFEST DESTINY!
They are ENTITLED TO YOUR CONTENT!
Not.
These lovely sneaky snoopers that boldly bypass your firewalling efforts are Lightspeed Technologies and they appear to be operating from this IP range 66.17.15.128 - 66.17.15.191.
Just block them now as this is about the lowest I've seen a corporate crawler get and they should be blocked on principle alone by not honoring internet standards.
Friday, March 17, 2006
Corporate Crawler Masking as MSIE
Subscribe to:
Post Comments (Atom)
1 comment:
Thanks for your post. I just started on my new site, and already those same fake browser things are all over the visitor log...? Strange stuff there.
Glad I know what it is now.
Post a Comment