Call me confused, but why is MyFamily.com crawling my site?
Yup, the genealogy people, that's them.
66.43.16.199 [nat.myfamilyinc.com] has requested 110 pages as Mozilla/4.0 (compatible; MyFamilyBot/1.0; +http://www.myfamilyinc.com)Even claims to be them, looked at robots.txt like they were supposed to, yada yada.
Still went home with a pile of crap instead of my pages.
Put that on your family tree and smoke it.
3 comments:
You have to admit Bill, the fact that NOBODY knows who you are or what site you run gives you the freedom to say what you most likely would not say under other circumstances, no?
EVO
Maybe I would dial it back a little, but you never know, when I'm bent it hits the fan.
I have no clue about this site but so far I am enjoying the reading about the bots......I m start from the begining.
Post a Comment