Trapped another SEO tool called SpyderMate that crawls your site analyzing data. 208.78.98.236 [208-78-98-236.slicehost.net.]
Nothing wrong with using it but don't let competitors get a free ride analyzing your site.
The spider details:
"MentorMate Spider"
Their host:
OrgName: Slicehost LLC
OrgID: SLICE
NetRange: 208.78.96.0 - 208.78.103.255
CIDR: 208.78.96.0/21
They just keep coming and eventually they'll run out of data centers we haven't blocked!
4 comments:
Hey IncrediBILL. Just so you know, SpyderMate respects robots.txt commands.
What other measures should we take to make sure webmasters such as yourself are not angered by our free software?
Andy,
Your software would have to first ask for robots.txt before you can honor it.
One of your customers attempted to crawl then I tried it from your demo version, no robots.txt request recorded from either IP address.
Want to try again?
Bill, do you know of a spider called WikiBot 1.0 ?? It has crawled my site (about 10 pages) twice in the last week. User Agent says wedwiki.com/robot.htm and this claims to crawl only wedding related sites (claims to be a new search engine and wiki) but my site sells embroidered pillows, not wedding stuff. Is this something to be concerened about or block? Everytime it comes it has a different IP but it always starts with 64.34.*.*
Perhaps because embroidered pillows are a plausible wedding gift?
Post a Comment