tag:blogger.com,1999:blog-19248375.post3653403572784122790..comments2023-10-18T05:54:12.748-07:00Comments on IncrediBILL's Random Rants: How to Block SpyderMate SEO ToolIncrediBILLhttp://www.blogger.com/profile/14244934627308399202noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-19248375.post-35092127308818397782009-01-04T02:25:00.000-08:002009-01-04T02:25:00.000-08:00Perhaps because embroidered pillows are a plausibl...Perhaps because embroidered pillows are a plausible wedding gift?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19248375.post-41038197832642216542008-12-11T12:32:00.000-08:002008-12-11T12:32:00.000-08:00Bill, do you know of a spider called WikiBot 1.0 ?...Bill, do you know of a spider called WikiBot 1.0 ?? It has crawled my site (about 10 pages) twice in the last week. User Agent says <A HREF="http://www.wedwiki.com/robot.htm" REL="nofollow">wedwiki.com/robot.htm</A> and this claims to crawl only wedding related sites (claims to be a new search engine and wiki) but my site sells embroidered pillows, not wedding stuff. Is this something to be PillowsGalorehttps://www.blogger.com/profile/09078544079538019112noreply@blogger.comtag:blogger.com,1999:blog-19248375.post-69474699338744875812008-12-08T16:04:00.000-08:002008-12-08T16:04:00.000-08:00Andy,Your software would have to first ask for rob...Andy,<BR/><BR/>Your software would have to first ask for robots.txt before you can honor it.<BR/><BR/>One of your customers attempted to crawl then I tried it from your demo version, no robots.txt request recorded from either IP address.<BR/><BR/>Want to try again?IncrediBILLhttps://www.blogger.com/profile/14244934627308399202noreply@blogger.comtag:blogger.com,1999:blog-19248375.post-53057484497214078062008-12-08T10:31:00.000-08:002008-12-08T10:31:00.000-08:00Hey IncrediBILL. Just so you know, SpyderMate resp...Hey IncrediBILL. Just so you know, SpyderMate respects robots.txt commands.<BR/><BR/>What other measures should we take to make sure webmasters such as yourself are not angered by our free software?Anonymousnoreply@blogger.com