tag:blogger.com,1999:blog-19248375.post116546639900907206..comments2023-10-18T05:54:12.748-07:00Comments on IncrediBILL's Random Rants: TopicBlogs Steps Over The LineIncrediBILLhttp://www.blogger.com/profile/14244934627308399202noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-19248375.post-10639708789820854242007-07-01T12:50:00.000-07:002007-07-01T12:50:00.000-07:00Ummm, Jeff, I don't write about things crawling my...Ummm, Jeff, I don't write about things crawling my BLOG, I'm on blogger, have no control...<BR/><BR/>It's a big website elsewhere with 1M visitors and the point wasn't robots.txt, the point was why are you grabbing the full text when you should only be taking the RSS feed?IncrediBILLhttps://www.blogger.com/profile/14244934627308399202noreply@blogger.comtag:blogger.com,1999:blog-19248375.post-16186675834234017392007-07-01T01:05:00.000-07:002007-07-01T01:05:00.000-07:00Hey Bill,This is Jeff from topicblogs. I just thou...Hey Bill,<BR/><BR/>This is Jeff from topicblogs. I just thought I'd point out that our crawler adheres strictly to the robots exclusion protocol.<BR/><BR/>Your robots.txt file:<BR/><BR/>User-agent: *<BR/>Disallow:<BR/><BR/>does not prohibit our crawler (or, in fact, any crawler) from crawling your blog, but I'll be happy to remove it from our crawl list if you so wish.<BR/><BR/>Feel free to Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19248375.post-1165499685184021872006-12-07T05:54:00.000-08:002006-12-07T05:54:00.000-08:00Hey Bill -- thanks for reporting these. Have you ...Hey Bill -- thanks for reporting these. Have you ever thought of creating a mod_security living ruleset to block out these intrusions? I think it would be a nice idea. :)Anonymousnoreply@blogger.com