tag:blogger.com,1999:blog-19248375.post7001146749274330353..comments2023-10-18T05:54:12.748-07:00Comments on IncrediBILL's Random Rants: Too Much FyberSpider In My Site's DietIncrediBILLhttp://www.blogger.com/profile/14244934627308399202noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-19248375.post-84709658776087144612007-07-08T14:19:00.000-07:002007-07-08T14:19:00.000-07:00You are welcome!1. Thanks for clarifying... I made...You are welcome!<BR/><BR/>1. Thanks for clarifying... I made the mistake of assuming it was doing something wrong after you said "assuming it actually honors robots.txt" and gave people info to help them block my bot. I see now that I didn't have to read so much into what you said, sorry.<BR/><BR/>2. You can't blame me for misunderstanding this one ;-) You only said it had a hard time finding Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19248375.post-31689850158351831942007-07-08T08:54:00.000-07:002007-07-08T08:54:00.000-07:00Nathan, thanks for dropping in but you didn't pay ...Nathan, thanks for dropping in but you didn't pay attention to the post...<BR/><BR/>1. The crawl bug was you used "Python-urllib/1.15" as the user agent when reading robots.txt, not FyberSpider as it should be set.<BR/><BR/>Didn't say you violated robots.txt standards, re-read a second time and maybe you'll get what I meant as your user agent setting was the python default, that's all.<BR/><BR/><IncrediBILLhttps://www.blogger.com/profile/14244934627308399202noreply@blogger.comtag:blogger.com,1999:blog-19248375.post-33445553141895014522007-07-08T05:59:00.000-07:002007-07-08T05:59:00.000-07:00Hi,My name is Nathan Enns from the search engine F...Hi,<BR/><BR/>My name is Nathan Enns from the search engine FyberSearch. I would like to address the two issues you raised in your post:<BR/><BR/>1. I do apologize for the crawler error you experienced. However, I did some tests after reading your post and things appear to be functioning correctly. I would like to solve whatever issue caused the malfunction on your site. If possible, could you Anonymousnoreply@blogger.com