Was looking over my log stats today for my site (not this blog) it was surprising to see Microsoft's Livebot crawling very aggressively, possibly taking more pages and returning faster then Googlebot this month. I'll give Livebot this much, the volume of crawling is pretty impressive compared to what it used to be so it looks like Microsoft is in this horse race to win and not just play 3rd fiddle.
Then of course we have our pals over at Ask slowly poking around my site. If Yahoo's Slurp is taking a nap this month, Ask must be in a coma or something. Slow as a snail and information seems to take forever to show up on their site. Maybe they cater to the dial-up crowd.
Then good old Gigabot seems to be crawling from a new bank of IPs and I don't even care.
Anyway, it will be interesting to see how quickly some of my new site changes show up in Live because crawling that fast and not updating as quickly would be silly!
Monday, March 26, 2007
Livebot vs. Googlebot - Microsoft trying to catch up?
Subscribe to:
Post Comments (Atom)
5 comments:
How do you avoid blocking Googlebot if they change their IP addresses without notice?
You do reverse DNS and then a forward DNS, per Google's instructions, to verify it's on googlebot.com
However, you still need to keep a list of Google IP ranges for other things such as the AdWords quality bot which don't support the reverse DNS method.
"Ask must be in a coma or something."
LOLL!
and yup, livebot it's crawling to aggressive now
11/7/07 Livebot has crashed our dedicated server and several others in our webhosts rack -
webhost is blocking Livebot as this is written.
Funny thing. I've added a robots.txt file to disallow all search bots and crawlers from visiting my site.
Apparently, the Microsoft bot has somehow found a way to circumvent that.
Post a Comment