Sensis Web Crawler

Maybe I'm leading a sheltered life but I've never heard of the Australian based Sensis Web Crawler before this week when I noticed it slow crawling my biggest web site over a period of days now. Just before I almost blocked this bot I noticed that it seemed to be a legit service and Alexa claims is reasonably popular in their top 10,000 sites so I let it crawl.

Probably one of the best behaved but persistent little bots I've seen in a while.

Sensis has only crawled about 1300 pages in 24 hours which is about 54 pages an hour so Sensis will probably be on my site 30 days depending on how deeply they index based on their current rate of crawl.

This is much less abusive than Yahoo, MSN or Google that want whatever they want whenever they want and sometimes as fast as they can get it just to be competitive with each other.

Maybe the Big 3 can take a clue from this little player and be a bit more web server and bandwidth friendly. Then again, maybe Sensis just doesn't have the bandwidth and computing horsepower to beat the crap out of my server like the Big 3.

Good luck Sensis, will keep an eye on my results down under.