Friday, April 04, 2008

Persaibot - The Rude Crawler

I saw this little Persaibot hit my site today without even looking at robots.txt and their website has the balls to say:

Persai uses this bot to crawl the web. It's probably the nicest bot with the greatest personality in the world. Seriously, give it some attention.
Exactly how nice can a bot be that doesn't read robots.txt?

Did you read it and cache it some other day?

Doesn't matter, that was more than 24 hours ago, read it again.

I checked my logs from yesterday, it didn't read it then either and Persai hadn't visited my site in about a month before that.

I'm sorry, you have made the huge faux pas in robot rudeness.

Here's the intel I have on this little bot: []
"PersaiBot/2.1-dev3a (Persai web crawler;; bot at persai dot com)" []
"Mozilla/5.0 (compatible; Persaibot/2.71828183; +" []
"Mozilla/5.0 (compatible; Persaibot/2.71828183; +"
Now the true irony here is that the CEO of Persai posted on his blog complaining about another search engine called Spock scraping every little bit of data about him but at least Spock claims to honor robots.txt.

Must be a karma thing ;)

1 comment:

Anonymous said...

Amusingly, this sounds like the kind of hypocrisy that Ted Dziuba would bitch about on his blog.