Some semantic search thing called TextDigger stumbled into my spider trap today.
I have nothing against semantic search, I'm not an anti-semantite (that's not the word you think it is, read it twice, i made it up just to be punny), but I'm definitely anti-stealth crawler.
According to the bot blocker, TextDigger requested 136 pages after being challenged while using the following user agent:
188.8.131.52 [nat1.textdigger.com]Here's their IP range:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
TextDigger MFN-B849-64-124-138-160-28 (NET-64-124-138-160-1)Not sure if what hit my server was their actual main crawler or not, but they aren't gaining any brownie points with me crawling in stealth for any reason.
184.108.40.206 - 220.127.116.11