Some semantic search thing called TextDigger stumbled into my spider trap today.
I have nothing against semantic search, I'm not an anti-semantite (that's not the word you think it is, read it twice, i made it up just to be punny), but I'm definitely anti-stealth crawler.
According to the bot blocker, TextDigger requested 136 pages after being challenged while using the following user agent:
18.104.22.168 [nat1.textdigger.com]Here's their IP range:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
TextDigger MFN-B849-64-124-138-160-28 (NET-64-124-138-160-1)Not sure if what hit my server was their actual main crawler or not, but they aren't gaining any brownie points with me crawling in stealth for any reason.
22.214.171.124 - 126.96.36.199