Some days you just wake up and BINGO! you have the best damn idea you ever had in a long time. That's right, something so simple and as plain as the nose on my face just slapped me upside the head today and I'm positive I can shut down bots masking as humans in 3 pages or less into a crawl.
However, it was real a bonus day for me as I came up with not one but TWO new techniques to add to my arsenal of bot blocking weapons. The only downside is both of these tricks require changes in the web pages in order to make it work, but it's well worth the trouble if it can stop bots dead in less than 3 pages.
We'll be doing some testing for a week or so to see if it's really as effective as I think it is and verify it's not snaring humans and let you all know how it works but I'm REALLY excited as this KICKS ASS so far!
Anyone know a good patent attorney that's reasonably priced? ;)
Wednesday, March 22, 2006
Stop that bot, 3 pages or LESS
Subscribe to:
Post Comments (Atom)
2 comments:
Patent Attorney? No. But I'd be damn interested in knowing how to do it mate.
Most of what I do can be done in .htaccess just to filter out the lamest of the lame.
However, you have to profile the rest of them in real time using a script to see how they're intereacting with your server. I've been profiling a variety of bots and techniques for months and some are trivial to stop, others take a little more finesse, but this much I can state with reasonable certainty, I don't think a single bot gets more than 20 pages tops and I'm going to crank down that limit and see if any screaming happens with my visitors.
'Nuff said :)
Post a Comment