How Much Nutch is TOO MUCH Nutch?

Not too long ago I set off a storm of comments when I called out the writer of nutch on the carpet claiming his creation was being used excessively and abusing my server all over the place.

I was told by the legions of nutchies out there that I sucked, was told to get off the public network, called everything from an idiot to a grumpy webmaster and worse. They all claimed that nutch was wonderful thing and made search engines that were beneficial and I should stop complaining, shut up and let them crawl.

Bullshit.

Being a patient man, I sat back and waited to collect enough data to show those nutchies that the usage of nutch is growing out of control and I really don't need 100+ unique IP addresses from everywhere from Turkey to Japan crawling my goddamn website.

Theoretically, if these 100 crawlers ask for my max of 40K+ pages each that's over 4 million pages served, assuming I let them have them in the first place, mostly for no purpose whatsoever.

Here's the list of the nutch plague seen on my site recently:

124.32.246.36 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

124.32.246.45 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

128.208.6.200 NutchCVS/0.7.1 (Nutch running at UW; http://crawlers.cs.washington.edu/; sycrawl@cs.washington.edu)

128.208.6.226 NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)

128.208.6.227 NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)

128.208.6.77 NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)

129.242.19.138 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

129.34.20.19 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

129.78.64.106 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

131.112.16.140 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

131.112.16.220 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

131.211.84.21 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

136.165.45.122 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

137.43.154.203 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

147.202.90.2 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

164.67.195.24 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

164.67.195.245 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

164.67.195.26 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

164.67.195.27 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

164.67.195.68 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

164.67.195.85 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

166.214.93.76 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

193.203.240.117 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

193.203.240.118 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

193.203.240.119 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

193.203.240.120 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

193.203.240.121 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

193.203.240.122 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

193.252.148.51 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

203.113.130.205 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

203.131.194.84 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

203.147.0.44 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

203.244.218.1 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

209.131.61.1 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

210.174.3.130 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

210.196.73.193 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

210.245.31.15 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

210.245.31.18 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

212.12.114.238 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

212.127.226.60 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

212.137.33.140 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

212.156.230.210 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

212.58.116.72 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

213.132.175.101 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

213.186.36.107 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

213.251.133.12 Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; nutch at misterbot.fr)

216.93.185.12 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

220.218.159.50 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

221.114.253.210 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

221.116.237.114 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

221.221.237.35 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

24.222.153.250 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

24.224.226.18 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

58.186.61.164 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

58.187.12.236 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

58.87.139.90 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

59.160.240.115 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

60.248.9.114 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

61.135.151.175 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

62.129.132.47 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

62.168.188.151 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

62.40.36.87 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

63.133.162.98 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

64.105.36.210 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

64.151.112.44 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

64.241.242.18 NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

64.242.88.10 NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

64.242.88.60 NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

64.34.172.78 BurstFind Crawler 1.0/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; crawler@burstfind.com)

64.34.180.167 Nokia6620/2.0 (4.22.1) SymbianOS/7.0s Series60/2.1 Profile/MIDP-2.0 Configuration/CLDC-1.0/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

64.38.10.26 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

64.71.164.103 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

64.71.164.107 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

64.71.164.108 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

64.71.164.125 Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com)

65.220.67.9 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

65.9.20.49 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

65.91.114.3 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

66.108.32.4 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

66.15.68.234 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

66.162.5.43 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

66.207.120.226 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

66.243.31.34 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

67.111.28.139 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

67.52.101.242 NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

68.205.124.164 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

68.205.127.94 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

69.248.26.83 Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

69.55.233.28 Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)

70.197.81.79 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

70.30.97.106 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

70.56.66.216 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

70.96.99.254 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

71.241.153.125 NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

71.35.163.79 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

72.0.207.162 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

72.2.25.67 NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

72.5.173.12 sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache.org/nutch/bot.html; researchbot@shopping.com)

72.51.37.148 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

81.203.142.109 NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

83.246.79.28 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

84.191.111.92 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

So we cranked all these IPs thru the reverse DNS grinder just to see who was running all this nutch without changing the default crawler strings. There are some that failed reverse DNS but I'm too lazy today to bother with WHOIS'ing that list so anyone that feels compelled to dig deeper feel free to post the results as a comment.

The reverse DNS of the IPs:

36.246.32.124.in-addr.arpa name = 124x32x246x36.ap124.ftth.ucom.ne.jp.
45.246.32.124.in-addr.arpa name = 124x32x246x45.ap124.ftth.ucom.ne.jp.
200.6.208.128.in-addr.arpa name = qbert.cs.washington.edu.
226.6.208.128.in-addr.arpa name = zork.cs.washington.edu.
227.6.208.128.in-addr.arpa name = nethack.cs.washington.edu.
77.6.208.128.in-addr.arpa name = pacman.cs.washington.edu.
138.19.242.129.in-addr.arpa name = vortex05.cs.uit.no.
19.20.34.129.in-addr.arpa name = yktgi01e0-s5.watson.ibm.com.
106.64.78.129.in-addr.arpa name = www-cacheC.usyd.edu.au.
140.16.112.131.in-addr.arpa name = ginga.ks.cs.titech.ac.jp.
220.16.112.131.in-addr.arpa name = endeavor.furui.cs.titech.ac.jp.
21.84.211.131.in-addr.arpa name = burum.labs.cs.uu.nl.
122.45.165.136.in-addr.arpa name = webmining.spd.louisville.edu.
203.154.43.137.in-addr.arpa name = dhcp-892b9acb.ucd.ie.
** server can't find 2.90.202.147.in-addr.arpa: SERVFAIL
24.195.67.164.in-addr.arpa name = cairo.ee.ucla.edu.
245.195.67.164.in-addr.arpa name = tacoma.ee.ucla.edu.
26.195.67.164.in-addr.arpa name = archer.ee.ucla.edu.
27.195.67.164.in-addr.arpa name = gutman.ee.ucla.edu.
68.195.67.164.in-addr.arpa name = treviso.ee.ucla.edu.
85.195.67.164.in-addr.arpa name = chandra.ee.ucla.edu.
** server can't find 76.93.214.166.in-addr.arpa: SERVFAIL
** server can't find 117.240.203.193.in-addr.arpa: NXDOMAIN
** server can't find 118.240.203.193.in-addr.arpa: NXDOMAIN
** server can't find 119.240.203.193.in-addr.arpa: NXDOMAIN
** server can't find 120.240.203.193.in-addr.arpa: NXDOMAIN
** server can't find 121.240.203.193.in-addr.arpa: NXDOMAIN
** server can't find 122.240.203.193.in-addr.arpa: NXDOMAIN
51.148.252.193.in-addr.arpa name = io.x-echo.com.
** server can't find 205.130.113.203.in-addr.arpa: NXDOMAIN
84.194.131.203.in-addr.arpa name = ldnatvip.livedoor.com.
44.0.147.203.in-addr.arpa name = proxy.ji-net.com.
1.218.244.203.in-addr.arpa name = u1.gpu42.samsung.co.kr.
1.61.131.209.in-addr.arpa name = nat1.burbank.corp.yahoo.com.
130.3.174.210.in-addr.arpa name = pae0382.tokyte00.ap.so-net.ne.jp.
193.73.196.210.in-addr.arpa canonical name = 193.192h.73.196.210.in-addr.arpa.
193.192h.73.196.210.in-addr.arpa name = aa2003080550002.userreverse.dion.ne.jp.
** server can't find 15.31.245.210.in-addr.arpa: NXDOMAIN
** server can't find 18.31.245.210.in-addr.arpa: NXDOMAIN
*** Can't find 238.114.12.212.in-addr.arpa.: No answer
60.226.127.212.in-addr.arpa name = 212-127-226-60.cable.quicknet.nl.
** server can't find 140.33.137.212.in-addr.arpa: NXDOMAIN
210.230.156.212.in-addr.arpa name = dsl.static212156230210.ttnet.net.tr.
** server can't find 72.116.58.212.in-addr.arpa: NXDOMAIN
101.175.132.213.in-addr.arpa name = webserver4.octrium.nl.
107.36.186.213.in-addr.arpa name = ns3490.ovh.net.
12.133.251.213.in-addr.arpa name = ns31793.ovh.net.
12.185.93.216.in-addr.arpa name = customer-reverse-entry.216.93.185.12.
50.159.218.220.in-addr.arpa name = 220x218x159x50.ap220.ftth.ucom.ne.jp.
210.253.114.221.in-addr.arpa name = 221x114x253x210.ap221.ftth.ucom.ne.jp.
114.237.116.221.in-addr.arpa name = 221x116x237x114.ap221.ftth.ucom.ne.jp.
** server can't find 35.237.221.221.in-addr.arpa: NXDOMAIN
250.153.222.24.in-addr.arpa name = blk-222-153-250.eastlink.ca.
18.226.224.24.in-addr.arpa name = blk-224-226-18.eastlink.ca.
164.61.186.58.in-addr.arpa name = 58-186-61-xxx-dynamic.hcm.fpt.vn.
236.12.187.58.in-addr.arpa name = adsl-dynamic-pool-xxx.fpt.vn.
90.139.87.58.in-addr.arpa name = p578b5a.tokyte00.ap.so-net.ne.jp.
115.240.160.59.in-addr.arpa name = 59.160.240.115.static.vsnl.net.in.
114.9.248.60.in-addr.arpa name = 60-248-9-114.HINET-IP.hinet.net.
** server can't find 175.151.135.61.in-addr.arpa: NXDOMAIN
47.132.129.62.in-addr.arpa name = HOSTED-BY.PBTECH.COM.
151.188.168.62.in-addr.arpa name = host-62-168-188-151.adsl.caucasus.net.
87.36.40.62.in-addr.arpa name = pythagoras.portal.o2.ie.
98.162.133.63.in-addr.arpa name = mail.visvo.com.
210.36.105.64.in-addr.arpa name = h-64-105-36-210.snvacaid.covad.net.
44.112.151.64.in-addr.arpa name = customer-reverse-entry.64.151.112.44.
18.242.241.64.in-addr.arpa name = sv-fw.looksmart.com.
10.88.242.64.in-addr.arpa name = sv-crawl.looksmart.com.
60.88.242.64.in-addr.arpa name = sv-crawlfw4.looksmart.com.
78.172.34.64.in-addr.arpa name = slimy.vhosting.com.
167.180.34.64.in-addr.arpa name = server7.springright.com.
26.10.38.64.in-addr.arpa name = server1.netsweeper.com.
** server can't find 103.164.71.64.in-addr.arpa: NXDOMAIN
** server can't find 107.164.71.64.in-addr.arpa: NXDOMAIN
** server can't find 108.164.71.64.in-addr.arpa: NXDOMAIN
** server can't find 125.164.71.64.in-addr.arpa: NXDOMAIN
9.67.220.65.in-addr.arpa name = mirror.setnine.com.
49.20.9.65.in-addr.arpa name = adsl-9-20-49.mia.bellsouth.net.
** server can't find 3.114.91.65.in-addr.arpa: NXDOMAIN
4.32.108.66.in-addr.arpa name = cpe-66-108-32-4.nyc.res.rr.com.
234.68.15.66.in-addr.arpa name = bdsl.66.15.68.234.gte.net.
43.5.162.66.in-addr.arpa name = 66-162-5-43.static.twtelecom.net.
226.120.207.66.in-addr.arpa name = firewall.net-sweeper.com.
** server can't find 34.31.243.66.in-addr.arpa: SERVFAIL
** server can't find 139.28.111.67.in-addr.arpa: SERVFAIL
242.101.52.67.in-addr.arpa name = rrcs-67-52-101-242.west.biz.rr.com.
164.124.205.68.in-addr.arpa name = 164.124.205.68.cfl.res.rr.com.
94.127.205.68.in-addr.arpa name = 94.127.205.68.cfl.res.rr.com.
83.26.248.69.in-addr.arpa name = c-69-248-26-83.hsd1.nj.comcast.net.
28.233.55.69.in-addr.arpa name = HVAR.SIMPY.com.
79.81.197.70.in-addr.arpa name = 79.sub-70-197-81.myvzw.com.
106.97.30.70.in-addr.arpa name = CPE00095b51f4c9-CM0013718d007c.cpe.net.cable.rogers.com.
216.66.56.70.in-addr.arpa name = 70-56-66-216.tukw.qwest.net.
** server can't find 254.99.96.70.in-addr.arpa: NXDOMAIN
125.153.241.71.in-addr.arpa name = pool-71-241-153-125.nycmny.fios.verizon.net.
79.163.35.71.in-addr.arpa name = 71-35-163-79.tukw.qwest.net.
162.207.0.72.in-addr.arpa canonical name = 162.160/27.207.0.72.in-addr.arpa.
162.160/27.207.0.72.in-addr.arpa name = link.enhancededge.com.
67.25.2.72.in-addr.arpa name = h72-2-25-67.bigpipeinc.com.
** server can't find 12.173.5.72.in-addr.arpa: NXDOMAIN
148.37.51.72.in-addr.arpa name = server1.properazzi.com.
109.142.203.81.in-addr.arpa name = 81-203-142-109.user.ono.com.
*** Can't find 28.79.246.83.in-addr.arpa.: No answer
92.111.191.84.in-addr.arpa name = p54BF6F5C.dip.t-dialin.net.

Looks like just about everybody's running it from colleges to corporations and even Uncle Bob crawling the web from a dial-in, talk about slow, but where is the benefit for those of us being abused with it?

I'll admit that a few of the nutches actually resulted in search engines showing up online but who uses these search engines? Best I can tell, none of the actual 400K visitors/month to my site that's being crawled use any of these so-called search engines and probably never will.

Here's the problem, and maybe I'm just using nutch as an example because it's so easy to spot this virulent trend with a single source, but the amount of things attempting [they didn't succeed] to crawl my site daily would easily become a significant portion of my daily traffic if I let them all in which is insane.

What happens when this trend reaches it's natural conclusion?

Where it's heading in that the crawlers will soon exceed the actual visitors in terms of daily pages downloaded as more and more search engines, aggregators, and spybots come online looking for more ways to sell a slice of the internet to an ever increasing bunch of specialized niche markets. Not to mention we're still dealing with all the scrapers, link checkers and down right dumb things like refererrer checkers abusing our bandwidth.

It's out of control and someone needs to put the breaks on this nonsense.

Someday soon crawlers, with the exception of the big search engines, will need to ask permission to get on just about any website of scale, and will need to make a compelling argument why they should be allowed to index the site. The day of just taking what you want and doing what you want with it will surely come screeching to a halt as the burden of all this bandwidth usage starts to hit the hosting companies and trickles down to the webmasters.

Maybe the webmasters will fight back first and take control before it's too late.

Here's hoping.

RED ALERT - ECOMMERCE SITES PROTECT YOURSELF PRONTO!

There's a new company called Pronto that has a product in beta that not only crawls your site, displays message toasts while visitors are looking at your products via a browser plug-in.

For instance, someone is looking at the widget your online store is selling and suddenly a window pops up telling your visitors that they can get this widget cheaper elsewhere and tries to direct your shoppers away from your store.

Basically, this takes something like a Shopping.com-type service one step further by incorporating it into the browser and the potential harm to all the smaller online stores is enormous.

Anyone running any kind of ecommerce or affiliate site will definitely want to block this:

Here's the critical info on this crawler:

User Agent: "RedCarpet/1.3 (http://www.pronto.com/robots.html)"

Actual IP's used:

66.45.38.54
66.45.38.56
66.45.38.59
66.45.38.86
66.45.38.88
66.45.38.90
66.45.38.92
66.45.38.91
66.45.38.94
66.179.107.117
216.183.117.132
216.183.117.135

Complete blocks of allocated IP's:

RedCarpet, Inc. INFLOW-9359-113352-18374 (NET-66-45-38-80-1)
66.45.38.80 - 66.45.38.95

RedCarpet, Inc. INFLOW-9359-113352-19316 (NET-66-179-107-112-1)
66.179.107.112 - 66.179.107.127

RedCarpet, Inc. INFLOW-9359-113352-19482 (NET-216-183-117-128-1)
216.183.117.128 - 216.183.117.143

This is definitely a company no ecommerce site wants crawling unless you're sure you have the best prices so block 'em!

Bad Karma is a potential DDoS threat

Some guy has something he's put out as freeware called the Referrer Karma which gets the referring page and checks to see if it actually has a link to the site referred. If no link to your site exists on the referring page it slams the door on the visitor assuming it's a referrer spammer.

Two problems with that approach:

1. Links that pass thru redirect pages from directories directory sites will fail this test every time as the referrer is the redirect page itself, not a web page with links on it.
2. Sites that block bots, like mine, toss out error pagess when stupid user agents appear and VOILA! the visitor from my site gets bounced off by this stupid script.

Here's the info:

65.98.116.226
cp5.secserverpros.com.
"Referrer Karma/2.0"

Next, let's explore my concern with potential vulernabilities with Referrer Karma.

If you think about the implementation of Referrer Karma for a minute you'll realize it would allow one kiddie script to potentially pull off a DDoS attack. This could be accomplished by issuing thousands of requests to a bunch of sites running this Referrer Karma and each request containing a faked referrer to the target site you're attacking.

You wouldn't need to wait for the page request to complete, just send out a ton of requests to a bunch of servers and terminate the socket when the websever respondes with data is ready. No need to download the resulting page as Referrer Karma has already done your dirty deed for you by hitting the other site asking for the requested page.

Ask for a few thousand pages in a few seconds from a a bunch of sites using Referrer Karma and step back and watch the fun as the target server melts.

Lack of Intelligence Competence Crawler

Well here's yet another site called the Intelligence Competence Center trying to crawl the web looking for things they can sell they to various industries.

Here's the crawler details:

212.227.103.133
s15208971.onlinehome-server.info.
"iCCrawler (http://www.iccenter.net/bot.htm)"

also...

82.165.39.218
p15197600.pureserver.info
"iCCrawler (http://www.iccenter.net/bot.htm)"

all IPs it's used with my site...

212.227.103.133
212.227.93.221
82.165.39.218

I'm really getting sick and tired of these fucking corporate leeches that keep crawling [pun intended] out of the woodwork.

HK Creepy Crawler

Been seeing this same bot "Java/1.4.1_04" asking for the same pages from the following IPs multiple times:

210.177.215.25
210.177.215.28
210.177.215.29

Don't know if it's a hosting account or ISP, but it's worth keeping an eye on these IPs.

My sites doesn't need UPDATED!

How many comparison shopping sites can one planet possibly need?

Well, whatever your answer was, it's wrong, there's one more called Updated crawling around.

The user agent info:

38.119.96.110 "updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)"

Block it, don't block it, I don't care...

Google Dance Makes You Shit Your Pants

What a day.

Got up this morning and my main sites traffic was wonky so I took a look around and Google traffic seemed to be askew. Checked a few data centers and my position was all over the map. Up and down, round and round, Google's chewing me up and spitting me out in all sorts of wacky places.

My default datacenter now shows me back in the top 10 but only God and Google knows where it'll be tomorrow and neither of them are talking.

What a mess, Google better have a new liver standing by for me...

Anonymous Media, just can't make this shit up

Here comes another scaper with a mission called Anonymous Media and they want your shit.

63.133.162.98 - "GET /robots.txt HTTP/1.0" 200 146 "-" "Anonymous/0.0 (Anonymous; http://www.anonymous.com; noreply@anonymous.com)"

Right on their website it says they:

• SPY "anonymous watches what consumers watch"
• SCRAPE "anonymous compiles data from multiple sources"
• PROFIT "anonymous generates market reports and analysis"
• TARGETS YOU "anonymous conducts custom market research"

OK, is it just me or are we all getting SICK OF THIS SPYBOT SHIT?

Maybe we should just install 301 redirects to other spybot companies when spybots come crawling it points to a competitor and just let them crawl each other's sites to death.

RED ALERT UPDATE - SnapBot and the Linux Firefox Revelation

Finally, after weeks of these idiots bombarding my servers with Linux Firefox for unknown reasons I finally connected all the dots as SnapBot is apparently running from 3 data centers that I know about and all 3 have run both the SnapBot crawler and Firefox from the same IPs.

65.38.102.0/24 "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1"

38.98.19.0/24 "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1"

66.234.139.0/24 "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1"

You know what SnapBot was doing from these 3 locations?

Making goddamn screen shots of every fucking webpage, or trying to anyway, all 40K+ of them too!

Fuckers.

I know this for a fact as I found a picture of my error message about their site being banned with their IP number embedded in it from the 65.38.102.0/24 block and the others are already known Snap IPs.

Listen assholes, take a picture of the home page and leave it at that, you aren't getting 40K screenshots and anyone that thinks they are can blow that shit out their ass.

Don't Worio, just another search engine

Something new hit my site claiming to be Worio and planning to go live in 2006 according to their website. Another site using Heritrix to crawl with, oh joy, almost becoming as annoying as the new-nutch-site-of-the-week club.

BAD_AGENT: 198.162.51.70 [worbo2.cs.ubc.ca.] requested 1 pages as "Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.worio.com/)"

What makes Worio special is it's supposed to be created for computer scientists and programmers.

OK, there goes my lunch.

Don't Worio, Be Happy.

REJECT SPAM, DO NOT BOUNCE!

I've been setting my email pref's on all my servers to REJECT unknown addresses instead of BOUNCE which opens your server up to all sorts of spam problems as your email queue fills up with undeliverable garbage and your server starts to choke.

Thought I was pretty clean but I imported some domains from another server and suddently started seeing 150+ emails piled up ever day. They were all failure notices just sitting there chewing up my box trying to deliver all day long.

Got sick of that shit last weekend and did a complete email audit and now ALL DOMAINS are set to REJECT unknown addresses.

Guess how many outbound emails are pending now?

ZERO!

Been zero since I did it and holding strong so spammers using randomname@mydomain.com are no longer bothering me whatsoever.

Just a reminder, check your email accounts and make sure you're locked up tight!

Block them PRONTO

Roll out the RedCarpet and let them scrape all they want.... NOT!

We found this little beast bothering us:

BAD_AGENT: 66.45.38.59 [unknown] as "RedCarpet/1.3 (http://www.pronto.com/robots.html)"

On their web site it says:

We use our patent-pending software [crawler] to scour the web [scrape] and identify the most merchant sites and products possible for you.

What a bunch of happy horseshit, web crawlers are now patent-pending software?

Go take a long walk of a short pier and jump into lake go-fuck-yourself.

PlanetLab comes knocking

Here we go again with what looks like someone testing my PlanetLab barriers to see which proxies do or don't work and this just started a few minutes ago and will probably get more interesting in an hour or two.

Note the same browser from each location:

BANNED: 128.4.36.12 [planetlab2.pc.cis.udel.edu.] requested 1 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
BANNED: 129.10.120.111 [planetlabone.ccs.neu.edu.] requested 1 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
BANNED: 155.98.35.2 [planetlab1.flux.utah.edu.] requested 1 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

This PlanetLab bullshit needs the plug pulled on it ASAP.

Google Gags on Drop Down Menu

This was an amusing find in my log file:

66.249.72.171 - "GET /%20Select%20An%20Item%20 HTTP/1.1" 404 "-" "Mediapartners-Google/2.1"

Wouldn't you think all those high-priced Google employees would know better?

RED ALERT #2 UPDATE - It's SNAP snapping images

Recently I issued a RED ALERT on BBCOM and it appears it's Snap making screen shots.

I'm positive it's Snap as one of those screen shots has my firewall message displaying the IP address of 65.38.102.146 and the user agent "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1"

Too fucking bad Snap, I'm still blocking your dumb ass until you put your ID on the User Agent from those IPs.

No wonder people think their conversion rates are declining when fucking search engines are even skewing the numbers will bullshit like this unprofessional horseshit.

Charlotte, what a web you weave

In the BIG SHOCKER of the day department, when I installed my prototype bot blocker in another of my own websites today to expand my data stream wouldn't you know it but it caught an active spider that was crawling many hundreds of pages the second it was enabled.

This crazy assed spider it still going strong, or thinks it is, well over an hour after stopping it from getting real data.

The spider is Charlotte, with these specs:

209.249.86.4 "Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com)"

Never heard of Charlotte before so I took a peek to see what others might know about it and stumbled into the most hysterical thread I've ever seen in the OsCommerce Forums about all these store owners frantically chasing down spider names and dropping spider names it into something called their spiders.txt file or some shit.

Dudes, blocking by user agent is so 1990's, you'll stress out and become incontinent doing it that way. What a waste of time, wish I was ready to help you all already, but such is life and quality takes time.

Blog Spam is not a problem

That's right, blog spam isn't the problem whatsoever and technically it shouldn't even exist if it wasn't for the sloppy work of the idiot programmers that write blog software. Anyone leaving the comment forms wide open so that any script kiddie could abuse it should have their programming license revoked.

Now there are even solutions springing up to monitor and stamp out blog spam and it's fucking ridiculous or would hysterical be the proper word?

For example, their feature claims:

AntiSpam Deluxe.
Get rid of referrer and comment spam thanks to a community maintained spam database.

Community maintained database, you must be kidding?

Can you people say CAPTCHA?

I know you can, put a captcha on the comments and registration page to stop automation.

The part that just kills me is whoever was naive enough to put "registration" as the only means of defense without captchas in blogs thinking that making people register to post would stop spam because bots don't use cookies or register. I've noticed a few people resorting to registration alone as an anti-spam technique lately and that simply won't cut it. Wrong again, bots use cookies, so you better put a captcha on your registration page otherwise the bots just register as some random name and post anyway.

Now address manual spamming as follows:

• Filter out all posts that contain embedded HTML and URLs from all first time posters, just bounce any post that looks like spam.

• Don't hyper-link names of posters to websites until they become a trusted poster or register.

I do these exact techniques on some of my websites (not this website, except the captcha) and the spammers simply went away. There was nothing they could do of value unless they wanted to just vandalize out of spite, and most are interested in making money so once I removed their incentive they stopped coming back.

Can't post a link to your website?

Oh boo-fucking-hoo, then your spam won't drive visitors, so go the fuck away.

Wow, that was easy wasn't it.

• No community database
• No anti-spam service
• No bullshit
• No spam

Now you're probably going to start yelling that my concept breaks the premise of blogs in that it's all about the community and the linking and all that other happy horseshit.

Nope, you can still link out, just not on your first damn post, or maybe your first 10 posts. The easiest way to build up "post count" would be using automation which is blocked again, that's where the captcha's come back into play.

If you really want to piss off the spammers, require both registration AND a captcha.

Now go fix your crappy blog software, download a captcha and install it, and stop whining about spam ya bunch of cry babies.

RED ALERT #5 - Everyones Scraping Internet

There seems to be both legitimate and questionable activity from Everyones Internet so it's been scrutinized a bunch before deciding to issue a warning about this mess.

Basically, to see what was coming out of ev1.net there were a couple of filters installed to see what was coming from them and we got several possible legit bots, some proxy servers, and some wacky crap.

You'll note Chitika listed which is definitely using part of that block:

Everyones Internet EVRY-BLK-15 (NET-67-15-0-0-1)
67.15.0.0 - 67.15.255.255
Chitika, Inc EVRY-398 (NET-67-15-219-0-1)
67.15.219.0 - 67.15.219.63

The linksmanager.com looks possibly legit based on reverse dns (linkchecker02.linksmanager.com), but I think picsearch.com (217.212.245.198) is potentially bogus.

On linksmanager website it says:

LinksManager.com runs an automated Reciprocal Link Checker and Dead Link Checker (User-agent: linksmanager and User-agent: linksmanager_bot ) for all LinksManager customer's web sites. If you are linking with a website that is powered by LinksManager.com, you might see LinksManager.com/linkchecker.html listed in your server log reports.

However, in the log file I see this:

67.15.16.30 Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)

So is linksmanager being spoofed, guilty of sloppy outdated documentation or all of the above?

Here's a sample of what I'm seeing:

67.15.0.24
67.15.0.89 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
67.15.119.25 User-Agent: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
67.15.126.25 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
67.15.136.199 psbot/0.1 (+http://www.picsearch.com/bot.html)
67.15.138.14 PyQuery / 0.1
67.15.14.5
67.15.143.22 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
67.15.16.30 Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)
67.15.182.4 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
67.15.184.3 HTTP/1.0
67.15.184.41 HTTP/1.0
67.15.189.16
67.15.191.19
67.15.2.67 WordPress/1.5.2 PHP/4.4.1
67.15.219.10 Chitika ContentHit 1.0
67.15.219.11 Chitika ContentHit 1.0
67.15.219.12 Chitika ContentHit 1.0
67.15.219.14 Chitika ContentHit 1.0
67.15.219.15 Chitika ContentHit 1.0
67.15.219.16 Chitika ContentHit 1.0
67.15.219.17 Chitika ContentHit 1.0
67.15.219.18 Chitika ContentHit 1.0
67.15.219.3 Chitika ContentHit 1.0
67.15.219.9 Chitika ContentHit 1.0
67.15.221.2 ia_archiver
67.15.221.26 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
67.15.232.3 User-Agent: Mozilla/5.0 (; U;; en-US; rv:1.7.10) Gecko/20050716 Firefox/1.0.6
67.15.35.26 HTTP/1.0
67.15.38.27 Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
67.15.56.4 Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)
67.15.6.64 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
67.15.76.148 Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412.6.2 (KHTML, like Gecko) Safari/412.2.2
67.15.77.119 Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412.6.2 (KHTML, like Gecko) Safari/412.2.2
67.15.77.223 Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412.6.2 (KHTML, like Gecko) Safari/412.2.2
67.15.78.93
67.15.8.2 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)

Did you note the various Netscape/7.1's in the list?

That was yours truly using an outdated Netscape browser very few use to spot proxy servers when I'm testing lists of proxies. Some of the smarter ones mask the browser which makes it's a little more difficult, but then I just check a special page that has never been indexed that only I know about so there is NO hiding from my prying eye.

OK, now you know a new trick, are you happy yet?

Anyway, we cranked this small list thru the reverse DNS meat grinder and here's the results:

24.0.15.67.in-addr.arpa name = hu-tethys.com.
89.0.15.67.in-addr.arpa name = cpanel.masgrafx.com.
25.119.15.67.in-addr.arpa name = ev1s-67-15-119-25.ev1servers.net.
25.126.15.67.in-addr.arpa name = ev1s-67-15-126-25.ev1servers.net.
199.136.15.67.in-addr.arpa name = ev1s-67-15-136-199.ev1servers.net.
14.138.15.67.in-addr.arpa name = ev1s-67-15-138-14.ev1servers.net.
5.14.15.67.in-addr.arpa name = ev1s-67-15-14-5.ev1servers.net.
22.143.15.67.in-addr.arpa name = ev1s-67-15-143-22.ev1servers.net.
30.16.15.67.in-addr.arpa name = linkchecker01.linksmanager.com.
4.182.15.67.in-addr.arpa name = ev1s-67-15-182-4.ev1servers.net.
3.184.15.67.in-addr.arpa canonical name = 3.184.15.67.in-addr.ev1.opticaljungle.com.
3.184.15.67.in-addr.ev1.opticaljungle.com name = lhb-us-b-1.mailhostingserver.com.
41.184.15.67.in-addr.arpa canonical name = 41.184.15.67.in-addr.ev1.opticaljungle.com.
41.184.15.67.in-addr.ev1.opticaljungle.com name = lhb-us-b-2.mailhostingserver.com.
16.189.15.67.in-addr.arpa name = jnchost.net.
19.191.15.67.in-addr.arpa name = ev1s-67-15-191-19.ev1servers.net.
67.2.15.67.in-addr.arpa name = ev1s-67-15-2-67.ev1servers.net.
10.219.15.67.in-addr.arpa name = ev1s-67-15-219-10.ev1servers.net.
11.219.15.67.in-addr.arpa name = ev1s-67-15-219-11.ev1servers.net.
12.219.15.67.in-addr.arpa name = ev1s-67-15-219-12.ev1servers.net.
14.219.15.67.in-addr.arpa name = ev1s-67-15-219-14.ev1servers.net.
15.219.15.67.in-addr.arpa name = ev1s-67-15-219-15.ev1servers.net.
16.219.15.67.in-addr.arpa name = ev1s-67-15-219-16.ev1servers.net.
17.219.15.67.in-addr.arpa name = ev1s-67-15-219-17.ev1servers.net.
18.219.15.67.in-addr.arpa name = ev1s-67-15-219-18.ev1servers.net.
3.219.15.67.in-addr.arpa name = ev1s-67-15-219-3.ev1servers.net.
9.219.15.67.in-addr.arpa name = ev1s-67-15-219-9.ev1servers.net.
2.221.15.67.in-addr.arpa name = arcadehub.com.
26.221.15.67.in-addr.arpa name = ev1s-67-15-221-26.ev1servers.net.
3.232.15.67.in-addr.arpa name = assista.com.
26.35.15.67.in-addr.arpa canonical name = 26.35.15.67.in-addr.ev1.opticaljungle.com.
26.35.15.67.in-addr.ev1.opticaljungle.com name = 67-15-35-26.opticaljungle.com.
27.38.15.67.in-addr.arpa name = web.ir.cx.
4.56.15.67.in-addr.arpa name = linkchecker02.linksmanager.com.
64.6.15.67.in-addr.arpa name = ev1s-67-15-6-64.ev1servers.net.
148.76.15.67.in-addr.arpa name = 67.15.76.148.
119.77.15.67.in-addr.arpa name = 67.15.77.119.
223.77.15.67.in-addr.arpa name = 67.15.77.223.
93.78.15.67.in-addr.arpa name = mail.aucoffre.com.
2.8.15.67.in-addr.arpa name = mail.caromhosting.com.

Since it's a mixed bag and I haven't really decided what to do with this mess yet I'm just blocking anything that responds to reverse DNS as ".ev1servers.net" and taking all the other accesses in this range on a case by case basis.

What a freak'n mess, oh my freak'n head...

Jeteye should Jetpak it up, not ready for primetime

Here we go again with another new web service JetEye that allows you to pack up your links and things you find in some nonsense called a Jetpak and share it with people.

71.5.15.254 [firewall.jeteye.com.] "jeteyebot/0.1; http://www.jeteye.com/bot.html"

I'll give them this much credit, at least they looked in robots.txt before hitting the website.

But I think I'm giving them too much credit for robots.txt, you'll find out at the bottom...

I don't even know what to say about Jeteye as they appear just to be passing around a bunch of links to stuff, except somewhere down the road when a page is moved or changed there will be a flood of 404's with outdated Jetpaks, oh joy.

Who am I kidding, it won't be down the road, I got some 404s with their current Jetpaks, let the bullshit begin!

We give Jeteye a try!

Maybe because it was late Friday afternoon and I was bored shitless, who knows, but I downloaded and installed the damn thing just for shits and giggles.

When I first open Jeteye and click the "sign up here" link the first thing it does is overwrite the current tab in Firefox with their stupid sign-up page.

Opening a new tab must've been too hard for them but I digress...

I fill out the form like the wannabe bleeding-edge netizen that I am and click SUBMIT and it kicks back telling me my verification code failed. OK, would it fucking kill you to tell us in the text above the verification box that the code is case sensitve? Some captchas are sensitive, some aren't, but it's fucking nice to know which is which.

Submit with the new verification code and it kicks back again with something about defining a fucking password. Listen assholes, I typed in a password the FIRST time I filled out the form and you discarded it when you bounced my verification code because I didn't capitalize the fucking "Q". You're starting to get me pissed off already with this bullshit form but I'm trying to remain calm and open minded.

Luckily for them the email validation was swift and went off without a hitch or I'd be going off on a rant about now, so we save the ranting for later.

Jeteye is starting to get on my nerves as clicking on anything in Jeteye keeps zapping my current window. Almost 20 tabs open and which tab is currently being viewed gets blasted when I click on something in Jeteye. They better work on that as I'm about to scream with that behavior, they need some options or some shit, maybe just look and reuse the tab already opened labelled "Jeteye" but that would make too much sense wouldn't it?

Figured this mess out, usability can be a bit challenging and I have a few more complaints but I'm bored writing about this as it's just not quite ready for primetime yet and they aren't paying me to QA the goddamn product. However, I've managed to build my first little Jetpak while watching the request for robots.txt from their site hit my server at almost every action which is pissing me off immensely. Cache the robots.txt for a while, geez, give me a break.

Now the fun stuff:

Just for fun I drop them in the robots.txt file like they say to do to see what happens.

User-agent: jeteyebot
Disallow: /

Well, it did FUCKING NOTHING!

Every link I drop into a Jetpak from my site still shows up in the Jetpak so I assume it means they won't crawl my site but I'm still forced to be an unwilling participant in these goddamn Jetpaks. They still hit the server reading the robots.txt file for every stinking link and it looks like maybe the page I linked as well. It certainly doesn't tell the end user "piss off, this site doesn't want to play with Jetpak" - it just keeps on accessing my site like nothing changed.

Let get this straight, it's MY FUCKING WEBSITE and if I don't want MY LINKS to be in JETPAKS you better give me a way to STOP THIS SHIT!

Guess what else?

It saves direct links to GOOGLESYNDICATION!

Holy Shit! You can save AdSense links in a Jetpak?!?

What a concept, load up a Jetpak full of CPC and Affiliate links and start your click ring Jetpak!

You can drop in some images from any old page that show up stand alone in a Jetpak without attribution to the author. Oh sure, it shows the page of origin but we're not naive and we know that even with copyright notices on a page staring people in the face they steal shit anyway. However, with images in a Jetpak taken out of context they are more prone to get the right mouse "save as..." without even a second thought, especially without any warning about the image being copyrighted or anything.

Taking a little liberty with a phrase from some movie critics I'm giving Jeteye "2 thumbs up"

.... "2 thumbs up the ass" that is, as this shit sucks!

Time to whip up a couple of rewrite rules to block their IP 71.5.15.254 and referer "http://www.jeteye.com/jetpak/" in .htaccess before this gets out of control.

Locating and Blocking Proxy Servers

Since some of my readers want to know how I'm doing it, here's a few tips on how you too can eliminate the anonymous proxies from your site. Probably won't get them all and you might get a few false positives as well but it's better to have some defense against this menace than none at all.

A large number of these proxy servers are on .EDU domains because of all the bleeding heart crap about making information free for all without censorship and being able to surf without fear of retribution. That's a very noble and altruistic motive but you open the doors for competitve spying, scrapers, phishing theives and a lot more so don't take this the wrong way when I don't appreciate what you're doing with our tax and college dollars and send out a big "FUCK YOU" to establishments of higher learning that permit this bullshit. If people in other countries don't like being censored, let them overthrow their fucking government, it's not our problem and my server and copyrighted content shouldn't be vulnerable to attack because of the gaping holes opened up by your bleeding heart asses, but I'm off on a tangent.

The other groups of asshole proxies are the many web-based CGI and PHP proxy servers (like eatmoreblueberries) being used to bypass restricted internet access imposed on corporate, library and school networks. Well I'm sorry but you're supposed to be WORKING or STUDYING so let me give you a big "FUCK YOU" as well. Not only do they download your pages, they strip out YOUR ads and insert their OWN ads, assholes. So for all you slackers using those proxies, zip it up, close the porn sites, go back to work, and get a life you little fuckers as MySpace isn't it.

So, with a bit of ranting aside, back to blocking proxies...

New proxy servers pop up every 5 seconds so my method requires multiple techniques:

1. Import lists of known proxies and block them
2. Look for proxy environment variables
3. Test the IP for typical proxy ports and see if it works
4. Check for a port number being appended to your domain done by lame proxies
5. Monitor for proxy crawl thru of known services
   

1. Import Lists

This step is pretty obvious and can be automated by downloading the lists from a few well known proxy list sites, or if you're lazy you can subscribe to a service or two already doing that.

Probably doesn't hurt to validate these proxies, which can be done automatically, otherwise your list will grow infinitely as they appear and disappear very quicky

2. Proxy Environment Variables

You can check for the following:
HTTP_VIA
HTTP_X_FORWARDED_FOR
HTTP_PROXY_CONNECTION

Yes, those will tell you a proxy made the request but remember that AOL and many others are also a proxy so then it becomes more complicated as you have to evolve a list of known good proxies vs. all the rest and do further processing on those you don't know.

FYI, the really good anonymous proxies don't send that information so you'll never know it's a proxy.

3. Test for Proxy Ports

It will look simple but it's way more complicated to get right.

in PHP you can check to see if you can open port 80 on the incoming IP to see if it's an open proxy like this:

$fp = @fsockopen($theIP, 80, $errno, $errstr, 5);
if ($fp) {
// OPEN PORT
}

But that's very simplistic as most don't use :80, they use port :8080 and other weird #s like :3128, to avoid what the admins are currently blocking.

Not to mention, some proxies are very slow so you want to do exhaustive testing on post-page processing so you don't slow down the user experience on the front end of the page. You only have to do this once per IP, but someone could think your website is down if the process takes to long and worse case you get a positive answer that it's a proxy they've only accessed one page and you block the next page.

Once you detect the proxy add it to your proxy lists built in step 1 above and you'll never have to worry about this one again.

Remember, you may end up blocking IPs from colleges and universities but remember our alma mater, good old FU.

4. Port Numbers Appended to Domain

The dumbest of the dumb append a port number to your domain name which is easy to test in the HTTP_HOST variable. The only exceptions I've have to make to this rule so far is for the poor dumb bastards still using prodigy.net.mx which astonished me that prodigy still existed even as a name on a block of IPs!

5. Proxy Crawl Thru

What some of these dumb fuck proxy operators do is set up a cloaked directory, probably a clone of DMOZ or some shit, and cloak this directory to the search engines.

When you see things like Googlebot, Mediabot, Msnbot, etc. hitting your servers outside of their known range of IP's it means only 1 of 2 possibilities.

1. Someone is trying to spoof the user agent to get onto your server
2. The crawler is coming thru a proxy port

The best defense is to throw up an error or something in this case as I've had some pages hijacked by this nonsense so you really don't want to serve up real pages in this event as the search engines simply aren't that smart.

BTW, before serving up an error message, it's wise to do a reverse DNS lookup to make sure that Googlebot really isn't on a new block of IP's owned by google.com.

Summary

Probably not as simple as you had hoped but a couple of techniques are very straight forward and stop some level of the proxy nonsense without fear of blocking innocents.

Good luck trying this and may all your proxy requests bounce off your server like a rock skipping across a pond.