AJAX vs JSON, does anyone care?

Apparently someone cares as the Silicon Valley WebGuild is going to host a program about Yahoo! Web Services Using JSON by Douglas Crockford on June 14th and it'll be hosted at Google.

I might wander down there just to see what Douglas has to say on this topic not to mention they usually put out some appetizers or something at these events, don't know about Google, but when Microsoft used to host it the food was sometimes the best part of the evening!

Not that I condone abandoning XML, but I'll listen to his reasons.

Anyone else in the Bay Area interested in going?

Stupid Spammers Snared

OK, calling this spammer stupid gives stupid spammers a bad name.

How do I define stupid?

All of the attempted submissions below from "195.225.177.6" don't even have active domains so THAT's fucking STUPID!

I've never really had a problem with spammers before but last week these idiots started using automated submission to pump junk into my site and a couple of hours of programming later they're all blocked.

Here's the first few that hit today:

85.140.17.80 "Mozilla/4.0 (AllSubmitter)" "Business" http://business.alti.ru/
210.214.192.42 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)" "sports book for free" http://sportsbookusa.us
210.214.192.42 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)" "sports book for free" http://sportsbookusa.us
200.208.239.2 "Mozilla/4.0 (compatible; MSIE 5.01; MSNIA; Windows 98)" "Protonix order" http://stvincent.uzhgorod.ua/anal-masturbation-tip.html
195.225.177.6 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET" "" http://loversweekend.org/~ahmad_3139/files/hornyhousewives.html
195.225.177.6 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET" "" http://lovingvacation.org/~ahmad_3139/files/amateursex.html
195.225.177.6 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET" "" http://realromanticlove.org/~ahmad_3139/files/wifelovers.html
195.225.177.6 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET" "" http://romanticdirect.org/~ahmad_3139/files/amatuersex.html
195.225.177.6 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET" "" http://thefranticromantic.org/~ahmad_3139/files/amatuersex.html
195.225.177.6 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET" "" http://theromanticwoman.org/~ahmad_3139/files/amateurbus.html

If you didn't catch it, block "AllSubmitter" from allowed user agents and stop this fucknut tool from being used on your site.

So sad, too bad, no spam for you.

Linux Rulez, Scraper Droolz

Just couldn't help it, this user agent cracked me up:

195.70.35.179 [palatinus.sanomabp.hu.] requested 3 pages as "KummHttp/1.1 (compatible; KummClient; Linux rulez)"

What exactly is the KummClient?

A porn scraper?

Is it looking for bukkake sites?

Anonymous Proxies Out Cheap Hosts

Now that I've figured out how to accurately detect and block most CGI and PHP proxy servers I'm just sitting here going down the list trying them all and they're leaving my presents.

Yes, they're giving me.... PRESENTS!

What kind of presents you might ask?

Well, how about a list of cheap web hosts that these sites use, also used by scrapers, so I'm picking up a ton of valuable intelligence on these operation in very little time and will be massively expanding the list of blocked hosts to monitor those locations moving forward.

ServePath to more IPs

Just after I blocked the last batch of IPs from ServePath someone popped up on yet a new location ranging from 69.59.128.0 to 69.59.191.255 . All of their reverse DNS starts with "customer-reverse-entry." so I think I'll just zap them by that host address phrase and save some trouble here.

Proxy List Connecting PlanetLab Dots

It's hard to imagine that the same servers hosting PlanetLab experiments are also hosting anonymous proxies, but it appears to be true so now it's hard to say whether it was the PlanetLab network or just the list of proxies that attacked the server last week.

The only clue that it was really the PlanetLab network in that original attack was the entry "pli1-pa-3.hpl.hp.com." as HP has a special hardware deal for PlanetLab members.

Discounted Hardware
We are pleased to announce that HP is now providing a special discount price on PlanetLab machines. PIs will be able to see configuration and pricing information when they log in.

Mostly what has been learned from this proxy list is that institutions of higher education appear to be giving the internet bottom feeders the ability to steal.

Here's the reverse DNS on most of the proxy list that I processed as it's interesting to see who's hosting these things:

1.2.103.142.in-addr.arpa name = planetlab1.cs.ubc.ca.
10.202.108.129.in-addr.arpa name = planetlab1.utep.edu.
10.23.72.132.in-addr.arpa name = planetlab1.bgu.ac.il.
101.65.151.128.in-addr.arpa name = planet1.cs.rochester.edu.
102.65.151.128.in-addr.arpa name = planet2.cs.rochester.edu.
105.150.22.129.in-addr.arpa name = planetlab-2.EECS.CWRU.Edu.
106.139.112.128.in-addr.arpa name = planetlab-10.CS.Princeton.EDU.
108.139.112.128.in-addr.arpa name = planetlab-9.CS.Princeton.EDU.
109.118.83.147.in-addr.arpa name = planetlab2.upc.es.
109.146.68.207.in-addr.arpa name = sasch1031210.phx.gbl.
11.1.31.128.in-addr.arpa name = planetlab1.csail.mit.edu.
11.23.72.132.in-addr.arpa name = planetlab2.bgu.ac.il.
11.36.4.128.in-addr.arpa name = planetlab1.pc.cis.udel.edu.
110.139.112.128.in-addr.arpa name = planetlab-11.CS.Princeton.EDU.
111.120.10.129.in-addr.arpa name = planetlabone.ccs.neu.edu.
111.126.8.128.in-addr.arpa name = salt.planetlab.cs.umd.edu.
111.139.112.128.in-addr.arpa name = planetlab-13.CS.Princeton.EDU.
112.120.10.129.in-addr.arpa name = planetlabtwo.ccs.neu.edu.
112.126.8.128.in-addr.arpa name = pepper.planetlab.cs.umd.edu.
12.36.4.128.in-addr.arpa name = planetlab2.pc.cis.udel.edu.
122.100.44.219.in-addr.arpa name = softbank219044100122.bbtec.net.
123.17.42.194.in-addr.arpa name = planetlab-1.cs.ucy.ac.cy.
124.17.42.194.in-addr.arpa name = planetlab-2.cs.ucy.ac.cy.
125.120.148.210.in-addr.arpa name = catv120-125.lan-do.ne.jp.
126.60.247.140.in-addr.arpa name = righthand.eecs.harvard.edu.
129.20.195.216.cpe.townisp.com name = dhcp-0-11-11-8e-ee-34.cpe.townisp.com.
129.20.195.216.in-addr.arpa canonical name = 129.20.195.216.cpe.townisp.com.
130.182.167.193.in-addr.arpa name = pl-1.hip.fi.
130.208.76.64.in-addr.arpa name = c6476112-130.impsat.com.co.
133.204.23.138.in-addr.arpa name = planet-lab1.cs.ucr.edu.
137.143.128.in-addr.arpa nameserver = athena.cs.Virginia.EDU.
137.143.128.in-addr.arpa nameserver = athena.cs.Virginia.EDU.
138.228.240.129.in-addr.arpa name = planetlab2.simula.no.
14.1.31.128.in-addr.arpa name = planetlab4.csail.mit.edu.
14.63.114.128.in-addr.arpa name = planetslug1.cse.ucsc.edu.
143.109.38.192.in-addr.arpa canonical name = 143.128-191.109.38.192.in-addr.arpa.
143.128-191.109.38.192.in-addr.arpa name = amigos13.distlab.diku.dk.
144.109.38.192.in-addr.arpa canonical name = 144.128-191.109.38.192.in-addr.arpa.
144.128-191.109.38.192.in-addr.arpa name = amigos14.distlab.diku.dk.
145.236.16.220.in-addr.arpa name = softbank220016236145.bbtec.net.
148.12.100.138.in-addr.arpa name = planetlab1.ls.fi.upm.es.
149.11.135.128.in-addr.arpa name = planetlab1.cs.uchicago.edu.
149.12.100.138.in-addr.arpa name = planetlab2.ls.fi.upm.es.
15.1.31.128.in-addr.arpa name = planetlab5.csail.mit.edu.
15.63.114.128.in-addr.arpa name = planetslug2.cse.ucsc.edu.
150.145.245.130.in-addr.arpa name = planetlab1.mnl.cs.sunysb.edu.
152.11.135.128.in-addr.arpa name = planetlab3.cs.uchicago.edu.
152.145.245.130.in-addr.arpa name = planetlab3.mnl.cs.sunysb.edu.
154.127.233.220.in-addr.arpa name = 154.127.233.220.exetel.com.au.
154.159.210.216.in-addr.arpa name = 216-210-159-154.atgi.net.
154.247.194.61.in-addr.arpa canonical name = 154.SUB152.247.194.61.in-addr.arpa.
154.40.161.130.in-addr.arpa name = planetlab2.ewi.tudelft.nl.
154.SUB152.247.194.61.in-addr.arpa name = ns.m-mbc.co.jp.
156.192.6.128.in-addr.arpa name = orbpl2.rutgers.edu.
157.112.20.221.in-addr.arpa name = softbank221020112157.bbtec.net.
157.12.27.220.in-addr.arpa name = softbank220027012157.bbtec.net.
159.19.200.in-addr.arpa nameserver = curau.pop-mg.rnp.br.
159.19.200.in-addr.arpa nameserver = quindim.pop-mg.rnp.br.
16.1.31.128.in-addr.arpa name = planetlab6.csail.mit.edu.
16.63.114.128.in-addr.arpa name = planetslug3.cse.ucsc.edu.
161.33.24.141.in-addr.arpa name = planet1.prakinf.tu-ilmenau.de.
161.76.186.219.in-addr.arpa name = softbank219186076161.bbtec.net.
168.128.38.202.in-addr.arpa name = vw.ihep.ac.cn.
17.1.31.128.in-addr.arpa name = planetlab7.csail.mit.edu.
176.160.37.220.in-addr.arpa name = softbank220037160176.bbtec.net.
178.234.59.200.in-addr.arpa name = inalambrico178-234-regina.neunet.com.ar.
18.75.63.193.in-addr.arpa name = planetlab-1.ic.ac.uk.
181.152.67.219.in-addr.arpa canonical name = 181.176.152.67.219.in-addr.arpa.
181.176.152.67.219.in-addr.arpa name = vcs.tokyuhotel.co.jp.
184.139.in-addr.arpa nameserver = dns0.cs.bham.ac.uk.
184.139.in-addr.arpa nameserver = ns1.susx.ac.uk.
184.139.in-addr.arpa nameserver = ns2.susx.ac.uk.
19.119.216.208.in-addr.arpa name = planetlab1.gti-dsl.nodes.planet-lab.org.
19.75.63.193.in-addr.arpa name = planetlab-2.ic.ac.uk.
190.193.6.207.in-addr.arpa name = d207-6-193-190.bchsia.telus.net.
190.68.30.220.in-addr.arpa name = softbank220030068190.bbtec.net.
191.214.170.129.in-addr.arpa name = planetlab1.cs.dartmouth.edu.
192.214.170.129.in-addr.arpa name = planetlab2.cs.dartmouth.edu.
193.128.77.199.in-addr.arpa name = planet1.cc.gt.atl.ga.us.
193.152.252.132.in-addr.arpa name = planetlab1.exp-math.uni-essen.de.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-due.de.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-duisburg-essen.de.
194.115.145.136.in-addr.arpa name = planetlab-01.ece.uprm.edu.
194.128.77.199.in-addr.arpa name = planet.cc.gt.atl.ga.us.
196.19.242.129.in-addr.arpa name = planetlab1.cs.uit.no.
197.19.242.129.in-addr.arpa name = planetlab2.cs.uit.no.
197.4.208.128.in-addr.arpa name = planetlab01.cs.washington.edu.
198.4.16.221.in-addr.arpa name = softbank221016004198.bbtec.net.
198.4.208.128.in-addr.arpa name = planetlab02.cs.washington.edu.
199.160.83.130.in-addr.arpa name = planetlab2.rbg.informatik.tu-darmstadt.de.
199.4.208.128.in-addr.arpa name = planetlab03.cs.washington.edu.
199.69.127.216.in-addr.arpa name = gilletts.com.au.
2.142.19.139.in-addr.arpa name = swsat1501.mpi-sws.mpg.de.
2.142.19.139.in-addr.arpa name = planetlab02.mpi-sws.mpg.de.
2.2.103.142.in-addr.arpa name = planetlab2.cs.ubc.ca.
2.60.116.195.in-addr.arpa name = planetlab2.warsaw.rd.tp.pl.
20.102.204.132.in-addr.arpa name = crt1.PLANETLAB.UMontreal.CA.
20.19.252.128.in-addr.arpa name = vn1.cse.wustl.edu.
200.160.83.130.in-addr.arpa name = planetlab3.rbg.informatik.tu-darmstadt.de.
201.4.213.141.in-addr.arpa name = planetlab1.eecs.umich.edu.
201.67.59.128.in-addr.arpa name = planetlab2.comet.columbia.edu.
202.215.117.219.in-addr.arpa name = 219.117.215.202.user.rb.il24.net.
202.4.213.141.in-addr.arpa name = planetlab2.eecs.umich.edu.
202.67.59.128.in-addr.arpa name = planetlab3.comet.columbia.edu.
203.103.232.128.in-addr.arpa name = planetlab3.xeno.cl.cam.ac.uk.
209.218.149.141.in-addr.arpa name = planetlab4-dsl.cs.cornell.edu.
21.19.252.128.in-addr.arpa name = vn2.cse.wustl.edu.
21.254.136.130.in-addr.arpa name = planetlab1.CS.UniBO.IT.
210.130.82.206.in-addr.arpa name = quize.onatel.bf.
210.91.100.12.in-addr.arpa name = 210.mula.mlwk.chcgil24.dsl.att.net.
217.101.192.128.in-addr.arpa name = itchy.cs.uga.edu.
218.101.192.128.in-addr.arpa name = scratchy.cs.uga.edu.
219.135.41.192.in-addr.arpa canonical name = 219.deleg-192.135.41.192.in-addr.arpa.
219.deleg-192.135.41.192.in-addr.arpa name = planetlab2.csg.unizh.ch.
22.102.204.132.in-addr.arpa name = crt3.PLANETLAB.UMontreal.CA.
22.19.252.128.in-addr.arpa name = vn3.cse.wustl.edu.
22.254.136.130.in-addr.arpa name = planetlab2.CS.UniBO.IT.
224.35.228.194.in-addr.arpa name = zakskola.nosovice.indos.cz.
225.17.239.132.in-addr.arpa name = planetlab2.ucsd.edu.
227.202.2.134.in-addr.arpa name = peace.ri.uni-tuebingen.de.
228.202.2.134.in-addr.arpa name = freedom.ri.uni-tuebingen.de.
230.144.90.85.in-addr.arpa name = 85-90-144-230.DSL.ycn.com.
230.152.163.198.in-addr.arpa name = planetlab2.win.trlabs.ca.
231.79.114.140.in-addr.arpa name = pads21.cs.nthu.edu.tw.
233.79.114.140.in-addr.arpa name = pads23.cs.nthu.edu.tw.
235.226.113.128.in-addr.arpa name = planet1.ecse.rpi.edu.
236.229.225.143.in-addr.arpa name = planetlab01.dis.unina.it.
238.229.225.143.in-addr.arpa name = planetlab02.dis.unina.it.
238.75.97.129.in-addr.arpa name = blast.cs.uwaterloo.ca.
242.38.80.194.in-addr.arpa name = planetlab1.cs-ipv6.lancs.ac.uk.
243.198.37.130.in-addr.arpa name = planetlab1.cs.vu.nl.
243.38.80.194.in-addr.arpa name = planetlab2.cs-ipv6.lancs.ac.uk.
244.198.37.130.in-addr.arpa name = planetlab2.cs.vu.nl.
244.47.240.60.in-addr.arpa name = infinitem.com.
246.3.150.142.in-addr.arpa name = planetlab01.erin.utoronto.ca.
247.3.150.142.in-addr.arpa name = planetlab02.erin.utoronto.ca.
249.137.143.128.in-addr.arpa name = planetlab1.cs.Virginia.EDU.
249.99.246.138.in-addr.arpa name = planetlab1.lkn.ei.tum.de.
25.18.246.64.in-addr.arpa name = ev1s-64-246-18-25.ev1servers.net.
25.191.136.193.in-addr.arpa name = planetlab-1.iscte.pt.
250.137.143.128.in-addr.arpa name = planetlab2.cs.Virginia.EDU.
251.70.92.130.in-addr.arpa name = planetlab01.cnds.unibe.ch.
252.70.92.130.in-addr.arpa name = planetlab02.cnds.unibe.ch.
253.253.137.129.in-addr.arpa name = planetlab1.uc.edu.
26.106.199.203.in-addr.arpa name = 203.199.106.26.static.vsnl.net.in.
26.191.136.193.in-addr.arpa name = planetlab-2.iscte.pt.
26.203.88.130.in-addr.arpa name = planet1.manchester.ac.uk.
26.27.9.35.in-addr.arpa name = planetlab1.cse.msu.edu.
27.203.88.130.in-addr.arpa name = planet2.manchester.ac.uk.
28.247.220.128.in-addr.arpa name = planetlab1.isi.jhu.edu.
29.247.220.128.in-addr.arpa name = planetlab2.isi.jhu.edu.
3.142.19.139.in-addr.arpa name = swsat1502.mpi-sws.mpg.de.
3.142.19.139.in-addr.arpa name = planetlab03.mpi-sws.mpg.de.
34.248.207.206.in-addr.arpa name = planetlab1.arizona-gigapop.net.
34.60.116.195.in-addr.arpa name = planetlab2.olsztyn.rd.tp.pl.
35.159.19.200.in-addr.arpa name = planetlab2.pop-mg.rnp.br.
35.248.207.206.in-addr.arpa name = planetlab2.arizona-gigapop.net.
4.20.6.193.in-addr.arpa name = planet1.colbud.hu.
40.127.203.130.in-addr.arpa name = planetlab00.cse.psu.edu.
40.221.49.130.in-addr.arpa name = planetlab1.cs.pitt.edu.
41.127.203.130.in-addr.arpa name = planetlab01.cse.psu.edu.
41.221.49.130.in-addr.arpa name = planetlab2.cs.pitt.edu.
42.188.161.205.in-addr.arpa name = cache.gua.net.
44.146.68.207.in-addr.arpa name = sasch1031305.phx.gbl.
49.60.116.195.in-addr.arpa name = planetlab1.swidnik.rd.tp.pl.
5.142.19.139.in-addr.arpa name = planetlab05.mpi-sws.mpg.de.
5.142.19.139.in-addr.arpa name = swsat1504.mpi-sws.mpg.de.
5.20.6.193.in-addr.arpa name = planet2.colbud.hu.
50.48.217.141.in-addr.arpa name = planetlab1.cs.wayne.edu.
51.249.107.210.in-addr.arpa name = planetlab2.icu.ac.kr.
51.48.217.141.in-addr.arpa name = planetlab2.cs.wayne.edu.
52.19.10.128.in-addr.arpa name = planetlab1.cs.purdue.edu.
52.2.216.144.in-addr.arpa name = planetlab-1.unk.edu.
53.19.10.128.in-addr.arpa name = planetlab2.cs.purdue.edu.
53.2.216.144.in-addr.arpa name = planetlab-2.unk.edu.
55.48.184.139.in-addr.arpa name = planetlab1.rn.informatics.scitech.susx.ac.uk.
56.218.37.80.in-addr.arpa name = 56.Red-80-37-218.staticIP.rima-tde.net.
56.240.11.133.in-addr.arpa name = planetlab1.iii.u-tokyo.ac.jp.
56.68.93.129.in-addr.arpa name = planetlab1.unl.edu.
57.240.11.133.in-addr.arpa name = planetlab2.iii.u-tokyo.ac.jp.
6.142.19.139.in-addr.arpa name = planetlab06.mpi-sws.mpg.de.
6.142.19.139.in-addr.arpa name = swsat1505.mpi-sws.mpg.de.
61.52.111.128.in-addr.arpa name = planet1.cs.ucsb.edu.
62.52.111.128.in-addr.arpa name = planet2.cs.ucsb.edu.
65.60.116.195.in-addr.arpa name = planetlab1.piotrkow.rd.tp.pl.
65.88.238.128.in-addr.arpa name = planetlab2.poly.edu.
66.169.49.62.in-addr.arpa name = no-dns-yet.demon.co.uk.
69.126.8.128.in-addr.arpa name = planetlab2.cs.umd.edu.
69.202.221.201.in-addr.arpa name = 201-221-202-69.bk11-dsl.surnet.cl.
70.0.132.200.in-addr.arpa name = planetlab2.pop-rs.rnp.br.
70.112.179.131.in-addr.arpa name = Planetlab1.CS.UCLA.EDU.
70.255.159.200.in-addr.arpa name = planetlab1.pop-rj.rnp.br.
71.112.179.131.in-addr.arpa name = Planetlab2.CS.UCLA.EDU.
71.139.112.128.in-addr.arpa name = planetlab-1.CS.Princeton.EDU.
71.70.91.139.in-addr.arpa name = planet2.ics.forth.gr.
72.139.112.128.in-addr.arpa name = planetlab-2.CS.Princeton.EDU.
73.139.112.128.in-addr.arpa name = planetlab-3.CS.Princeton.EDU.
74.139.112.128.in-addr.arpa name = planetlab-6.CS.Princeton.EDU.
74.3.12.129.in-addr.arpa name = planetlab1.ukc.ac.uk.
74.44.201.212.in-addr.arpa canonical name = 74.72/29.44.201.212.in-addr.arpa.
74.72/29.44.201.212.in-addr.arpa name = planetlab2.eecs.iu-bremen.de.
75.3.12.129.in-addr.arpa name = planetlab2.ukc.ac.uk.
77.35.248.60.in-addr.arpa name = 60-248-35-77.HINET-IP.hinet.net.
79.109.165.216.in-addr.arpa name = planetx.scs.cs.nyu.edu.
80.139.112.128.in-addr.arpa name = alice.CS.Princeton.EDU.
81.108.54.202.in-addr.arpa name = delhi-202.54.108-81.vsnl.net.in.
81.109.165.216.in-addr.arpa name = planet1.scs.cs.nyu.edu.
82.109.165.216.in-addr.arpa name = planet2.scs.cs.nyu.edu.
82.139.112.128.in-addr.arpa name = planetlab-8.CS.Princeton.EDU.
82.56.227.128.in-addr.arpa name = planetlab2.acis.ufl.edu.
82.60.116.195.in-addr.arpa name = planetlab1.krakow.rd.tp.pl.
83.60.116.195.in-addr.arpa name = planetlab2.krakow.rd.tp.pl.
87.11.99.137.in-addr.arpa name = planetlab2.engr.uconn.edu.
87.2.17.66.in-addr.arpa name = 66-17-2-87.biz.bkfd.arrival.net.
90.150.22.129.in-addr.arpa name = planetlab-1.EECS.CWRU.Edu.
91.112.214.128.in-addr.arpa name = planetlab1.hiit.fi.
92.112.214.128.in-addr.arpa name = planetlab2.hiit.fi.
96.139.112.128.in-addr.arpa name = planetlab-4.CS.Princeton.EDU.
97.139.112.128.in-addr.arpa name = planetlab-5.CS.Princeton.EDU.

Someone else has also published a list showing PlanetLab proxies all over the place.

Scrapers Impacting Conversion Rates?

Was just reading on Threadwatch about the Shop.org report on the decline in conversion rates for online stores and suddenly had an epihany that scrapers may be involved in this equation.

Let's assume that these conversion rate facts and figures include many of the non-human stealth crawlers that I'm blocking on a daily basis. There's no way your average online retailer is probably aware of this situation and you know they're being scraped just like the rest of us, maybe even scraped MORE than the rest of us, who knows.

Using one of my websites as an example, it averages 13,500 visitors a day and 50-200 stealth crawlers are being blocked which accounts for .5% - 1.5% of my daily traffic, which would definitely impact the conversion rate for any store with similar traffic.

Perhaps scrapers on a very large website getting a million visitors a day wouldn't have much impact unless the site attracts a lot more scrapers than my site. However, a smaller online retailer with similar traffic to the site I'm protecting would obviously notice a difference in their conversion rate, a HUGE difference, just by adjusting their stats to include pages downloaded by stealth crawlers.

Just another example of how scraping and stealth crawling is BAD FOR THE WEB and needs to be stopped.

ServePath to being banned

Found a bunch of random stuff coming from a hosting company called ServePath today while running historical analysis on a batch of IPs.

Now these are the visible crawlers that came from ServePath:

64.151.75.252 PEAR HTTP_Request class ( http://pear.php.net/ )
64.151.64.212 "Jakarta Commons-HttpClient/3.0"
64.151.65.12 "Jakarta Commons-HttpClient/3.0"
64.151.111.116 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
64.151.112.44 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

Here's the whole range:

OrgName: ServePath, LLC
NetRange: 64.151.64.0 - 64.151.127.255
CIDR: 64.151.64.0/18

I'm going to block the whole thing and see if there are any stealth crawlers operating out of that location that haven't tripped any alarms yet and see what happens.

Amazon's A9 Amateur Hour

Guess what boys and girls?

We've all been forced to OPT-IN to yet another non-standard "web tool" that Amazon's A9 has thrust upon us. A9's blog said they introduced this crap last July but it's obviously been so low key compared to everything else hitting my server that I overlooked this small slice of idiocy.

This has been showing up in my logs for a while now"

207.171.167.25 - “GET /siteinfo.xml HTTP/1.1″ 404 1639 “-” “Java/1.5.0_04″

The only reason I noticed it today was the amount of times it hit the server escalated and they're racking up a bunch of 404 errors requesting this file I've never heard about which is idiotic and stupid.

Ever hear of any internet standard such as ROBOTS.TXT to see if I even want you looking for this stupid file on my server?

Apparently not as the only file being hit is "siteinfo.xml".

Had to resort to a reverse DNS lookup just to find out it was iad-fw-global.amazon.com who was doing this stupid crap. Didn't the vaudeville programmers that wrote this joke ever hear of setting the USER AGENT to identify who and what this is instead of Java/1.5?

Amazon, if you happen to read this pay very close attention to the fact that many web applications bombard my server with the user agent of "Java/1.whatever" on a daily basis which are all BLOCKED so you will never ever get access to siteinfo.xml until you properly identify yourself.

Here's a sample "siteinfo.xml" file that you can install in your root web directory:

<?xml version="1.0"?>
<siteinfo xmlns="http://a9.com/-/spec/siteinfo/1.0/">
<webmenu>
<name>Amazon SiteInfo Sucks</name>
<menu>
<item>
<text>Doesn't use standards</text>
<url>http://www.robotstxt.org/</url>
</item>
<item>
<text>Doesn't identify itself</text>
<url>http://www.mozilla.org/build/revised-user-agent-strings.html</url>
</item>
</menu>
</webmenu>
</siteinfo>

I commented about their lack of professionalism and standards being used in this implementation on their blog but it's awaiting moderation and I doubt they'll let my less than happy comments be published, but we shall see.

PlanetLabs Bombards Server - Abused or Compromised?

Well here's a new one that was uncovered this week when a tipster wishing to remain anonymous sent me a very suspicious looking log file snippet with a bunch of identical accesses from over 130 IP addresses ranging over a couple of hours.

After doing a little bit of research it looks like this "attack" came from a consortium of computers called PlanetLab located in various universities and research institutions around the world and this appears to be only a portion of the network that was aimed at our tipsters server. We don't know at this point if this was an isolated demonstration of their network, whether they were being abused by a member or if a hacker has breeched the protocol, but the potential for damage here is huge.

Their website claims the following stats:

PlanetLab currently consists of 668 machines, hosted by 325 sites, spanning over 25 countries. Most of the machines are hosted by research institutions, although some are located in co-location and routing centers (e.g., on Internet2's Abilene backbone). All of the machines are connected to the Internet. The goal is for PlanetLab to grow to 1,000 widely distributed nodes that peer with the majority of the Internet's regional and long-haul backbones.

Below are sample of the log files, IPs involved, and the reverse DNS of all the IPs which is what we used to figure out this was probably PlanetLab. There were other files accessed as well, but browsers don't typically look at robots.txt so that's all we needed to suspect something was wrong with this situation and treated it as a potential attack.

If this was an actual PlanetLab project aimed at crawling the web undetected and aggregate tons of data, then it failed miserably. Now that we know who you are and where you are, our servers will be watching to see if you strike again.

If this was an unauthorized test then PlanetLab better beef up security as this network is one big DDoS attack just waiting to happen under control of the wrong person.

Here's a sample snippet of the log file:

216.165.109.81 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
216.165.109.82 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
216.165.109.79 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
141.161.20.32 - - [11/May/2006:08:45:59 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
138.100.12.149 - - [11/May/2006:08:46:03 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
138.100.12.148 - - [11/May/2006:08:46:03 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
199.77.128.193 - - [11/May/2006:08:46:38 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
199.77.128.194 - - [11/May/2006:08:46:39 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"

Here's the complete list of IP's involved:

216.165.109.81
216.165.109.82
216.165.109.79
141.161.20.32
138.100.12.149
138.100.12.148
199.77.128.193
199.77.128.194
128.31.1.11
128.31.1.15
128.31.1.14
128.31.1.16
138.23.204.232
138.23.204.133
128.31.1.12
128.31.1.13
130.245.145.152
128.83.122.181
128.83.122.180
152.3.138.2
152.3.138.3
128.220.247.28
12.46.129.21
169.229.50.13
169.229.50.10
169.229.50.17
12.46.129.22
169.229.50.8
12.46.129.23
169.229.50.9
169.229.50.12
169.229.50.16
198.133.224.145
129.115.248.225
194.36.10.154
194.36.10.156
128.208.4.199
171.66.3.181
205.189.33.178
193.6.20.4
193.6.20.5
129.170.214.191
129.170.214.192
130.37.198.243
128.143.137.250
128.111.52.62
194.80.38.242
194.80.38.243
131.234.66.161
131.234.66.160
132.227.74.40
169.229.50.18
169.229.50.7
129.186.205.77
155.98.35.4
155.98.35.3
130.104.72.201
130.104.72.200
128.227.56.82
132.252.152.193
147.83.118.123
147.83.118.124
147.83.118.109
147.83.118.125
130.88.203.26
130.88.203.27
193.10.64.36
142.103.2.2
142.103.2.1
193.10.133.128
193.1.201.26
212.201.44.74
133.11.240.56
133.11.240.57
193.167.182.130
132.72.23.11
132.72.23.10
195.116.60.82
195.116.60.83
132.204.102.20
132.204.102.22
130.203.127.40
130.203.127.41
129.242.19.196
169.229.50.11
128.252.19.21
129.242.19.197
129.22.150.105
138.251.214.18
138.251.214.19
128.151.65.101
128.151.65.102
193.63.75.19
134.76.81.241
134.76.81.242
128.59.67.202
130.136.254.22
210.125.84.16
210.125.84.15
140.109.17.181
200.132.0.70
195.116.60.65
204.123.28.53
131.188.44.101
128.8.126.112
128.8.126.69
128.8.126.111
131.246.19.202
163.221.11.73
163.221.11.71
163.221.11.72
193.144.21.130
193.144.21.131
165.230.49.114
165.230.49.115
143.248.139.170
128.232.103.201
128.232.103.203
202.249.37.212
192.33.210.16
193.136.191.26
193.136.191.25
130.49.221.41
192.17.239.251
192.17.239.250
192.41.135.218
192.41.135.219
142.150.3.247
142.150.3.246
200.159.255.70
128.232.103.202
134.226.52.34
134.226.52.35

To make sense of all this mess, I crunched them all thru NSLOOKUP to see if any patterns emerged and what was a common theme was .EDU and PLANETLAB all over the place.

Here's the reverse DNS on all the IPs for your viewing pleasure:

35.52.226.134.in-addr.arpa name = planetlab02.cs.tcd.ie.
81.109.165.216.in-addr.arpa name = planet1.scs.cs.nyu.edu.
82.109.165.216.in-addr.arpa name = planet2.scs.cs.nyu.edu.
79.109.165.216.in-addr.arpa name = planetx.scs.cs.nyu.edu.
32.20.161.141.in-addr.arpa name = planetlab1.georgetown.edu.
149.12.100.138.in-addr.arpa name = planetlab2.ls.fi.upm.es.
148.12.100.138.in-addr.arpa name = planetlab1.ls.fi.upm.es.
193.128.77.199.in-addr.arpa name = planet1.cc.gt.atl.ga.us.
194.128.77.199.in-addr.arpa name = planet.cc.gt.atl.ga.us.
11.1.31.128.in-addr.arpa name = planetlab1.csail.mit.edu.
15.1.31.128.in-addr.arpa name = planetlab5.csail.mit.edu.
14.1.31.128.in-addr.arpa name = planetlab4.csail.mit.edu.
16.1.31.128.in-addr.arpa name = planetlab6.csail.mit.edu.
232.204.23.138.in-addr.arpa name = planet-lab2.cs.ucr.edu.
133.204.23.138.in-addr.arpa name = planet-lab1.cs.ucr.edu.
12.1.31.128.in-addr.arpa name = planetlab2.csail.mit.edu.
13.1.31.128.in-addr.arpa name = planetlab3.csail.mit.edu.
152.145.245.130.in-addr.arpa name = planetlab3.mnl.cs.sunysb.edu.
181.122.83.128.in-addr.arpa name = planetlab3.csres.utexas.edu.
180.122.83.128.in-addr.arpa name = planetlab2.csres.utexas.edu.
2.138.3.152.in-addr.arpa name = planetlab2.cs.duke.edu.
3.138.3.152.in-addr.arpa name = planetlab3.cs.duke.edu.
28.247.220.128.in-addr.arpa name = planetlab1.isi.jhu.edu.
21.129.46.12.in-addr.arpa canonical name = 21.0/25.129.46.12.in-addr.arpa.
21.0/25.129.46.12.in-addr.arpa name = planet1.berkeley.intel-research.net.
13.50.229.169.in-addr.arpa name = planetlab11.Millennium.Berkeley.EDU.
10.50.229.169.in-addr.arpa name = planetlab8.Millennium.Berkeley.EDU.
17.50.229.169.in-addr.arpa name = planetlab15.Millennium.Berkeley.EDU.
22.129.46.12.in-addr.arpa canonical name = 22.0/25.129.46.12.in-addr.arpa.
22.0/25.129.46.12.in-addr.arpa name = planet2.berkeley.intel-research.net.
8.50.229.169.in-addr.arpa name = planetlab6.Millennium.Berkeley.EDU.
23.129.46.12.in-addr.arpa canonical name = 23.0/25.129.46.12.in-addr.arpa.
23.0/25.129.46.12.in-addr.arpa name = planet3.berkeley.intel-research.net.
9.50.229.169.in-addr.arpa name = planetlab7.Millennium.Berkeley.EDU.
12.50.229.169.in-addr.arpa name = planetlab10.Millennium.Berkeley.EDU.
16.50.229.169.in-addr.arpa name = planetlab14.Millennium.Berkeley.EDU.
145.224.133.198.in-addr.arpa name = planetlab1.cs.wisc.edu.
225.248.115.129.in-addr.arpa name = pl1a.pl.utsa.edu.
154.10.36.194.in-addr.arpa name = planetlab1.nrl.dcs.qmul.ac.uk.
156.10.36.194.in-addr.arpa name = planetlab2.nrl.dcs.qmul.ac.uk.
199.4.208.128.in-addr.arpa name = planetlab03.cs.washington.edu.
181.3.66.171.in-addr.arpa name = planet1.scs.stanford.edu.
178.33.189.205.in-addr.arpa name = planet1.ottawa.canet4.nodes.planet-lab.org.
4.20.6.193.in-addr.arpa name = planet1.colbud.hu.
5.20.6.193.in-addr.arpa name = planet2.colbud.hu.
191.214.170.129.in-addr.arpa name = planetlab1.cs.dartmouth.edu.
192.214.170.129.in-addr.arpa name = planetlab2.cs.dartmouth.edu.
243.198.37.130.in-addr.arpa name = planetlab1.cs.vu.nl.
250.137.143.128.in-addr.arpa name = planetlab2.cs.Virginia.EDU.
62.52.111.128.in-addr.arpa name = planet2.cs.ucsb.edu.
242.38.80.194.in-addr.arpa name = planetlab1.cs-ipv6.lancs.ac.uk.
243.38.80.194.in-addr.arpa name = planetlab2.cs-ipv6.lancs.ac.uk.
161.66.234.131.in-addr.arpa name = planetlab-2.cs.upb.de.
160.66.234.131.in-addr.arpa name = planetlab-1.cs.upb.de.
40.74.227.132.in-addr.arpa name = planetlab-01.ipv6.lip6.fr.
18.50.229.169.in-addr.arpa name = planetlab16.Millennium.Berkeley.EDU.
7.50.229.169.in-addr.arpa name = planetlab5.Millennium.Berkeley.EDU.
77.205.186.129.in-addr.arpa name = planetlab-4.ece.iastate.edu.
4.35.98.155.in-addr.arpa name = planetlab3.flux.utah.edu.
3.35.98.155.in-addr.arpa name = planetlab2.flux.utah.edu.
201.72.104.130.in-addr.arpa name = planetlab2.info.ucl.ac.be.
200.72.104.130.in-addr.arpa name = planetlab1.info.ucl.ac.be.
82.56.227.128.in-addr.arpa name = planetlab2.acis.ufl.edu.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-duisburg-essen.de.
193.152.252.132.in-addr.arpa name = planetlab1.exp-math.uni-essen.de.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-due.de.
123.118.83.147.in-addr.arpa name = planetlab3.upc.es.
124.118.83.147.in-addr.arpa name = planetlab4.upc.es.
109.118.83.147.in-addr.arpa name = planetlab2.upc.es.
125.118.83.147.in-addr.arpa name = planetlab5.upc.es.
26.203.88.130.in-addr.arpa name = planet1.manchester.ac.uk.
27.203.88.130.in-addr.arpa name = planet2.manchester.ac.uk.
36.64.10.193.in-addr.arpa name = planetlab2.sics.se.
2.2.103.142.in-addr.arpa name = planetlab2.cs.ubc.ca.
1.2.103.142.in-addr.arpa name = planetlab1.cs.ubc.ca.
128.133.10.193.in-addr.arpa name = planetlab-1.it.uu.se.
26.201.1.193.in-addr.arpa name = planetlab-1.tssg.org.
74.44.201.212.in-addr.arpa canonical name = 74.72/29.44.201.212.in-addr.arpa.
74.72/29.44.201.212.in-addr.arpa name = planetlab2.eecs.iu-bremen.de.
56.240.11.133.in-addr.arpa name = planetlab1.iii.u-tokyo.ac.jp.
57.240.11.133.in-addr.arpa name = planetlab2.iii.u-tokyo.ac.jp.
130.182.167.193.in-addr.arpa name = pl-1.hip.fi.
11.23.72.132.in-addr.arpa name = planetlab2.bgu.ac.il.
10.23.72.132.in-addr.arpa name = planetlab1.bgu.ac.il.
82.60.116.195.in-addr.arpa name = planetlab1.krakow.rd.tp.pl.
83.60.116.195.in-addr.arpa name = planetlab2.krakow.rd.tp.pl.
20.102.204.132.in-addr.arpa name = crt1.PLANETLAB.UMontreal.CA.
22.102.204.132.in-addr.arpa name = crt3.PLANETLAB.UMontreal.CA.
40.127.203.130.in-addr.arpa name = planetlab00.cse.psu.edu.
41.127.203.130.in-addr.arpa name = planetlab01.cse.psu.edu.
196.19.242.129.in-addr.arpa name = planetlab1.cs.uit.no.
11.50.229.169.in-addr.arpa name = planetlab9.Millennium.Berkeley.EDU.
21.19.252.128.in-addr.arpa name = vn2.cse.wustl.edu.
197.19.242.129.in-addr.arpa name = planetlab2.cs.uit.no.
105.150.22.129.in-addr.arpa name = planetlab-2.EECS.CWRU.Edu.
18.214.251.138.in-addr.arpa name = planetlab1.dcs.st-and.ac.uk.
19.214.251.138.in-addr.arpa name = planetlab2.dcs.st-and.ac.uk.
101.65.151.128.in-addr.arpa name = planet1.cs.rochester.edu.
102.65.151.128.in-addr.arpa name = planet2.cs.rochester.edu.
19.75.63.193.in-addr.arpa name = planetlab-2.ic.ac.uk.
241.81.76.134.in-addr.arpa name = planetlab1.informatik.uni-goettingen.de.
242.81.76.134.in-addr.arpa name = planetlab2.informatik.uni-goettingen.de.
202.67.59.128.in-addr.arpa name = planetlab3.comet.columbia.edu.
22.254.136.130.in-addr.arpa name = planetlab2.CS.UniBO.IT.
** server can't find 16.84.125.210.in-addr.arpa: NXDOMAIN
** server can't find 15.84.125.210.in-addr.arpa: NXDOMAIN
181.17.109.140.in-addr.arpa name = planetlab2.iis.sinica.edu.tw.
70.0.132.200.in-addr.arpa name = planetlab2.pop-rs.rnp.br.
65.60.116.195.in-addr.arpa name = planetlab1.piotrkow.rd.tp.pl.
53.28.123.204.in-addr.arpa name = pli1-pa-3.hpl.hp.com.
101.44.188.131.in-addr.arpa name = planetlab2.informatik.uni-erlangen.de.
112.126.8.128.in-addr.arpa name = pepper.planetlab.cs.umd.edu.
69.126.8.128.in-addr.arpa name = planetlab2.cs.umd.edu.
111.126.8.128.in-addr.arpa name = salt.planetlab.cs.umd.edu.
202.19.246.131.in-addr.arpa name = planetlab2.informatik.uni-kl.de.
73.11.221.163.in-addr.arpa name = planetlab-03.naist.jp.
71.11.221.163.in-addr.arpa name = planetlab-01.naist.jp.
72.11.221.163.in-addr.arpa name = planetlab-02.naist.jp.
130.21.144.193.in-addr.arpa name = planetlab.urv.net.
131.21.144.193.in-addr.arpa name = planetlab2.urv.net.
114.49.230.165.in-addr.arpa name = planetlab1.rutgers.edu.
115.49.230.165.in-addr.arpa name = planetlab2.rutgers.edu.
170.139.248.143.in-addr.arpa name = csplanetlab3.kaist.ac.kr.
201.103.232.128.in-addr.arpa name = planetlab1.xeno.cl.cam.ac.uk.
203.103.232.128.in-addr.arpa name = planetlab3.xeno.cl.cam.ac.uk.
212.37.249.202.in-addr.arpa name = planetlab2.koganei.wide.ad.jp.
16.210.33.192.in-addr.arpa name = lsirextpc01.epfl.ch.
26.191.136.193.in-addr.arpa name = planetlab-2.iscte.pt.
25.191.136.193.in-addr.arpa name = planetlab-1.iscte.pt.
41.221.49.130.in-addr.arpa name = planetlab2.cs.pitt.edu.
251.239.17.192.in-addr.arpa name = planetlab2.cs.uiuc.edu.
250.239.17.192.in-addr.arpa name = planetlab1.cs.uiuc.edu.
218.135.41.192.in-addr.arpa canonical name = 218.deleg-192.135.41.192.in-addr.arpa.
218.deleg-192.135.41.192.in-addr.arpa name = planetlab1.csg.unizh.ch.
219.135.41.192.in-addr.arpa canonical name = 219.deleg-192.135.41.192.in-addr.arpa.
219.deleg-192.135.41.192.in-addr.arpa name = planetlab2.csg.unizh.ch.
247.3.150.142.in-addr.arpa name = planetlab02.erin.utoronto.ca.
246.3.150.142.in-addr.arpa name = planetlab01.erin.utoronto.ca.
70.255.159.200.in-addr.arpa name = planetlab1.pop-rj.rnp.br.
202.103.232.128.in-addr.arpa name = planetlab2.xeno.cl.cam.ac.uk.
34.52.226.134.in-addr.arpa name = planetlab01.cs.tcd.ie.
35.52.226.134.in-addr.arpa name = planetlab02.cs.tcd.ie.

Best we can tell it was definitely PlanetLab involved with this and I'm very upset that an organization like this would aim a large section of their network at a single server at the same time without permission.

This is abuse, pure and simple, without proper user agent attribution or anything, and I welcome them to come here and let us know what really happened.

While we're waiting on PlanetLab to respond, and I wouldn't hold my breath, I'm going to block the IPs listed above and probably ban anything with "planetlab" or "planet-lab" in the reverse DNS location name until further notice.

RED ALERT #4 - NiceBot Neighborhood

Found another distributed IP batch sitting in a hosting farm claiming to be "nicebot".

Nicebot my ass...

Here's the range of IP's spotted with user agent nicebot:

69.60.120.165 - nicebot
69.60.120.166 - nicebot
69.60.120.167 - nicebot
69.60.120.168 - nicebot
69.60.120.169 - nicebot
69.60.120.172 - nicebot
69.60.120.173 - nicebot
69.60.120.174 - nicebot
69.60.120.176 - nicebot

NSLOOKUP claims they belong to ServerPronto.

nslookup 69.60.120.169
Server: 64.34.160.92
Address: 64.34.160.92#53

Non-authoritative answer:
169.120.60.69.in-addr.arpa name = 169-120-60-69.serverpronto.com.

So I think I'm going to just block this range from ServerPronto as it's a hosting farm:

Serverpronto INMM-69-60-114-0 (NET-69-60-114-0-1)
69.60.114.0 - 69.60.125.255

Some of you might naively think that you can just block "nicebot" with rewrite rules and solve your problem. However, my research has shown that many of these bots eventually change names when they get blocked by too many sites. You're best off blocking the source permanently so they don't slip thru the cracks next week crawling as something like ""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; BTOW V9.0; SV1)" which you can't detect.

Remember, they're desperate when you cut off their source of revenue and they'll attempt to adapt so use the best prevention up front which is lock them out by location and don't waste your time fighting changing user agent names.

Bots gone WILD!

This is just a follow up on a couple of the bots using distributed IP's I've highlighted recently which just won't take NO for an answer. Ever since their little cluster of scraping IPs has been uncovered and blocked it's still been a non-stop daily request for hundreds of pages per scraper.

These bots are very nasty so if you weren't paying attention the first time, go back and block THIS, THIS and THIS as they are some hungry-assed bots that need to be stopped.

BEZEQINT-HOSTING has a scraper

Coming from the lovely land of Israel is a scraper from Bezeq International, and I can't tell if this is a hosting IP or a DSL connection, but I'm guessing it's hosting but could just be DHCP.

Like I can read their website, feh!

Anyway, the bot always claims to be:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; (R1 1.1); .NET CLR 1.1.4322)

Comes from the following addresses:

82.80.249.201
82.80.249.203
82.80.249.204

So block those at a minimum and 82.80.249.0/24 if you want to be safe.

Ta ta, no scrape for you!

Odd traffic from Hong Kong, Middle-East and Africa

Has anyone noticed any huge spikes in traffic from Hong Kong, Saudi Arabia, South Africa or Dubai lately?

They are setting off alarms all over the place with my bot blocker and it's all coming from shared networks so I can't tell yet if it's just a lot of people using a few IPs or a few crawlers going crazy in a scraper haven.

I'm thinking about just setting the whole bunch of them to "CAPTCHA-mode" which is the equivalent of forcing them to login before accessing my site. This will quickly determine the number source of the activity based on the number of unanswered CAPTCHA's vs. a valid response from a human.

Let's see what happens next, I'll keep you posted ;)

RED ALERT #3 - GoDaddy hosting distributed scraper

This one may have just moved to a new location as I've been watching similar activity before which stopped. These new antics have been going on at this location for a week now and I waited just to make sure it was really coming from a common location which appears to be a block of IPs on some GoDaddy hosting farm secureserver.net.

This creepy crawler doesn't use any user agent string whatsoever and keeps asking for pages like "/#top" and other stupid stuff. Below is the range of IPs and the number of pages asked for just today. You'll note it was a slow day for them asking for only 75 pages, but the day isn't over yet.

68.178.242.111 [ip-68-178-242-111.ip.secureserver.net.] requested 30 pages as ""
68.178.242.126 [ip-68-178-242-126.ip.secureserver.net.] requested 15 pages as ""
68.178.242.128 [ip-68-178-242-128.ip.secureserver.net.] requested 15 pages as ""
68.178.242.127 [ip-68-178-242-127.ip.secureserver.net.] requested 15 pages as ""

Performed an nslookup and got this:

nslookup 68.178.242.111
Server: 64.34.160.92
Address: 64.34.160.92#53

Non-authoritative answer:
111.242.178.68.in-addr.arpa name = ip-68-178-242-111.ip.secureserver.net.

When I did a whois on the IP there came the surprise:

[Querying whois.arin.net]
[whois.arin.net]

OrgName: Go Daddy Software, Inc.
OrgID: GDS-31
Address: 14455 N Hayden Road
Address: Suite 226
City: Scottsdale
StateProv: AZ
PostalCode: 85260
Country: US
178.128.0 - 178.255.255

Now do a whois on secureserver.net:

NetRange: Registrant:
Special Domain Services, Inc.
14455 N Hayden Rd
Scottsdale, Arizona 85260
United States

Registered through: WWDomains.com
Domain Name: SECURESERVER.NET
Created on: 30-Mar-98
Expires on: 29-Mar-12
Last Updated on: 07-Feb-06

Not sure it makes sense to block the entire GoDaddy IP range, so for now 68.178.242.0/24 is all I'm blocking unless I see more rogue activity in their network.

BTW, anyone notice how many sneaky crawler networks I'm busting now that I have proximity alarms in place to spot organized activity?

This proximity alarm is great as it doesn't care if the crawlers ask for 1 page or 100 pages, the minute it detects multiple IP addresses in a similar range doing these things it pops up on my radar. The best thing is that the distributed crawler doesn't even have to use more than one IP address per day as long as they break one of my "bad bot rules" on each visit so the IP is flagged and archived. The proximity report of archived bad bot activity will then expose those archived bots operating from a single location.

Pretty tricky, eh?

You stupid bots better wise up quick, you can't hide behind a bank of IPs, your days are numbered!

Publicly Available Website

That's the current buzzword most often used when you confront someone crawling your site, especially a corporation, that it's a "Publicly Available Website".

Well just because something is publicly available doesn't mean you have the right to do whatever you like with it. It's publicly available for the PUBLIC, meaning visitors, to read individual pages and it's also available to the 6 search engines that I permit to crawl my site. Other than that, just like any other publicly available business, I have the RIGHT TO RESTRICT ACCESS to anyone else that I so desire.

For instance many brick and mortar businesses say "No Shoes, No Shirt, No Service".

Well my website has similar rules "No Humans, No Permission, No Service".

If I even get a whiff off a robot on the site, permission denied.

You corporate and private scrapers just better get over your loser mantra as putting a website online, even on a public network, does NOT give everyone complete access to do whatever they feel like with your site. There are terms of service on that site which distinctly prohibit the use of unauthorized tools to crawl that site, and if you have to ask what's authorized then you don't have permission in the first place so go away.

The site doesn't have a "GNU Free Documentation License", instead it has one of those funny things called a "copyright" which means I own it, not YOU. Additionally, I pay for the server, not YOU. Which means, it's up to ME what is and isn't allowed, even when it's a "Publicly Available Website", NOT YOU!

Let's make it so simple even a 2 year old can understand it:

The website is MINE! MINE! MINE! ALL MINE! and NOT YOURS!

Is that language clear enough for the mental midgets scraping the web to comprehend?

RED ALERT #2 - Distributed IP Scraper on BBCOM

This one is virtually identical to scraper spotted on Vericenter, same profile to the letter.

Claims to be the same exact browser:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1

This spider was seen operating from these IP addresses:

66.234.139.195
66.234.139.200
66.234.139.203
66.234.139.204
66.234.139.206
66.234.139.207
66.234.139.211
66.234.139.213
66.234.139.216
66.234.139.217

This IP range belongs to BBCOM:

OrgName: Backbone Communications, Inc.
OrgID: BBCM
Address: 515 South Flower Street
Address: Suite 4350
City: Los Angeles
StateProv: CA
PostalCode: 90071
Country: US

NetRange: 66.234.128.0 - 66.234.159.255

I'm blocking 66.234.139.0/24 for the moment, but keeping an eye on the rest of their network in case these scrapers switch to a new block.

Another Proximity Alert on PCCW

Getting multiple hits from the same IPs over and over asking for the exact same web page, no images, nothing else, just the same web page day after day. One day the user agent was blank and another day simply"Mozilla/5.0".

This is the range of IPs that just keep repeating the same request.

210.87.251.111
210.87.251.107
210.87.251.41
210.87.251.106

Appears to be coming from HK, can't tell if it's a shared DHCP situation or what, but I'm blocking 210.87.251.0/24 just to be safe.

Another DUMBASS Crawler

This one hails from South Africa, tried to crawl "/#top" as "GET /%23top"

You poor dumb fucker, BUSTED!

RED ALERT - Distributed IP Scraper Hosted on Vericenter

Here's a real sneaky scraper using distributed IPs that is using a bot that almost appears designed to fly under my bot blockers radar. No single IP address accessed enough pages or did anything obnoxious enough to set off any triggers but the collective accesses set off a proximity alarm and they got nailed anyway.

The scraper is pretending to be Firefox for Linux:

http://www.Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1/

The range of IP's noticed in this scrape attack are as follows:

65.38.102.138 ip-65-38-102-138.hou.vericenter.com
65.38.102.139 ip-65-38-102-139.hou.vericenter.com
65.38.102.141 ip-65-38-102-141.hou.vericenter.com
65.38.102.143 ip-65-38-102-143.hou.vericenter.com
65.38.102.145 ip-65-38-102-145.hou.vericenter.com
65.38.102.146 ip-65-38-102-146.hou.vericenter.com
65.38.102.147 ip-65-38-102-147.hou.vericenter.com
65.38.102.148 ip-65-38-102-148.hou.vericenter.com
65.38.102.150 ip-65-38-102-150.hou.vericenter.com
65.38.102.153 ip-65-38-102-153.hou.vericenter.com
65.38.102.155 ip-65-38-102-155.hou.vericenter.com
65.38.102.157 ip-65-38-102-157.hou.vericenter.com

The host information is as follows:

OrgName: VeriCenter, Inc.
OrgID: VRCT
Address: 757 N Eldridge Parkway
City: Houston
StateProv: TX
PostalCode: 77079
Country: US
NetRange: 65.38.96.0 - 65.38.111.255

Athough the attack seems to be centered on the 65.38.102.0/24 block at the Houston datacenter of Vericenter, I think I'm going to completely block Vericenter as it doesn't appear to have any ISP facilities [ie. NO HUMANS] and see if anything else bounces off the bot blocker from their facilities.