Scrapers Impacting Conversion Rates?

Was just reading on Threadwatch about the Shop.org report on the decline in conversion rates for online stores and suddenly had an epihany that scrapers may be involved in this equation.

Let's assume that these conversion rate facts and figures include many of the non-human stealth crawlers that I'm blocking on a daily basis. There's no way your average online retailer is probably aware of this situation and you know they're being scraped just like the rest of us, maybe even scraped MORE than the rest of us, who knows.

Using one of my websites as an example, it averages 13,500 visitors a day and 50-200 stealth crawlers are being blocked which accounts for .5% - 1.5% of my daily traffic, which would definitely impact the conversion rate for any store with similar traffic.

Perhaps scrapers on a very large website getting a million visitors a day wouldn't have much impact unless the site attracts a lot more scrapers than my site. However, a smaller online retailer with similar traffic to the site I'm protecting would obviously notice a difference in their conversion rate, a HUGE difference, just by adjusting their stats to include pages downloaded by stealth crawlers.

Just another example of how scraping and stealth crawling is BAD FOR THE WEB and needs to be stopped.

ServePath to being banned

Found a bunch of random stuff coming from a hosting company called ServePath today while running historical analysis on a batch of IPs.

Now these are the visible crawlers that came from ServePath:

64.151.75.252 PEAR HTTP_Request class ( http://pear.php.net/ )
64.151.64.212 "Jakarta Commons-HttpClient/3.0"
64.151.65.12 "Jakarta Commons-HttpClient/3.0"
64.151.111.116 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
64.151.112.44 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

Here's the whole range:

OrgName: ServePath, LLC
NetRange: 64.151.64.0 - 64.151.127.255
CIDR: 64.151.64.0/18

I'm going to block the whole thing and see if there are any stealth crawlers operating out of that location that haven't tripped any alarms yet and see what happens.

Amazon's A9 Amateur Hour

Guess what boys and girls?

We've all been forced to OPT-IN to yet another non-standard "web tool" that Amazon's A9 has thrust upon us. A9's blog said they introduced this crap last July but it's obviously been so low key compared to everything else hitting my server that I overlooked this small slice of idiocy.

This has been showing up in my logs for a while now"

207.171.167.25 - “GET /siteinfo.xml HTTP/1.1″ 404 1639 “-” “Java/1.5.0_04″

The only reason I noticed it today was the amount of times it hit the server escalated and they're racking up a bunch of 404 errors requesting this file I've never heard about which is idiotic and stupid.

Ever hear of any internet standard such as ROBOTS.TXT to see if I even want you looking for this stupid file on my server?

Apparently not as the only file being hit is "siteinfo.xml".

Had to resort to a reverse DNS lookup just to find out it was iad-fw-global.amazon.com who was doing this stupid crap. Didn't the vaudeville programmers that wrote this joke ever hear of setting the USER AGENT to identify who and what this is instead of Java/1.5?

Amazon, if you happen to read this pay very close attention to the fact that many web applications bombard my server with the user agent of "Java/1.whatever" on a daily basis which are all BLOCKED so you will never ever get access to siteinfo.xml until you properly identify yourself.

Here's a sample "siteinfo.xml" file that you can install in your root web directory:

<?xml version="1.0"?>
<siteinfo xmlns="http://a9.com/-/spec/siteinfo/1.0/">
<webmenu>
<name>Amazon SiteInfo Sucks</name>
<menu>
<item>
<text>Doesn't use standards</text>
<url>http://www.robotstxt.org/</url>
</item>
<item>
<text>Doesn't identify itself</text>
<url>http://www.mozilla.org/build/revised-user-agent-strings.html</url>
</item>
</menu>
</webmenu>
</siteinfo>

I commented about their lack of professionalism and standards being used in this implementation on their blog but it's awaiting moderation and I doubt they'll let my less than happy comments be published, but we shall see.

PlanetLabs Bombards Server - Abused or Compromised?

Well here's a new one that was uncovered this week when a tipster wishing to remain anonymous sent me a very suspicious looking log file snippet with a bunch of identical accesses from over 130 IP addresses ranging over a couple of hours.

After doing a little bit of research it looks like this "attack" came from a consortium of computers called PlanetLab located in various universities and research institutions around the world and this appears to be only a portion of the network that was aimed at our tipsters server. We don't know at this point if this was an isolated demonstration of their network, whether they were being abused by a member or if a hacker has breeched the protocol, but the potential for damage here is huge.

Their website claims the following stats:

PlanetLab currently consists of 668 machines, hosted by 325 sites, spanning over 25 countries. Most of the machines are hosted by research institutions, although some are located in co-location and routing centers (e.g., on Internet2's Abilene backbone). All of the machines are connected to the Internet. The goal is for PlanetLab to grow to 1,000 widely distributed nodes that peer with the majority of the Internet's regional and long-haul backbones.

Below are sample of the log files, IPs involved, and the reverse DNS of all the IPs which is what we used to figure out this was probably PlanetLab. There were other files accessed as well, but browsers don't typically look at robots.txt so that's all we needed to suspect something was wrong with this situation and treated it as a potential attack.

If this was an actual PlanetLab project aimed at crawling the web undetected and aggregate tons of data, then it failed miserably. Now that we know who you are and where you are, our servers will be watching to see if you strike again.

If this was an unauthorized test then PlanetLab better beef up security as this network is one big DDoS attack just waiting to happen under control of the wrong person.

Here's a sample snippet of the log file:

216.165.109.81 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
216.165.109.82 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
216.165.109.79 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
141.161.20.32 - - [11/May/2006:08:45:59 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
138.100.12.149 - - [11/May/2006:08:46:03 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
138.100.12.148 - - [11/May/2006:08:46:03 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
199.77.128.193 - - [11/May/2006:08:46:38 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
199.77.128.194 - - [11/May/2006:08:46:39 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"

Here's the complete list of IP's involved:

216.165.109.81
216.165.109.82
216.165.109.79
141.161.20.32
138.100.12.149
138.100.12.148
199.77.128.193
199.77.128.194
128.31.1.11
128.31.1.15
128.31.1.14
128.31.1.16
138.23.204.232
138.23.204.133
128.31.1.12
128.31.1.13
130.245.145.152
128.83.122.181
128.83.122.180
152.3.138.2
152.3.138.3
128.220.247.28
12.46.129.21
169.229.50.13
169.229.50.10
169.229.50.17
12.46.129.22
169.229.50.8
12.46.129.23
169.229.50.9
169.229.50.12
169.229.50.16
198.133.224.145
129.115.248.225
194.36.10.154
194.36.10.156
128.208.4.199
171.66.3.181
205.189.33.178
193.6.20.4
193.6.20.5
129.170.214.191
129.170.214.192
130.37.198.243
128.143.137.250
128.111.52.62
194.80.38.242
194.80.38.243
131.234.66.161
131.234.66.160
132.227.74.40
169.229.50.18
169.229.50.7
129.186.205.77
155.98.35.4
155.98.35.3
130.104.72.201
130.104.72.200
128.227.56.82
132.252.152.193
147.83.118.123
147.83.118.124
147.83.118.109
147.83.118.125
130.88.203.26
130.88.203.27
193.10.64.36
142.103.2.2
142.103.2.1
193.10.133.128
193.1.201.26
212.201.44.74
133.11.240.56
133.11.240.57
193.167.182.130
132.72.23.11
132.72.23.10
195.116.60.82
195.116.60.83
132.204.102.20
132.204.102.22
130.203.127.40
130.203.127.41
129.242.19.196
169.229.50.11
128.252.19.21
129.242.19.197
129.22.150.105
138.251.214.18
138.251.214.19
128.151.65.101
128.151.65.102
193.63.75.19
134.76.81.241
134.76.81.242
128.59.67.202
130.136.254.22
210.125.84.16
210.125.84.15
140.109.17.181
200.132.0.70
195.116.60.65
204.123.28.53
131.188.44.101
128.8.126.112
128.8.126.69
128.8.126.111
131.246.19.202
163.221.11.73
163.221.11.71
163.221.11.72
193.144.21.130
193.144.21.131
165.230.49.114
165.230.49.115
143.248.139.170
128.232.103.201
128.232.103.203
202.249.37.212
192.33.210.16
193.136.191.26
193.136.191.25
130.49.221.41
192.17.239.251
192.17.239.250
192.41.135.218
192.41.135.219
142.150.3.247
142.150.3.246
200.159.255.70
128.232.103.202
134.226.52.34
134.226.52.35

To make sense of all this mess, I crunched them all thru NSLOOKUP to see if any patterns emerged and what was a common theme was .EDU and PLANETLAB all over the place.

Here's the reverse DNS on all the IPs for your viewing pleasure:

35.52.226.134.in-addr.arpa name = planetlab02.cs.tcd.ie.
81.109.165.216.in-addr.arpa name = planet1.scs.cs.nyu.edu.
82.109.165.216.in-addr.arpa name = planet2.scs.cs.nyu.edu.
79.109.165.216.in-addr.arpa name = planetx.scs.cs.nyu.edu.
32.20.161.141.in-addr.arpa name = planetlab1.georgetown.edu.
149.12.100.138.in-addr.arpa name = planetlab2.ls.fi.upm.es.
148.12.100.138.in-addr.arpa name = planetlab1.ls.fi.upm.es.
193.128.77.199.in-addr.arpa name = planet1.cc.gt.atl.ga.us.
194.128.77.199.in-addr.arpa name = planet.cc.gt.atl.ga.us.
11.1.31.128.in-addr.arpa name = planetlab1.csail.mit.edu.
15.1.31.128.in-addr.arpa name = planetlab5.csail.mit.edu.
14.1.31.128.in-addr.arpa name = planetlab4.csail.mit.edu.
16.1.31.128.in-addr.arpa name = planetlab6.csail.mit.edu.
232.204.23.138.in-addr.arpa name = planet-lab2.cs.ucr.edu.
133.204.23.138.in-addr.arpa name = planet-lab1.cs.ucr.edu.
12.1.31.128.in-addr.arpa name = planetlab2.csail.mit.edu.
13.1.31.128.in-addr.arpa name = planetlab3.csail.mit.edu.
152.145.245.130.in-addr.arpa name = planetlab3.mnl.cs.sunysb.edu.
181.122.83.128.in-addr.arpa name = planetlab3.csres.utexas.edu.
180.122.83.128.in-addr.arpa name = planetlab2.csres.utexas.edu.
2.138.3.152.in-addr.arpa name = planetlab2.cs.duke.edu.
3.138.3.152.in-addr.arpa name = planetlab3.cs.duke.edu.
28.247.220.128.in-addr.arpa name = planetlab1.isi.jhu.edu.
21.129.46.12.in-addr.arpa canonical name = 21.0/25.129.46.12.in-addr.arpa.
21.0/25.129.46.12.in-addr.arpa name = planet1.berkeley.intel-research.net.
13.50.229.169.in-addr.arpa name = planetlab11.Millennium.Berkeley.EDU.
10.50.229.169.in-addr.arpa name = planetlab8.Millennium.Berkeley.EDU.
17.50.229.169.in-addr.arpa name = planetlab15.Millennium.Berkeley.EDU.
22.129.46.12.in-addr.arpa canonical name = 22.0/25.129.46.12.in-addr.arpa.
22.0/25.129.46.12.in-addr.arpa name = planet2.berkeley.intel-research.net.
8.50.229.169.in-addr.arpa name = planetlab6.Millennium.Berkeley.EDU.
23.129.46.12.in-addr.arpa canonical name = 23.0/25.129.46.12.in-addr.arpa.
23.0/25.129.46.12.in-addr.arpa name = planet3.berkeley.intel-research.net.
9.50.229.169.in-addr.arpa name = planetlab7.Millennium.Berkeley.EDU.
12.50.229.169.in-addr.arpa name = planetlab10.Millennium.Berkeley.EDU.
16.50.229.169.in-addr.arpa name = planetlab14.Millennium.Berkeley.EDU.
145.224.133.198.in-addr.arpa name = planetlab1.cs.wisc.edu.
225.248.115.129.in-addr.arpa name = pl1a.pl.utsa.edu.
154.10.36.194.in-addr.arpa name = planetlab1.nrl.dcs.qmul.ac.uk.
156.10.36.194.in-addr.arpa name = planetlab2.nrl.dcs.qmul.ac.uk.
199.4.208.128.in-addr.arpa name = planetlab03.cs.washington.edu.
181.3.66.171.in-addr.arpa name = planet1.scs.stanford.edu.
178.33.189.205.in-addr.arpa name = planet1.ottawa.canet4.nodes.planet-lab.org.
4.20.6.193.in-addr.arpa name = planet1.colbud.hu.
5.20.6.193.in-addr.arpa name = planet2.colbud.hu.
191.214.170.129.in-addr.arpa name = planetlab1.cs.dartmouth.edu.
192.214.170.129.in-addr.arpa name = planetlab2.cs.dartmouth.edu.
243.198.37.130.in-addr.arpa name = planetlab1.cs.vu.nl.
250.137.143.128.in-addr.arpa name = planetlab2.cs.Virginia.EDU.
62.52.111.128.in-addr.arpa name = planet2.cs.ucsb.edu.
242.38.80.194.in-addr.arpa name = planetlab1.cs-ipv6.lancs.ac.uk.
243.38.80.194.in-addr.arpa name = planetlab2.cs-ipv6.lancs.ac.uk.
161.66.234.131.in-addr.arpa name = planetlab-2.cs.upb.de.
160.66.234.131.in-addr.arpa name = planetlab-1.cs.upb.de.
40.74.227.132.in-addr.arpa name = planetlab-01.ipv6.lip6.fr.
18.50.229.169.in-addr.arpa name = planetlab16.Millennium.Berkeley.EDU.
7.50.229.169.in-addr.arpa name = planetlab5.Millennium.Berkeley.EDU.
77.205.186.129.in-addr.arpa name = planetlab-4.ece.iastate.edu.
4.35.98.155.in-addr.arpa name = planetlab3.flux.utah.edu.
3.35.98.155.in-addr.arpa name = planetlab2.flux.utah.edu.
201.72.104.130.in-addr.arpa name = planetlab2.info.ucl.ac.be.
200.72.104.130.in-addr.arpa name = planetlab1.info.ucl.ac.be.
82.56.227.128.in-addr.arpa name = planetlab2.acis.ufl.edu.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-duisburg-essen.de.
193.152.252.132.in-addr.arpa name = planetlab1.exp-math.uni-essen.de.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-due.de.
123.118.83.147.in-addr.arpa name = planetlab3.upc.es.
124.118.83.147.in-addr.arpa name = planetlab4.upc.es.
109.118.83.147.in-addr.arpa name = planetlab2.upc.es.
125.118.83.147.in-addr.arpa name = planetlab5.upc.es.
26.203.88.130.in-addr.arpa name = planet1.manchester.ac.uk.
27.203.88.130.in-addr.arpa name = planet2.manchester.ac.uk.
36.64.10.193.in-addr.arpa name = planetlab2.sics.se.
2.2.103.142.in-addr.arpa name = planetlab2.cs.ubc.ca.
1.2.103.142.in-addr.arpa name = planetlab1.cs.ubc.ca.
128.133.10.193.in-addr.arpa name = planetlab-1.it.uu.se.
26.201.1.193.in-addr.arpa name = planetlab-1.tssg.org.
74.44.201.212.in-addr.arpa canonical name = 74.72/29.44.201.212.in-addr.arpa.
74.72/29.44.201.212.in-addr.arpa name = planetlab2.eecs.iu-bremen.de.
56.240.11.133.in-addr.arpa name = planetlab1.iii.u-tokyo.ac.jp.
57.240.11.133.in-addr.arpa name = planetlab2.iii.u-tokyo.ac.jp.
130.182.167.193.in-addr.arpa name = pl-1.hip.fi.
11.23.72.132.in-addr.arpa name = planetlab2.bgu.ac.il.
10.23.72.132.in-addr.arpa name = planetlab1.bgu.ac.il.
82.60.116.195.in-addr.arpa name = planetlab1.krakow.rd.tp.pl.
83.60.116.195.in-addr.arpa name = planetlab2.krakow.rd.tp.pl.
20.102.204.132.in-addr.arpa name = crt1.PLANETLAB.UMontreal.CA.
22.102.204.132.in-addr.arpa name = crt3.PLANETLAB.UMontreal.CA.
40.127.203.130.in-addr.arpa name = planetlab00.cse.psu.edu.
41.127.203.130.in-addr.arpa name = planetlab01.cse.psu.edu.
196.19.242.129.in-addr.arpa name = planetlab1.cs.uit.no.
11.50.229.169.in-addr.arpa name = planetlab9.Millennium.Berkeley.EDU.
21.19.252.128.in-addr.arpa name = vn2.cse.wustl.edu.
197.19.242.129.in-addr.arpa name = planetlab2.cs.uit.no.
105.150.22.129.in-addr.arpa name = planetlab-2.EECS.CWRU.Edu.
18.214.251.138.in-addr.arpa name = planetlab1.dcs.st-and.ac.uk.
19.214.251.138.in-addr.arpa name = planetlab2.dcs.st-and.ac.uk.
101.65.151.128.in-addr.arpa name = planet1.cs.rochester.edu.
102.65.151.128.in-addr.arpa name = planet2.cs.rochester.edu.
19.75.63.193.in-addr.arpa name = planetlab-2.ic.ac.uk.
241.81.76.134.in-addr.arpa name = planetlab1.informatik.uni-goettingen.de.
242.81.76.134.in-addr.arpa name = planetlab2.informatik.uni-goettingen.de.
202.67.59.128.in-addr.arpa name = planetlab3.comet.columbia.edu.
22.254.136.130.in-addr.arpa name = planetlab2.CS.UniBO.IT.
** server can't find 16.84.125.210.in-addr.arpa: NXDOMAIN
** server can't find 15.84.125.210.in-addr.arpa: NXDOMAIN
181.17.109.140.in-addr.arpa name = planetlab2.iis.sinica.edu.tw.
70.0.132.200.in-addr.arpa name = planetlab2.pop-rs.rnp.br.
65.60.116.195.in-addr.arpa name = planetlab1.piotrkow.rd.tp.pl.
53.28.123.204.in-addr.arpa name = pli1-pa-3.hpl.hp.com.
101.44.188.131.in-addr.arpa name = planetlab2.informatik.uni-erlangen.de.
112.126.8.128.in-addr.arpa name = pepper.planetlab.cs.umd.edu.
69.126.8.128.in-addr.arpa name = planetlab2.cs.umd.edu.
111.126.8.128.in-addr.arpa name = salt.planetlab.cs.umd.edu.
202.19.246.131.in-addr.arpa name = planetlab2.informatik.uni-kl.de.
73.11.221.163.in-addr.arpa name = planetlab-03.naist.jp.
71.11.221.163.in-addr.arpa name = planetlab-01.naist.jp.
72.11.221.163.in-addr.arpa name = planetlab-02.naist.jp.
130.21.144.193.in-addr.arpa name = planetlab.urv.net.
131.21.144.193.in-addr.arpa name = planetlab2.urv.net.
114.49.230.165.in-addr.arpa name = planetlab1.rutgers.edu.
115.49.230.165.in-addr.arpa name = planetlab2.rutgers.edu.
170.139.248.143.in-addr.arpa name = csplanetlab3.kaist.ac.kr.
201.103.232.128.in-addr.arpa name = planetlab1.xeno.cl.cam.ac.uk.
203.103.232.128.in-addr.arpa name = planetlab3.xeno.cl.cam.ac.uk.
212.37.249.202.in-addr.arpa name = planetlab2.koganei.wide.ad.jp.
16.210.33.192.in-addr.arpa name = lsirextpc01.epfl.ch.
26.191.136.193.in-addr.arpa name = planetlab-2.iscte.pt.
25.191.136.193.in-addr.arpa name = planetlab-1.iscte.pt.
41.221.49.130.in-addr.arpa name = planetlab2.cs.pitt.edu.
251.239.17.192.in-addr.arpa name = planetlab2.cs.uiuc.edu.
250.239.17.192.in-addr.arpa name = planetlab1.cs.uiuc.edu.
218.135.41.192.in-addr.arpa canonical name = 218.deleg-192.135.41.192.in-addr.arpa.
218.deleg-192.135.41.192.in-addr.arpa name = planetlab1.csg.unizh.ch.
219.135.41.192.in-addr.arpa canonical name = 219.deleg-192.135.41.192.in-addr.arpa.
219.deleg-192.135.41.192.in-addr.arpa name = planetlab2.csg.unizh.ch.
247.3.150.142.in-addr.arpa name = planetlab02.erin.utoronto.ca.
246.3.150.142.in-addr.arpa name = planetlab01.erin.utoronto.ca.
70.255.159.200.in-addr.arpa name = planetlab1.pop-rj.rnp.br.
202.103.232.128.in-addr.arpa name = planetlab2.xeno.cl.cam.ac.uk.
34.52.226.134.in-addr.arpa name = planetlab01.cs.tcd.ie.
35.52.226.134.in-addr.arpa name = planetlab02.cs.tcd.ie.

Best we can tell it was definitely PlanetLab involved with this and I'm very upset that an organization like this would aim a large section of their network at a single server at the same time without permission.

This is abuse, pure and simple, without proper user agent attribution or anything, and I welcome them to come here and let us know what really happened.

While we're waiting on PlanetLab to respond, and I wouldn't hold my breath, I'm going to block the IPs listed above and probably ban anything with "planetlab" or "planet-lab" in the reverse DNS location name until further notice.

RED ALERT #4 - NiceBot Neighborhood

Found another distributed IP batch sitting in a hosting farm claiming to be "nicebot".

Nicebot my ass...

Here's the range of IP's spotted with user agent nicebot:

69.60.120.165 - nicebot
69.60.120.166 - nicebot
69.60.120.167 - nicebot
69.60.120.168 - nicebot
69.60.120.169 - nicebot
69.60.120.172 - nicebot
69.60.120.173 - nicebot
69.60.120.174 - nicebot
69.60.120.176 - nicebot

NSLOOKUP claims they belong to ServerPronto.

nslookup 69.60.120.169
Server: 64.34.160.92
Address: 64.34.160.92#53

Non-authoritative answer:
169.120.60.69.in-addr.arpa name = 169-120-60-69.serverpronto.com.

So I think I'm going to just block this range from ServerPronto as it's a hosting farm:

Serverpronto INMM-69-60-114-0 (NET-69-60-114-0-1)
69.60.114.0 - 69.60.125.255

Some of you might naively think that you can just block "nicebot" with rewrite rules and solve your problem. However, my research has shown that many of these bots eventually change names when they get blocked by too many sites. You're best off blocking the source permanently so they don't slip thru the cracks next week crawling as something like ""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; BTOW V9.0; SV1)" which you can't detect.

Remember, they're desperate when you cut off their source of revenue and they'll attempt to adapt so use the best prevention up front which is lock them out by location and don't waste your time fighting changing user agent names.

Bots gone WILD!

This is just a follow up on a couple of the bots using distributed IP's I've highlighted recently which just won't take NO for an answer. Ever since their little cluster of scraping IPs has been uncovered and blocked it's still been a non-stop daily request for hundreds of pages per scraper.

These bots are very nasty so if you weren't paying attention the first time, go back and block THIS, THIS and THIS as they are some hungry-assed bots that need to be stopped.

BEZEQINT-HOSTING has a scraper

Coming from the lovely land of Israel is a scraper from Bezeq International, and I can't tell if this is a hosting IP or a DSL connection, but I'm guessing it's hosting but could just be DHCP.

Like I can read their website, feh!

Anyway, the bot always claims to be:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; (R1 1.1); .NET CLR 1.1.4322)

Comes from the following addresses:

82.80.249.201
82.80.249.203
82.80.249.204

So block those at a minimum and 82.80.249.0/24 if you want to be safe.

Ta ta, no scrape for you!

Odd traffic from Hong Kong, Middle-East and Africa

Has anyone noticed any huge spikes in traffic from Hong Kong, Saudi Arabia, South Africa or Dubai lately?

They are setting off alarms all over the place with my bot blocker and it's all coming from shared networks so I can't tell yet if it's just a lot of people using a few IPs or a few crawlers going crazy in a scraper haven.

I'm thinking about just setting the whole bunch of them to "CAPTCHA-mode" which is the equivalent of forcing them to login before accessing my site. This will quickly determine the number source of the activity based on the number of unanswered CAPTCHA's vs. a valid response from a human.

Let's see what happens next, I'll keep you posted ;)

RED ALERT #3 - GoDaddy hosting distributed scraper

This one may have just moved to a new location as I've been watching similar activity before which stopped. These new antics have been going on at this location for a week now and I waited just to make sure it was really coming from a common location which appears to be a block of IPs on some GoDaddy hosting farm secureserver.net.

This creepy crawler doesn't use any user agent string whatsoever and keeps asking for pages like "/#top" and other stupid stuff. Below is the range of IPs and the number of pages asked for just today. You'll note it was a slow day for them asking for only 75 pages, but the day isn't over yet.

68.178.242.111 [ip-68-178-242-111.ip.secureserver.net.] requested 30 pages as ""
68.178.242.126 [ip-68-178-242-126.ip.secureserver.net.] requested 15 pages as ""
68.178.242.128 [ip-68-178-242-128.ip.secureserver.net.] requested 15 pages as ""
68.178.242.127 [ip-68-178-242-127.ip.secureserver.net.] requested 15 pages as ""

Performed an nslookup and got this:

nslookup 68.178.242.111
Server: 64.34.160.92
Address: 64.34.160.92#53

Non-authoritative answer:
111.242.178.68.in-addr.arpa name = ip-68-178-242-111.ip.secureserver.net.

When I did a whois on the IP there came the surprise:

[Querying whois.arin.net]
[whois.arin.net]

OrgName: Go Daddy Software, Inc.
OrgID: GDS-31
Address: 14455 N Hayden Road
Address: Suite 226
City: Scottsdale
StateProv: AZ
PostalCode: 85260
Country: US
178.128.0 - 178.255.255

Now do a whois on secureserver.net:

NetRange: Registrant:
Special Domain Services, Inc.
14455 N Hayden Rd
Scottsdale, Arizona 85260
United States

Registered through: WWDomains.com
Domain Name: SECURESERVER.NET
Created on: 30-Mar-98
Expires on: 29-Mar-12
Last Updated on: 07-Feb-06

Not sure it makes sense to block the entire GoDaddy IP range, so for now 68.178.242.0/24 is all I'm blocking unless I see more rogue activity in their network.

BTW, anyone notice how many sneaky crawler networks I'm busting now that I have proximity alarms in place to spot organized activity?

This proximity alarm is great as it doesn't care if the crawlers ask for 1 page or 100 pages, the minute it detects multiple IP addresses in a similar range doing these things it pops up on my radar. The best thing is that the distributed crawler doesn't even have to use more than one IP address per day as long as they break one of my "bad bot rules" on each visit so the IP is flagged and archived. The proximity report of archived bad bot activity will then expose those archived bots operating from a single location.

Pretty tricky, eh?

You stupid bots better wise up quick, you can't hide behind a bank of IPs, your days are numbered!

Publicly Available Website

That's the current buzzword most often used when you confront someone crawling your site, especially a corporation, that it's a "Publicly Available Website".

Well just because something is publicly available doesn't mean you have the right to do whatever you like with it. It's publicly available for the PUBLIC, meaning visitors, to read individual pages and it's also available to the 6 search engines that I permit to crawl my site. Other than that, just like any other publicly available business, I have the RIGHT TO RESTRICT ACCESS to anyone else that I so desire.

For instance many brick and mortar businesses say "No Shoes, No Shirt, No Service".

Well my website has similar rules "No Humans, No Permission, No Service".

If I even get a whiff off a robot on the site, permission denied.

You corporate and private scrapers just better get over your loser mantra as putting a website online, even on a public network, does NOT give everyone complete access to do whatever they feel like with your site. There are terms of service on that site which distinctly prohibit the use of unauthorized tools to crawl that site, and if you have to ask what's authorized then you don't have permission in the first place so go away.

The site doesn't have a "GNU Free Documentation License", instead it has one of those funny things called a "copyright" which means I own it, not YOU. Additionally, I pay for the server, not YOU. Which means, it's up to ME what is and isn't allowed, even when it's a "Publicly Available Website", NOT YOU!

Let's make it so simple even a 2 year old can understand it:

The website is MINE! MINE! MINE! ALL MINE! and NOT YOURS!

Is that language clear enough for the mental midgets scraping the web to comprehend?