Amazon's A9 Amateur Hour

Guess what boys and girls?

We've all been forced to OPT-IN to yet another non-standard "web tool" that Amazon's A9 has thrust upon us. A9's blog said they introduced this crap last July but it's obviously been so low key compared to everything else hitting my server that I overlooked this small slice of idiocy.

This has been showing up in my logs for a while now"

207.171.167.25 - “GET /siteinfo.xml HTTP/1.1″ 404 1639 “-” “Java/1.5.0_04″

The only reason I noticed it today was the amount of times it hit the server escalated and they're racking up a bunch of 404 errors requesting this file I've never heard about which is idiotic and stupid.

Ever hear of any internet standard such as ROBOTS.TXT to see if I even want you looking for this stupid file on my server?

Apparently not as the only file being hit is "siteinfo.xml".

Had to resort to a reverse DNS lookup just to find out it was iad-fw-global.amazon.com who was doing this stupid crap. Didn't the vaudeville programmers that wrote this joke ever hear of setting the USER AGENT to identify who and what this is instead of Java/1.5?

Amazon, if you happen to read this pay very close attention to the fact that many web applications bombard my server with the user agent of "Java/1.whatever" on a daily basis which are all BLOCKED so you will never ever get access to siteinfo.xml until you properly identify yourself.

Here's a sample "siteinfo.xml" file that you can install in your root web directory:

<?xml version="1.0"?>
<siteinfo xmlns="http://a9.com/-/spec/siteinfo/1.0/">
<webmenu>
<name>Amazon SiteInfo Sucks</name>
<menu>
<item>
<text>Doesn't use standards</text>
<url>http://www.robotstxt.org/</url>
</item>
<item>
<text>Doesn't identify itself</text>
<url>http://www.mozilla.org/build/revised-user-agent-strings.html</url>
</item>
</menu>
</webmenu>
</siteinfo>

I commented about their lack of professionalism and standards being used in this implementation on their blog but it's awaiting moderation and I doubt they'll let my less than happy comments be published, but we shall see.

PlanetLabs Bombards Server - Abused or Compromised?

Well here's a new one that was uncovered this week when a tipster wishing to remain anonymous sent me a very suspicious looking log file snippet with a bunch of identical accesses from over 130 IP addresses ranging over a couple of hours.

After doing a little bit of research it looks like this "attack" came from a consortium of computers called PlanetLab located in various universities and research institutions around the world and this appears to be only a portion of the network that was aimed at our tipsters server. We don't know at this point if this was an isolated demonstration of their network, whether they were being abused by a member or if a hacker has breeched the protocol, but the potential for damage here is huge.

Their website claims the following stats:

PlanetLab currently consists of 668 machines, hosted by 325 sites, spanning over 25 countries. Most of the machines are hosted by research institutions, although some are located in co-location and routing centers (e.g., on Internet2's Abilene backbone). All of the machines are connected to the Internet. The goal is for PlanetLab to grow to 1,000 widely distributed nodes that peer with the majority of the Internet's regional and long-haul backbones.

Below are sample of the log files, IPs involved, and the reverse DNS of all the IPs which is what we used to figure out this was probably PlanetLab. There were other files accessed as well, but browsers don't typically look at robots.txt so that's all we needed to suspect something was wrong with this situation and treated it as a potential attack.

If this was an actual PlanetLab project aimed at crawling the web undetected and aggregate tons of data, then it failed miserably. Now that we know who you are and where you are, our servers will be watching to see if you strike again.

If this was an unauthorized test then PlanetLab better beef up security as this network is one big DDoS attack just waiting to happen under control of the wrong person.

Here's a sample snippet of the log file:

216.165.109.81 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
216.165.109.82 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
216.165.109.79 - - [11/May/2006:08:45:18 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
141.161.20.32 - - [11/May/2006:08:45:59 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
138.100.12.149 - - [11/May/2006:08:46:03 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
138.100.12.148 - - [11/May/2006:08:46:03 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
199.77.128.193 - - [11/May/2006:08:46:38 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"
199.77.128.194 - - [11/May/2006:08:46:39 -0400] "GET /robots.txt HTTP/1.1" 200 452 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050720 Fedora/1.0.6-1.1.fc3 Firefox/1.0.6" "-"

Here's the complete list of IP's involved:

216.165.109.81
216.165.109.82
216.165.109.79
141.161.20.32
138.100.12.149
138.100.12.148
199.77.128.193
199.77.128.194
128.31.1.11
128.31.1.15
128.31.1.14
128.31.1.16
138.23.204.232
138.23.204.133
128.31.1.12
128.31.1.13
130.245.145.152
128.83.122.181
128.83.122.180
152.3.138.2
152.3.138.3
128.220.247.28
12.46.129.21
169.229.50.13
169.229.50.10
169.229.50.17
12.46.129.22
169.229.50.8
12.46.129.23
169.229.50.9
169.229.50.12
169.229.50.16
198.133.224.145
129.115.248.225
194.36.10.154
194.36.10.156
128.208.4.199
171.66.3.181
205.189.33.178
193.6.20.4
193.6.20.5
129.170.214.191
129.170.214.192
130.37.198.243
128.143.137.250
128.111.52.62
194.80.38.242
194.80.38.243
131.234.66.161
131.234.66.160
132.227.74.40
169.229.50.18
169.229.50.7
129.186.205.77
155.98.35.4
155.98.35.3
130.104.72.201
130.104.72.200
128.227.56.82
132.252.152.193
147.83.118.123
147.83.118.124
147.83.118.109
147.83.118.125
130.88.203.26
130.88.203.27
193.10.64.36
142.103.2.2
142.103.2.1
193.10.133.128
193.1.201.26
212.201.44.74
133.11.240.56
133.11.240.57
193.167.182.130
132.72.23.11
132.72.23.10
195.116.60.82
195.116.60.83
132.204.102.20
132.204.102.22
130.203.127.40
130.203.127.41
129.242.19.196
169.229.50.11
128.252.19.21
129.242.19.197
129.22.150.105
138.251.214.18
138.251.214.19
128.151.65.101
128.151.65.102
193.63.75.19
134.76.81.241
134.76.81.242
128.59.67.202
130.136.254.22
210.125.84.16
210.125.84.15
140.109.17.181
200.132.0.70
195.116.60.65
204.123.28.53
131.188.44.101
128.8.126.112
128.8.126.69
128.8.126.111
131.246.19.202
163.221.11.73
163.221.11.71
163.221.11.72
193.144.21.130
193.144.21.131
165.230.49.114
165.230.49.115
143.248.139.170
128.232.103.201
128.232.103.203
202.249.37.212
192.33.210.16
193.136.191.26
193.136.191.25
130.49.221.41
192.17.239.251
192.17.239.250
192.41.135.218
192.41.135.219
142.150.3.247
142.150.3.246
200.159.255.70
128.232.103.202
134.226.52.34
134.226.52.35

To make sense of all this mess, I crunched them all thru NSLOOKUP to see if any patterns emerged and what was a common theme was .EDU and PLANETLAB all over the place.

Here's the reverse DNS on all the IPs for your viewing pleasure:

35.52.226.134.in-addr.arpa name = planetlab02.cs.tcd.ie.
81.109.165.216.in-addr.arpa name = planet1.scs.cs.nyu.edu.
82.109.165.216.in-addr.arpa name = planet2.scs.cs.nyu.edu.
79.109.165.216.in-addr.arpa name = planetx.scs.cs.nyu.edu.
32.20.161.141.in-addr.arpa name = planetlab1.georgetown.edu.
149.12.100.138.in-addr.arpa name = planetlab2.ls.fi.upm.es.
148.12.100.138.in-addr.arpa name = planetlab1.ls.fi.upm.es.
193.128.77.199.in-addr.arpa name = planet1.cc.gt.atl.ga.us.
194.128.77.199.in-addr.arpa name = planet.cc.gt.atl.ga.us.
11.1.31.128.in-addr.arpa name = planetlab1.csail.mit.edu.
15.1.31.128.in-addr.arpa name = planetlab5.csail.mit.edu.
14.1.31.128.in-addr.arpa name = planetlab4.csail.mit.edu.
16.1.31.128.in-addr.arpa name = planetlab6.csail.mit.edu.
232.204.23.138.in-addr.arpa name = planet-lab2.cs.ucr.edu.
133.204.23.138.in-addr.arpa name = planet-lab1.cs.ucr.edu.
12.1.31.128.in-addr.arpa name = planetlab2.csail.mit.edu.
13.1.31.128.in-addr.arpa name = planetlab3.csail.mit.edu.
152.145.245.130.in-addr.arpa name = planetlab3.mnl.cs.sunysb.edu.
181.122.83.128.in-addr.arpa name = planetlab3.csres.utexas.edu.
180.122.83.128.in-addr.arpa name = planetlab2.csres.utexas.edu.
2.138.3.152.in-addr.arpa name = planetlab2.cs.duke.edu.
3.138.3.152.in-addr.arpa name = planetlab3.cs.duke.edu.
28.247.220.128.in-addr.arpa name = planetlab1.isi.jhu.edu.
21.129.46.12.in-addr.arpa canonical name = 21.0/25.129.46.12.in-addr.arpa.
21.0/25.129.46.12.in-addr.arpa name = planet1.berkeley.intel-research.net.
13.50.229.169.in-addr.arpa name = planetlab11.Millennium.Berkeley.EDU.
10.50.229.169.in-addr.arpa name = planetlab8.Millennium.Berkeley.EDU.
17.50.229.169.in-addr.arpa name = planetlab15.Millennium.Berkeley.EDU.
22.129.46.12.in-addr.arpa canonical name = 22.0/25.129.46.12.in-addr.arpa.
22.0/25.129.46.12.in-addr.arpa name = planet2.berkeley.intel-research.net.
8.50.229.169.in-addr.arpa name = planetlab6.Millennium.Berkeley.EDU.
23.129.46.12.in-addr.arpa canonical name = 23.0/25.129.46.12.in-addr.arpa.
23.0/25.129.46.12.in-addr.arpa name = planet3.berkeley.intel-research.net.
9.50.229.169.in-addr.arpa name = planetlab7.Millennium.Berkeley.EDU.
12.50.229.169.in-addr.arpa name = planetlab10.Millennium.Berkeley.EDU.
16.50.229.169.in-addr.arpa name = planetlab14.Millennium.Berkeley.EDU.
145.224.133.198.in-addr.arpa name = planetlab1.cs.wisc.edu.
225.248.115.129.in-addr.arpa name = pl1a.pl.utsa.edu.
154.10.36.194.in-addr.arpa name = planetlab1.nrl.dcs.qmul.ac.uk.
156.10.36.194.in-addr.arpa name = planetlab2.nrl.dcs.qmul.ac.uk.
199.4.208.128.in-addr.arpa name = planetlab03.cs.washington.edu.
181.3.66.171.in-addr.arpa name = planet1.scs.stanford.edu.
178.33.189.205.in-addr.arpa name = planet1.ottawa.canet4.nodes.planet-lab.org.
4.20.6.193.in-addr.arpa name = planet1.colbud.hu.
5.20.6.193.in-addr.arpa name = planet2.colbud.hu.
191.214.170.129.in-addr.arpa name = planetlab1.cs.dartmouth.edu.
192.214.170.129.in-addr.arpa name = planetlab2.cs.dartmouth.edu.
243.198.37.130.in-addr.arpa name = planetlab1.cs.vu.nl.
250.137.143.128.in-addr.arpa name = planetlab2.cs.Virginia.EDU.
62.52.111.128.in-addr.arpa name = planet2.cs.ucsb.edu.
242.38.80.194.in-addr.arpa name = planetlab1.cs-ipv6.lancs.ac.uk.
243.38.80.194.in-addr.arpa name = planetlab2.cs-ipv6.lancs.ac.uk.
161.66.234.131.in-addr.arpa name = planetlab-2.cs.upb.de.
160.66.234.131.in-addr.arpa name = planetlab-1.cs.upb.de.
40.74.227.132.in-addr.arpa name = planetlab-01.ipv6.lip6.fr.
18.50.229.169.in-addr.arpa name = planetlab16.Millennium.Berkeley.EDU.
7.50.229.169.in-addr.arpa name = planetlab5.Millennium.Berkeley.EDU.
77.205.186.129.in-addr.arpa name = planetlab-4.ece.iastate.edu.
4.35.98.155.in-addr.arpa name = planetlab3.flux.utah.edu.
3.35.98.155.in-addr.arpa name = planetlab2.flux.utah.edu.
201.72.104.130.in-addr.arpa name = planetlab2.info.ucl.ac.be.
200.72.104.130.in-addr.arpa name = planetlab1.info.ucl.ac.be.
82.56.227.128.in-addr.arpa name = planetlab2.acis.ufl.edu.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-duisburg-essen.de.
193.152.252.132.in-addr.arpa name = planetlab1.exp-math.uni-essen.de.
193.152.252.132.in-addr.arpa name = planetlab1.iem.uni-due.de.
123.118.83.147.in-addr.arpa name = planetlab3.upc.es.
124.118.83.147.in-addr.arpa name = planetlab4.upc.es.
109.118.83.147.in-addr.arpa name = planetlab2.upc.es.
125.118.83.147.in-addr.arpa name = planetlab5.upc.es.
26.203.88.130.in-addr.arpa name = planet1.manchester.ac.uk.
27.203.88.130.in-addr.arpa name = planet2.manchester.ac.uk.
36.64.10.193.in-addr.arpa name = planetlab2.sics.se.
2.2.103.142.in-addr.arpa name = planetlab2.cs.ubc.ca.
1.2.103.142.in-addr.arpa name = planetlab1.cs.ubc.ca.
128.133.10.193.in-addr.arpa name = planetlab-1.it.uu.se.
26.201.1.193.in-addr.arpa name = planetlab-1.tssg.org.
74.44.201.212.in-addr.arpa canonical name = 74.72/29.44.201.212.in-addr.arpa.
74.72/29.44.201.212.in-addr.arpa name = planetlab2.eecs.iu-bremen.de.
56.240.11.133.in-addr.arpa name = planetlab1.iii.u-tokyo.ac.jp.
57.240.11.133.in-addr.arpa name = planetlab2.iii.u-tokyo.ac.jp.
130.182.167.193.in-addr.arpa name = pl-1.hip.fi.
11.23.72.132.in-addr.arpa name = planetlab2.bgu.ac.il.
10.23.72.132.in-addr.arpa name = planetlab1.bgu.ac.il.
82.60.116.195.in-addr.arpa name = planetlab1.krakow.rd.tp.pl.
83.60.116.195.in-addr.arpa name = planetlab2.krakow.rd.tp.pl.
20.102.204.132.in-addr.arpa name = crt1.PLANETLAB.UMontreal.CA.
22.102.204.132.in-addr.arpa name = crt3.PLANETLAB.UMontreal.CA.
40.127.203.130.in-addr.arpa name = planetlab00.cse.psu.edu.
41.127.203.130.in-addr.arpa name = planetlab01.cse.psu.edu.
196.19.242.129.in-addr.arpa name = planetlab1.cs.uit.no.
11.50.229.169.in-addr.arpa name = planetlab9.Millennium.Berkeley.EDU.
21.19.252.128.in-addr.arpa name = vn2.cse.wustl.edu.
197.19.242.129.in-addr.arpa name = planetlab2.cs.uit.no.
105.150.22.129.in-addr.arpa name = planetlab-2.EECS.CWRU.Edu.
18.214.251.138.in-addr.arpa name = planetlab1.dcs.st-and.ac.uk.
19.214.251.138.in-addr.arpa name = planetlab2.dcs.st-and.ac.uk.
101.65.151.128.in-addr.arpa name = planet1.cs.rochester.edu.
102.65.151.128.in-addr.arpa name = planet2.cs.rochester.edu.
19.75.63.193.in-addr.arpa name = planetlab-2.ic.ac.uk.
241.81.76.134.in-addr.arpa name = planetlab1.informatik.uni-goettingen.de.
242.81.76.134.in-addr.arpa name = planetlab2.informatik.uni-goettingen.de.
202.67.59.128.in-addr.arpa name = planetlab3.comet.columbia.edu.
22.254.136.130.in-addr.arpa name = planetlab2.CS.UniBO.IT.
** server can't find 16.84.125.210.in-addr.arpa: NXDOMAIN
** server can't find 15.84.125.210.in-addr.arpa: NXDOMAIN
181.17.109.140.in-addr.arpa name = planetlab2.iis.sinica.edu.tw.
70.0.132.200.in-addr.arpa name = planetlab2.pop-rs.rnp.br.
65.60.116.195.in-addr.arpa name = planetlab1.piotrkow.rd.tp.pl.
53.28.123.204.in-addr.arpa name = pli1-pa-3.hpl.hp.com.
101.44.188.131.in-addr.arpa name = planetlab2.informatik.uni-erlangen.de.
112.126.8.128.in-addr.arpa name = pepper.planetlab.cs.umd.edu.
69.126.8.128.in-addr.arpa name = planetlab2.cs.umd.edu.
111.126.8.128.in-addr.arpa name = salt.planetlab.cs.umd.edu.
202.19.246.131.in-addr.arpa name = planetlab2.informatik.uni-kl.de.
73.11.221.163.in-addr.arpa name = planetlab-03.naist.jp.
71.11.221.163.in-addr.arpa name = planetlab-01.naist.jp.
72.11.221.163.in-addr.arpa name = planetlab-02.naist.jp.
130.21.144.193.in-addr.arpa name = planetlab.urv.net.
131.21.144.193.in-addr.arpa name = planetlab2.urv.net.
114.49.230.165.in-addr.arpa name = planetlab1.rutgers.edu.
115.49.230.165.in-addr.arpa name = planetlab2.rutgers.edu.
170.139.248.143.in-addr.arpa name = csplanetlab3.kaist.ac.kr.
201.103.232.128.in-addr.arpa name = planetlab1.xeno.cl.cam.ac.uk.
203.103.232.128.in-addr.arpa name = planetlab3.xeno.cl.cam.ac.uk.
212.37.249.202.in-addr.arpa name = planetlab2.koganei.wide.ad.jp.
16.210.33.192.in-addr.arpa name = lsirextpc01.epfl.ch.
26.191.136.193.in-addr.arpa name = planetlab-2.iscte.pt.
25.191.136.193.in-addr.arpa name = planetlab-1.iscte.pt.
41.221.49.130.in-addr.arpa name = planetlab2.cs.pitt.edu.
251.239.17.192.in-addr.arpa name = planetlab2.cs.uiuc.edu.
250.239.17.192.in-addr.arpa name = planetlab1.cs.uiuc.edu.
218.135.41.192.in-addr.arpa canonical name = 218.deleg-192.135.41.192.in-addr.arpa.
218.deleg-192.135.41.192.in-addr.arpa name = planetlab1.csg.unizh.ch.
219.135.41.192.in-addr.arpa canonical name = 219.deleg-192.135.41.192.in-addr.arpa.
219.deleg-192.135.41.192.in-addr.arpa name = planetlab2.csg.unizh.ch.
247.3.150.142.in-addr.arpa name = planetlab02.erin.utoronto.ca.
246.3.150.142.in-addr.arpa name = planetlab01.erin.utoronto.ca.
70.255.159.200.in-addr.arpa name = planetlab1.pop-rj.rnp.br.
202.103.232.128.in-addr.arpa name = planetlab2.xeno.cl.cam.ac.uk.
34.52.226.134.in-addr.arpa name = planetlab01.cs.tcd.ie.
35.52.226.134.in-addr.arpa name = planetlab02.cs.tcd.ie.

Best we can tell it was definitely PlanetLab involved with this and I'm very upset that an organization like this would aim a large section of their network at a single server at the same time without permission.

This is abuse, pure and simple, without proper user agent attribution or anything, and I welcome them to come here and let us know what really happened.

While we're waiting on PlanetLab to respond, and I wouldn't hold my breath, I'm going to block the IPs listed above and probably ban anything with "planetlab" or "planet-lab" in the reverse DNS location name until further notice.

RED ALERT #4 - NiceBot Neighborhood

Found another distributed IP batch sitting in a hosting farm claiming to be "nicebot".

Nicebot my ass...

Here's the range of IP's spotted with user agent nicebot:

69.60.120.165 - nicebot
69.60.120.166 - nicebot
69.60.120.167 - nicebot
69.60.120.168 - nicebot
69.60.120.169 - nicebot
69.60.120.172 - nicebot
69.60.120.173 - nicebot
69.60.120.174 - nicebot
69.60.120.176 - nicebot

NSLOOKUP claims they belong to ServerPronto.

nslookup 69.60.120.169
Server: 64.34.160.92
Address: 64.34.160.92#53

Non-authoritative answer:
169.120.60.69.in-addr.arpa name = 169-120-60-69.serverpronto.com.

So I think I'm going to just block this range from ServerPronto as it's a hosting farm:

Serverpronto INMM-69-60-114-0 (NET-69-60-114-0-1)
69.60.114.0 - 69.60.125.255

Some of you might naively think that you can just block "nicebot" with rewrite rules and solve your problem. However, my research has shown that many of these bots eventually change names when they get blocked by too many sites. You're best off blocking the source permanently so they don't slip thru the cracks next week crawling as something like ""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; BTOW V9.0; SV1)" which you can't detect.

Remember, they're desperate when you cut off their source of revenue and they'll attempt to adapt so use the best prevention up front which is lock them out by location and don't waste your time fighting changing user agent names.

Bots gone WILD!

This is just a follow up on a couple of the bots using distributed IP's I've highlighted recently which just won't take NO for an answer. Ever since their little cluster of scraping IPs has been uncovered and blocked it's still been a non-stop daily request for hundreds of pages per scraper.

These bots are very nasty so if you weren't paying attention the first time, go back and block THIS, THIS and THIS as they are some hungry-assed bots that need to be stopped.

BEZEQINT-HOSTING has a scraper

Coming from the lovely land of Israel is a scraper from Bezeq International, and I can't tell if this is a hosting IP or a DSL connection, but I'm guessing it's hosting but could just be DHCP.

Like I can read their website, feh!

Anyway, the bot always claims to be:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; (R1 1.1); .NET CLR 1.1.4322)

Comes from the following addresses:

82.80.249.201
82.80.249.203
82.80.249.204

So block those at a minimum and 82.80.249.0/24 if you want to be safe.

Ta ta, no scrape for you!

Odd traffic from Hong Kong, Middle-East and Africa

Has anyone noticed any huge spikes in traffic from Hong Kong, Saudi Arabia, South Africa or Dubai lately?

They are setting off alarms all over the place with my bot blocker and it's all coming from shared networks so I can't tell yet if it's just a lot of people using a few IPs or a few crawlers going crazy in a scraper haven.

I'm thinking about just setting the whole bunch of them to "CAPTCHA-mode" which is the equivalent of forcing them to login before accessing my site. This will quickly determine the number source of the activity based on the number of unanswered CAPTCHA's vs. a valid response from a human.

Let's see what happens next, I'll keep you posted ;)

RED ALERT #3 - GoDaddy hosting distributed scraper

This one may have just moved to a new location as I've been watching similar activity before which stopped. These new antics have been going on at this location for a week now and I waited just to make sure it was really coming from a common location which appears to be a block of IPs on some GoDaddy hosting farm secureserver.net.

This creepy crawler doesn't use any user agent string whatsoever and keeps asking for pages like "/#top" and other stupid stuff. Below is the range of IPs and the number of pages asked for just today. You'll note it was a slow day for them asking for only 75 pages, but the day isn't over yet.

68.178.242.111 [ip-68-178-242-111.ip.secureserver.net.] requested 30 pages as ""
68.178.242.126 [ip-68-178-242-126.ip.secureserver.net.] requested 15 pages as ""
68.178.242.128 [ip-68-178-242-128.ip.secureserver.net.] requested 15 pages as ""
68.178.242.127 [ip-68-178-242-127.ip.secureserver.net.] requested 15 pages as ""

Performed an nslookup and got this:

nslookup 68.178.242.111
Server: 64.34.160.92
Address: 64.34.160.92#53

Non-authoritative answer:
111.242.178.68.in-addr.arpa name = ip-68-178-242-111.ip.secureserver.net.

When I did a whois on the IP there came the surprise:

[Querying whois.arin.net]
[whois.arin.net]

OrgName: Go Daddy Software, Inc.
OrgID: GDS-31
Address: 14455 N Hayden Road
Address: Suite 226
City: Scottsdale
StateProv: AZ
PostalCode: 85260
Country: US
178.128.0 - 178.255.255

Now do a whois on secureserver.net:

NetRange: Registrant:
Special Domain Services, Inc.
14455 N Hayden Rd
Scottsdale, Arizona 85260
United States

Registered through: WWDomains.com
Domain Name: SECURESERVER.NET
Created on: 30-Mar-98
Expires on: 29-Mar-12
Last Updated on: 07-Feb-06

Not sure it makes sense to block the entire GoDaddy IP range, so for now 68.178.242.0/24 is all I'm blocking unless I see more rogue activity in their network.

BTW, anyone notice how many sneaky crawler networks I'm busting now that I have proximity alarms in place to spot organized activity?

This proximity alarm is great as it doesn't care if the crawlers ask for 1 page or 100 pages, the minute it detects multiple IP addresses in a similar range doing these things it pops up on my radar. The best thing is that the distributed crawler doesn't even have to use more than one IP address per day as long as they break one of my "bad bot rules" on each visit so the IP is flagged and archived. The proximity report of archived bad bot activity will then expose those archived bots operating from a single location.

Pretty tricky, eh?

You stupid bots better wise up quick, you can't hide behind a bank of IPs, your days are numbered!

Publicly Available Website

That's the current buzzword most often used when you confront someone crawling your site, especially a corporation, that it's a "Publicly Available Website".

Well just because something is publicly available doesn't mean you have the right to do whatever you like with it. It's publicly available for the PUBLIC, meaning visitors, to read individual pages and it's also available to the 6 search engines that I permit to crawl my site. Other than that, just like any other publicly available business, I have the RIGHT TO RESTRICT ACCESS to anyone else that I so desire.

For instance many brick and mortar businesses say "No Shoes, No Shirt, No Service".

Well my website has similar rules "No Humans, No Permission, No Service".

If I even get a whiff off a robot on the site, permission denied.

You corporate and private scrapers just better get over your loser mantra as putting a website online, even on a public network, does NOT give everyone complete access to do whatever they feel like with your site. There are terms of service on that site which distinctly prohibit the use of unauthorized tools to crawl that site, and if you have to ask what's authorized then you don't have permission in the first place so go away.

The site doesn't have a "GNU Free Documentation License", instead it has one of those funny things called a "copyright" which means I own it, not YOU. Additionally, I pay for the server, not YOU. Which means, it's up to ME what is and isn't allowed, even when it's a "Publicly Available Website", NOT YOU!

Let's make it so simple even a 2 year old can understand it:

The website is MINE! MINE! MINE! ALL MINE! and NOT YOURS!

Is that language clear enough for the mental midgets scraping the web to comprehend?

RED ALERT #2 - Distributed IP Scraper on BBCOM

This one is virtually identical to scraper spotted on Vericenter, same profile to the letter.

Claims to be the same exact browser:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1

This spider was seen operating from these IP addresses:

66.234.139.195
66.234.139.200
66.234.139.203
66.234.139.204
66.234.139.206
66.234.139.207
66.234.139.211
66.234.139.213
66.234.139.216
66.234.139.217

This IP range belongs to BBCOM:

OrgName: Backbone Communications, Inc.
OrgID: BBCM
Address: 515 South Flower Street
Address: Suite 4350
City: Los Angeles
StateProv: CA
PostalCode: 90071
Country: US

NetRange: 66.234.128.0 - 66.234.159.255

I'm blocking 66.234.139.0/24 for the moment, but keeping an eye on the rest of their network in case these scrapers switch to a new block.

Another Proximity Alert on PCCW

Getting multiple hits from the same IPs over and over asking for the exact same web page, no images, nothing else, just the same web page day after day. One day the user agent was blank and another day simply"Mozilla/5.0".

This is the range of IPs that just keep repeating the same request.

210.87.251.111
210.87.251.107
210.87.251.41
210.87.251.106

Appears to be coming from HK, can't tell if it's a shared DHCP situation or what, but I'm blocking 210.87.251.0/24 just to be safe.

Another DUMBASS Crawler

This one hails from South Africa, tried to crawl "/#top" as "GET /%23top"

You poor dumb fucker, BUSTED!

RED ALERT - Distributed IP Scraper Hosted on Vericenter

Here's a real sneaky scraper using distributed IPs that is using a bot that almost appears designed to fly under my bot blockers radar. No single IP address accessed enough pages or did anything obnoxious enough to set off any triggers but the collective accesses set off a proximity alarm and they got nailed anyway.

The scraper is pretending to be Firefox for Linux:

http://www.Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060124 Firefox/1.5.0.1/

The range of IP's noticed in this scrape attack are as follows:

65.38.102.138 ip-65-38-102-138.hou.vericenter.com
65.38.102.139 ip-65-38-102-139.hou.vericenter.com
65.38.102.141 ip-65-38-102-141.hou.vericenter.com
65.38.102.143 ip-65-38-102-143.hou.vericenter.com
65.38.102.145 ip-65-38-102-145.hou.vericenter.com
65.38.102.146 ip-65-38-102-146.hou.vericenter.com
65.38.102.147 ip-65-38-102-147.hou.vericenter.com
65.38.102.148 ip-65-38-102-148.hou.vericenter.com
65.38.102.150 ip-65-38-102-150.hou.vericenter.com
65.38.102.153 ip-65-38-102-153.hou.vericenter.com
65.38.102.155 ip-65-38-102-155.hou.vericenter.com
65.38.102.157 ip-65-38-102-157.hou.vericenter.com

The host information is as follows:

OrgName: VeriCenter, Inc.
OrgID: VRCT
Address: 757 N Eldridge Parkway
City: Houston
StateProv: TX
PostalCode: 77079
Country: US
NetRange: 65.38.96.0 - 65.38.111.255

Athough the attack seems to be centered on the 65.38.102.0/24 block at the Houston datacenter of Vericenter, I think I'm going to completely block Vericenter as it doesn't appear to have any ISP facilities [ie. NO HUMANS] and see if anything else bounces off the bot blocker from their facilities.

Another Web 2.0 Scraper Company

Don't roll your eyes and think that Bill's just making a fuss as this company claims they scrape:

Real-time Data Collection - Technologies for crawling, monitoring and scraping newly posted web content including the content from the “deep web."

See, I'm not making this shit up, honest to god admitted scrapers!

Not only that, they proudly display the number of sources they scrape updated constantly on their home page.

They appear to crawl without looking at robots.txt best I can tell, don't identify the source of the crawler other than it's the "Jakarta Commons-HttpClient", and their primary interest in my site seems to be attempting to crawl content referenced from my XML feed.

I'm not sure what information they could possibly think is on my site that could help "Institutional Investors leverage the latest technology and data to make better investment decisions" but they'll just have to be in the dark and use the Magic 8-ball from now own.

They have been seen using these IPs:

206.188.0.11 "Jakarta Commons-HttpClient/3.0"
206.188.0.22 "Jakarta Commons-HttpClient/3.0"
206.188.0.10 "Jakarta Commons-HttpClient/3.0"
206.188.0.23 "Jakarta Commons-HttpClient/3.0"

They are all part of the Geometric Group:

Geometric Group DP-206-188-0-0 (NET-206-188-0-0-2)
206.188.0.0 - 206.188.0.63

I'd just block the whole range and be done with it and hope we don't cause the market to crash.

UPDATE: They switched to Java in 2007!

01/22/2007 206.188.0.22 "Java/1.5.0_03"
01/22/2007 206.188.0.23 "Java/1.5.0_06"

Then mysteriously, stopped pinging my server on 03/15/2007 after a year of being fed garbage.

Think someone finally realized they were getting bounced?

Blue Frog Legs and Spam for Dinner

Normally I don't comment on the news and such but Blue Security shutting down their anti-spam Blue Frog operation is stunning. Obviously the spammers were feeling the pressure and it was working or they wouldn't have attacked, the plan was working. Then in this shocking turn of events the anti-spamming Generals leading the attack not only retreated, they resigned from the effort!

So what if the spammers brought down a few servers and services here and there?

Isn't that the whole idea to get those spamming bastards out in the open so people can track them and block their asses once and for all, put them out of business?

When you start a war you certainly don't pack up and go home the minute you get a bloody nose so I suspect that there's a lot more to this story as people just don't fold so easily from a purely technological war. With all the money at stake in spamming, I'm suspecting the threats got a bit more personal which resulted in the sudden shut down, but that's purely speculating on my part.

If this escalated into seriously dangerous territory, where it seemed to be heading, the big service providers, government and everyone else would've gotten involved and put a permanent stop to those responsible.

Unfortunately you Blue Pussies gave up before we ever got a chance for it to get really interesting.

Maybe someone with balls will step up and continue where you left off.

Port 80 Proxies Expose Themselves

Don't know whoever the fucking morons are writing those stupid fucking proxy servers, but a shitload of them were just blocked today when we noticed they were appending ":80" to the URL and it shows up in the HTTP_HOST parameter.

Normally HTTP_HOST just has something like "www.domain.com" but when the connection is initiated from a certain cluster of proxy servers it shows up as "www.domain.com:80" which is trivial to block.

Saved me a shitload of work tracking down the IPs to block.

Thank you VERY MUCH you stupid fucking assholes!

Search Engine Harvesting

Probably never would've noticed this but the referral string changing set off my referral spam trap. Turned out it wasn't a referral spammer at all but someone crawling my site trying to mine specific topic search engine results from both Google and Yahoo.

Very strange behavior too.

Cloaked as MSIE, downloaded all the images as well to remain hidden, yet crawled so fast it also set off a speed trap.

Stupid stupid stupid.

Dragonfly Crawler

No clue what this thing is and nobody seems to have any real information about it, but it's been seen visiting from 5 different IP addresses.

72.29.233.182 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.183 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.185 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.186 "dragonfly(ebingbong@playstarmusic.com)"
72.29.233.188 "dragonfly(ebingbong@playstarmusic.com)"

Claims to be associated with www.playstarmusic.com (72.29.233.167) and the IP address is so close it's possible.

If anyone has any additional information it would be helpful.

When Innovations Collide

Some web crawler has hit my site a few times called Heritrix which appears to be written mostly by the team at Archive.org, the same team that created ia_archiver for those of you that haven't had your coffee yet.

Yes, it supports robots.txt, but if you didn't know this damn thing existed you wouldn't bother blocking it now would you?

People writing crawlers wonder why webmasters get pissed tracking and opt-ing out all this nuisance crawling on their websites, but I digress, that's an old rant.

The real amusement is that Heritrix claims their technology is designed to "collect the digital artifacts of our culture and preserve them for the benefit of future researchers and generations" which is a bunch of pretty language to try to sidestep downloading a website without permission, especially when the webmaster probably isn't aware of your crawler, doesn't matter how you try to candy coat it.

Now comes the fun part,
let's see who was using it and why!

Today's attempted crawl was HUGE so it's safe to assume this thing has been on my site in the past and apparently the crawler was even banned on a previous bayarea.net IP address:

209.128.119.46 "Mozilla/5.0 (compatible; heritrix/1.6.0 +http://innovationblog.com)

Today the crawler used a different bayarea.net IP, could be DHCP, could be on purpose to sidestep the previous ban. Who knows, but it only got a couple of pages before the doors were automatically slammed by the bot blocker:

209.128.119.17 - "Mozilla/5.0 (compatible; heritrix/1.6.0 +http://innovationblog.com)"

With my curiosity in overdrive, it was time to research innovationblog.com and see why they were crawling my site. Not a clue as there's nothing but a "This Web Site Coming Soon" site under construction page, but the WHOIS for the site was very revealing.

Registrant:
Michael Osofsky
1758 Shoreline Blvd. Suite B
Mountain View, California 94043
United States

Registered through: GoDaddy.com
Domain Name: INNOVATIONBLOG.COM
Created on: 09-Mar-05
Expires on: 09-Mar-07
Last Updated on: 06-Mar-06

Administrative Contact:
Osofsky, Michael mosofsky@accelovation.com
1758 Shoreline Blvd. Suite B
Mountain View, California 94043
United States
(650) 968-4741 Fax --

Technical Contact:
Osofsky, Michael mosofsky@accelovation.com
1758 Shoreline Blvd. Suite B
Mountain View, California 94043
United States
(650) 968-4741 Fax --

Domain servers in listed order:
WSC1.JOMAX.NET
WSC2.JOMAX.NET

This Michael seems to be involved with a company called accelovation.com and he seems to be big in the innovation circles having founded the MIT Innovation Club.

According to the Accelovation website:

Accelovation is the first and only Market Discovery System (MDS) that allows innovators to mine the online world for insights into unmet needs, trends, innovations and market activity.

Sound familiar?

We crawl you and use your information without permission to make a profit.

Where have we heard this before?

I'll bet they'll be surprised at my attitudes about this but they should try reading some webmaster forums and find out what they're doing probably isn't welcome without permission, some clue posted about what the benefits are to the webmaster to allow his site to be crawled, yada yada yada we've been down this path a few times before, it's getting old.

Sorry, but your innovation collided with my innovation called a bot blocker.

Your crawl is denied, and thanks for playing.

My Family is CRAWLING?

Call me confused, but why is MyFamily.com crawling my site?

Yup, the genealogy people, that's them.

66.43.16.199 [nat.myfamilyinc.com] has requested 110 pages as Mozilla/4.0 (compatible; MyFamilyBot/1.0; +http://www.myfamilyinc.com)

Even claims to be them, looked at robots.txt like they were supposed to, yada yada.

Still went home with a pile of crap instead of my pages.

Put that on your family tree and smoke it.

John has an interesting theory

I'm not sure if any of you have ever seen John Gabriel's Greater Internet Fuckwad Theory but take a gander, quite amusing.

Just in case one of you thinks "Well, that explains Bill in a nutshell!" then think again as I've always been very vocal, blunt, and to the point.

Doesn't mean I'm not a fuckwad, but the internet had nothing to do with that!

Snared a Layered proxy web host

Yup, that's right, someone decided we need to hide on the net so much they actually created a Proxy Web Host and it appears scrapers are using them, what a shocker.

I found them as AdSense was trying to crawl thru the proxy:

BAD_AGENT: 72.232.31.226 [prx1.proxywebhost.com.] requested 3 pages as "Mediapartners-Google/2.1"

So who are these people using for their provider?

Layered Technologies
72.232.0.0 - 72.232.255.255

That's a HUGE block of IP's to just block out of hand, so how much abuse has been coming from this range? Let's search on "72.232." and see what pops up.

First, it appears I already banned a c-block over there running a multi-IP scraper trying random user agents:

BANNED=72.232.67.222 yrkqi3jrmnbrsk3mUpnrwung
BANNED=72.232.67.219 vhsflbuwvLsbwmyp8xse8hvpdpplLdxdx
BANNED=72.232.67.219 utgkm gylmugtdblyppqqu
BANNED=72.232.67.220 pt tkglswaqatq k rfxqolbtqbygxlhvS0qqv
BANNED=72.232.67.221 djpqaegrbxpfbqnkxvqeniqfogyb rnt
BANNED=72.232.67.221 wbdprvjiqbw jbsvqse7
BANNED=72.232.67.220 upehrsqqqevdljtwrgkkbthk e
BANNED=72.232.67.220 sjxgohdtum3yybmbfyembisbxibei
BANNED=72.232.67.219 7jrxquabdwlgn wyjnoxtyxdryvffjbVdjw
BANNED=72.232.67.219 umesjoxmwrwdvjeqmfsreYfenxqel6d
BANNED=72.232.67.219 kdxiqiyu3yicfupymhimbp nlb v oghtqre
BANNED=72.232.67.222 henlvvdiranneq0cddlfdiXeivbwylon bxic
BANNED=72.232.67.222 vdpPPvxlkwmwpPyy8gpshni8y dwe q8lewlhfl
BANNED=72.232.67.219 didII6ye6It wermhvcx 6jmwcblyxj
BANNED=72.232.67.219 jevltpcioxefrooirvcumd
BANNED=72.232.67.222 r8nawcyepuDfymmbdi8xdsfah8sfqkwhuy eu

Why were they banned?

On a different day they claimed to be this:

72.232.67.221 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.219 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.220 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
72.232.67.222 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"

Reverse DNS claimed it was galaxy-webhosting.co.uk which is in Layered's IP block.

Now, for your amusement, here's the same IP within an hour trying more than one user agent:

00:14:06 72.232.67.222 "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"
00:52:07 72.232.67.222 "plblilwkchhs2qfkv rbXbgveu xsxwsxauspuX"

Hey, if one user agent doesn't work, spin the roulette wheel, right?

Sorry idiot, NO user agents work on my site, so let's move along.

OK, at least this one was creative, someone decided to explain it was a User-Agent:

72.232.77.34 "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;"

Just a few garden variety scrapings:

72.232.52.58 "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)"
72.232.185.170 "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"

Then another random attempt on .58 from above:

72.232.52.58 "qyerqcuylypknmarpuoudyeawwft"
72.232.52.58 "blcrnhbulYypqfbtasciqc"
72.232.52.58 "pwifascemkr4abimihq4ybhbusv"
72.232.52.58 "hbcudylrrturxtxwtMhoqq9sMsr uw pfM"
72.232.52.58 "brn jxvcgitdurvqhivtrhthtknu"
72.232.52.58 "fxddbq qxduqghdpbdgnptqrCtioive"
72.232.52.58 "jni0 kjn0flJxuenr0oek0b0rpjx"

It's just so cute that they fucking don't get it, random user agents or valid user agents, you just keep knocking but you can't come in and play so piss off.

Another proxy event that I banned:

72.232.20.146 "Mediapartners-Google/2.1"

Legit spiders crawling outside their range just scream "BLOCK ME! PROXY!", gotta love it.

Then this idiot thought no user agent would work, WRONG!

72.232.13.2 ""

... and a bunch more IPs doing stupid shit, but I'm too lazy to list 'em all here

Word to the wise, it looks like a scraper haven over there so consider blocking it.

According to their web site it looks like all server hosting so probably safe to block the whole range, but they have provided some amusement with their vaudeville scraper show thus far so maybe I'll just keep an eye on them for now and see if they come up with something new to toss at the bot blocker.

Circle Jerk Scraping

Not sure what's going on with some of the scrapers exactly but it seems my "CHALLENGE" page, which has about 5 links on the page, is sending them in circles asking for the same pages over and over.

Lately I'm seeing a lot of lost scraper bots doing something like this:

GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
... and so on and so forth...

Sometimes they'll ask for hundreds of pages like this before breaking out of that loop.

Quite amusing that I seem to have put a kink in their gears.

Scraping is fucking COMCASTIC!

That's right, most of the assholes abusing my website are using COMCAST!

Let's explore what happened today:

• CHALLENGE: 68.44.91.83 [c-68-44-91-83.hsd1.nj.comcast.net.] requested 229 pages as "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"
• SPIDER: 70.88.200.205 [70-88-200-205-measured-progress-ne-ma.hfc.comcastbusiness.net.] requested 75 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
• CHALLENGE: 68.83.187.18 [c-68-83-187-18.hsd1.nj.comcast.net.] requested 59 pages as "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"
• CHALLENGE: 24.63.11.72 [c-24-63-11-72.hsd1.ma.comcast.net.] requested 25 pages as "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
• CHALLENGE: 24.0.49.82 [c-24-0-49-82.hsd1.tx.comcast.net.] requested 44 pages as "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+98;+Win+9x+4.90)"
• SPEED: 68.53.108.148 [c-68-53-108-148.hsd1.tn.comcast.net.] requested 477 pages as "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)"

Let's explain what the reasons were why they were blocked which is the FIRST WORD shown before their information.

CHALLENGE means that after 20 pages downloaded there was a challenge put out that a human could answer but the computer didn't, and it just kept on crawling getting challenges page after page.

SPIDER means they stepped in a spider trap or read robots.txt, something humans don't do.

SPEED means just that, the pages were downloaded faster than Superman could click on them, even with a Kyptonite hangover.

Add it all up and thats 909 pages of COMCASTIC scraping!

That's 1/3 of the abuse to my website coming from a single internet service provider.

Fucking lovely, is it Scraping OnDemand too or is it High Definition Scraping?

Just take your fiber optic cables and give yourself a colonoscopy.

Omni-Crawler now USEFUL?

I was just looking around to see if there was any change in the status of a few old crawlers that never seemed to do anything useful, and suddenly Omni-Crawler's web site claims they actually have a customer!

Holy crap, someone uses it?

They say that their little crawler fuels Vast which looks like a pale Oodle wannabe.

Well guess what?

I don't have any classifieds on my site so leave the shields up set omni-crawler on condition red.

Would hosts care?

What if every time you had an attempted scraping your bot blocker sent an email detailing the specifics to the abuse dept. of the company hosting the scraper?

Would they even care if it weren't for the sudden flood of automated messages?

I'm thinking it might be amusing to give it a shot and see if they block any of these idiots.

Scraping My Big Pipe

And I'm not talking about shaving my cock either, I'm talking about some scrapers running via bigpipeinc.com. From their web site it all looks like pro pipelines but I'd hate to whack an entire network just because of one business connection behaving badly.

Anyone else know anything about these guys?

Can't disable it even to upgrade, this is CRAZY!

To easily performs some upgrades I took my bot blocker offline for a couple of hours. This way I could update the tracking database and software without worrying about accidentally killing the site. Then I completed installing the latest changes and turned the bot blocker back on and the site was running slower than shit.

My first reaction was panic that I'd busted something big time but my local version was faster than hell, so I looked into the current server accesses .... holy crap!

Guess what, while the bot blocker was offline for less than 2 hours, some asshole came and overloaded the fucking server when I wasn't watching. The running task list was enormous, the number of concurrent page requests was HUGE!

I punted the asshole's current tasks off the server and the bot blocker took over from there and restored calm to the website.

Seriously though, it's pretty scary when I can't even disable the bot blocked for a brief time without all hell breaking loose. Kind of reminds me of someone I knew that got hacked in less than 5 minutes while trying to replace a firewall with the machine still stupidly connected to the net.

Guess next time I'll either have to work faster or come up with an update plan that has no down time.

More work, less blogging, less forums

Yup, it had to happen, and people obviously noticed as I'm getting PM's here and there "nice to see you posting again" or "where have you been hiding?". Not like I ever went away, just doing work on trying to make my stupid bot blocker actually see the light of day and doing some R&D with a couple of new concepts that have been very promising.

So if you think I'm slowing down, it's just the opposite, as things are seriously HEATING UP!

If you catch me posting too much, smack me and tell me to get back to work ;)

WARNING - User Agent Script Vulnerability

Don't know if anyone has ever seen this script vulnerability before but it just happened to me and I almost shit myself.

Who would ever consider that pulling up your log analyzer summary page could fuck you over?

Luckily for me, it was just a simple redirect:

203.214.25.21 "<SCRIPT>window.location='http://www.syncrisis.com'</script> (compatible; MSIE 6.0; Windows NT 5.1; SV1; iP-CMPQ-2003; .NET CLR 1.1.4322)"

These fuckers better watch out cause this is fucking BULLSHIT and you're on my shitlist now you cocksuckers.

Something new to add to the bot blocker and just bounce assholes that do this shit.

First sighting of predicted next evolution in bots.

Remember a while back I predicted that bots would evolve to use random IPs and random user agent strings and it's already happening. Good thing I had already planned for this contingency and was ahead of the curve waiting for them when they tried this trick.

Not 100% positive, but it looks like it all kind of started with this very prolific bot called T8Abot/v0.0.7-beta (3724461@gmail.com) which seems to get around.

The most I can find about it without wasting too much time is this reference:

unknown bot, hosted by FDC Servers, fdcservers.net, US. Massive operation using many IP addresses (66.90.110.199 ... 66.90.110.254)

That bot seems to have gone away, at least it doesn't visit my site anymore, maybe they noticed it was being blocked by user agent string.

However, it didn't go away and they have changed their tactics as I'm seeing the following [small sample] from the same IPs:

06:44:13 66.90.95.225 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
06:44:32 66.90.95.249 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:45:44 66.90.95.243 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:45:49 66.90.95.249 "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1"
06:45:53 66.90.95.249 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
06:45:54 66.90.103.69 "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1"
06:45:59 66.90.95.225 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.54 [en]"
06:46:07 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:46:10 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:46:19 66.90.103.69 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Media Center PC 3.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
06:46:27 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50"
06:46:28 66.90.95.243 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:46:37 66.90.95.245 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; InfoPath.1)"
06:46:39 66.90.103.69 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:46:40 66.90.95.216 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
06:46:41 66.90.95.211 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; ru) Opera 7.60"
06:46:43 66.90.95.211 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.54 [en]"

Lots of requests per minute coming from a dedicated hosting operation, doesn't appear to be dial-ups or broadband, and the same IPs used to claim to be T8Abot but I think they unleashed this random IP/random user agent that they thought would fly under my radar.

Too fucking bad, I built a better mousetrap, you STILL got caught, try again.

I would block everything in the fdcservers.net range: 66.90.64.0 - 66.90.127.255

The important thing with this particular excercise that my long term profiling strategy is paying off by extended profiling of bad neighborhoods. Once IPs start to set off a few traps it becomes obvious where the problem children are as they unwittingly expose themselves.

Scrape-a-thon endurance test

Just for giggles I tweaked the parameters of the bot blocker a few weeks ago to see if I could snare any super slow crawlers trying to fly under the radar.

So far, the grand prize winner has been scraping a few pages daily for 13 days now for a grand total of 1567 pages.

What a loser, sheesh.

Unfortunately, it looks like this latest change has snared about 20-30 innocents so we'll be retooling it today, but I learned a lot about slow scrapers during this process and it was quite an eye opener.

DDoS against rogue bots and scrapers!

Don't know if anyone has been following the trials and tribulations of Blue Security and their anti-spam tool the Blue Frog with over 400K members, but their technique could be useful against scrapers. Basically what Blue Frog does is automatically send opt-out requests to spammers from every member in their network which overloads the spammers servers and sends them crashing or maybe even blocks their ability to spam while the pipeline is choked with inbound requests.

So this concept stewed in my brain for a couple of days and suddenly it occurs to me that bots could be stopped the same way with a large enough network of anti-bot members. The concept is simple in that any active crawlers caught in the act could simply be pinged to death for a brief period to stop them with brute force. That's right, we could aim thousands of servers at unwanted active crawlers and literally DDoS them into leaving us alone.

Worse case, they aren't crawling when their pipeline is choked.

Best case, they take a hint and stop hitting member sites.

That method in itself COULD make an entire community of networked webmasters deal death blows to scrapers, blog spammers, and all sorts of nasty vermin.

Hell, why not fight fire with fire and use the tools of the underground against them?

If you can locate a scraper's site we could also just deploy spiders to simply crawl them offline. That's right, do the same thing they do and crawl them so hard and fast the damn server can't even respond to requests. Hit them with so many different servers at the same time they can't even identify who's doing it in time to stop it.

Remember, if visitors can't get to your web site then the purpose of scraping to build the website becomes meaningless.

Here's where I think this strategy might pay off as scrapers on shared servers just might get the boot when the host figures out that they're the reason for the attacks. Likewise, colo facilities might even boot dedicated servers when a network of unhappy webmasters retaliate and choke a major hosts data pipe repeatedly.

Does is sound like vigilanteism?

HELL YES IT DOES!

Why shouldn't we do it?

Nobody else is helping us as neither their web hosts, the copyright laws, Google AdSense nor Yahoo Publisher Network seem to be interested in helping us stop this plague so maybe it's time to deploy full blown internet warfare tactics like Blue Security in order to stop the madness!

In the immortal words of Mel the cook on Alice:
"The best DEFENSE is a good OFFENSE!"

Crawlers of the Dumbed

OK, so I slapped some javascript in all my links just for shits and giggles to make sure it was REALLY a human and sure enough some dumb fuck is now attempting to crawl my site with "OnClick", "OnMouseDown", and other stupid shit as part of the URL.

My god the 12 year olds writing these crawlers aren't as smart as 12 year olds used to be.

FrontPage theft THWARTED

Lazy assed stealing webmaster used to just aim FrontPage at my site and lift a web page, steal layouts, whatever they did, but that was THEN.

THIS is NOW and my bot blocker slammed their head into the wall:

BAD_AGENT: x.x.x.x requested 1 pages as "MSFrontPage/4.0"

Too fucking bad, learn to code your own goddamn webpages you fucking freak!

Blogger ALSO Busted all to Hell

In more news of broken Google property, Blogger's edit mode has been FUCKED for a couple of weeks and I assumed someone would fix it but I was sadly mistaken.

It used to work just fine but most of the time now when I edit a previous post in Firefox or Netscape it just hangs as the Blogger code spins out of control and I have to kill the browser. The only way I can edit in those two browsers most of the time, starting a couple of weeks ago, is to copy the entire article into a new post, edit and save which seems to work.

When you edit in Blogger on Internet Explorer it's doing something completely different and chews up articles, trashes the HTML, jacks up fonts and all sorts of fun shit.

Basically, other than the original first time posting, using Blogger to edit in it's current state can be a death sentence to your post.

Based on all the crap posted in Blogger by illiterate people that can't even tell when they make actual spelling mistakes, let alone attempt to edit them, this problem probably hasn't even been reported as nobody else has noticed.

Funny, I don't see the word BETA on Blogger and if it was, it would be a piss poor Beta at the moment.

Google Exact Phrase Search is BUSTED ALL TO HELL!

Have you done any exact phrase searches in Google lately?

It's serving up all sorts of off topic shit, I mean, it's really busted.

Couple of examples of topics from this blog:

Do a search on "Lava Pockets" and you get crap liks this in listing #13:

Take Five - The Boston Globe(Correction: Because of a photographer's error, the names of two members of the Click Five were transposed in some edition's of Sunday's Arts ...
www.boston.com/ae/music/articles/2005/08/07/take_five/ - 41k - Supplemental Result - Cached - Similar pages

Neither the word LAVA or POCKET are even on that page

Even better is "Florida Drinks Cat Piss" with 99% of the results complete shit!

Let's examine the #2 result:

eg Forums -> Pretentious bottle--ordinary wineOutline · [ Standard ] · Linear+. Pretentious bottle - - ordinary wine. Track this topic | Email this topic | Print this topic ...
forums.egullet.org/index.php?showtopic=72214 - 71k - Supplemental Result - Cached - Similar pages

That's right, nothing on that page about Florida whatsoever but the irony of wine coming up doing a search for cat piss certainly didn't pass me by.

Hey Google, you even know about SQA?

That's Software Quality Assurance for those that don't know, and I can recommend some very good people if you need them down at Google.

Some freaky shit

OK, today when I was taking a shower a very loud noise happened in the bathroom, nearly jumped out of my skin. Looked out and the door was still shut, so the cat hadn't been in the room and the wife was at work.

When I got out of the shower it turned out a tall perfume bottle, one that has sat in the same spot for ages, just randomly fell over and bounced on the counter top.

What the fuck?

I immediately thought perhaps it was an earthquake jolt but the dining room chandelier wasn't swinging, and it swings for the mildest of jolts, so I'm not sure what's going on at this point.

Just what I need, ghosts in the crapper, nothing the aftermath of a good mexican lunch won't scare away.

Jews for sale!

Who'll start the bidding at 10 shekels, do I hear 15?

That's right, some dumb fuck didn't filter out the plural of kike on Google Adwords and some dumber fuck is now trying to sell them on eBay.

Oy vey!

Microsoft Promotes Web Theft

Microsoft is promoting a whole page full of offline browsers just to help people steal shit.

They gleefully promote it too:

Quickly and efficiently download entire sites or sections of a site for viewing and browsing when you are not connected to the Internet.

Just what we need is a major corporation sticking some shit like this in front of people's faces that may not even be aware such tools existed in the first place. That's right, America's largest software company has become the Pied Piper of scraping websites.

Any words of caution about copyright, that your IP may get banned for abusive practices or anything else?

Hell no.

Hey Microsoft, you gonna pay for my web hosting bill when the assholes that download all these offline browsers blow my fucking server out of the water with ridiculously high bandwidth abuse?

I didn't think so.

Well, fuck you Microsoft, just fuck you.

India Link Whores Just Don't Understand

It's the spring link season once again and all the India link whores are out doing their ritualistic website link mating dance. Unfortunately, the shit they want me to link to would be like trying to mate a monkey with a goat, wrong species pal, nothing going to come from this attempt to defy nature.

Here's a slightly modified version of the most recent link blackmail to conceal the identity of the clueless asshole:

Hi webmaster,

We have already added your site to our resource page
http://www.OFFTOPICBULLSHIT.com/ long back.

However, we just visited your site and we still do not find the reciprocal link to our site. If we do not found our reciprocal link we will remove your site link from our resouce page. Please confirm the url where the link is posted as soon as possible to allow us keep your link.

Title :- Spamming link whores at nominal rates.

Description :- We don't fucking understand that people don't like off topic links, link to us or we'll add your link to Jihad.com and then you'll be screwed

URL :- http://www.REALANNOYINGFUCKERS.com

We look forward to receiving this information promptly.

Thank
Stoopidja Dumbfukja
webmaster@IMFUCKINGCLUELESS.com

Not only was the site completely off topic, the whole fucking thing was ORANGE, even a significant amount of the text!

My eyes started bleeding the minute I saw that shit and I couldn't tell if my link was on their site or not, but it didn't really matter as I told them to take their link and shove it up their ass sideways.

Crawl my site, go to jail, it's the law.

Who the hell put open season on my website?

Not talking about this blog, but the website that pays my bills and shit.

It's NOT a free articles site, plainly posted copyright notices, technology is in place to stop assholes dead in their tracks and they just keep coming.

Did someone post a sign on the internet that I've never seen that says:
"HEY! FREE SHIT TO SCRAPE HERE! COME AND FUCKING GET IT WHILE IT'S HOT!"

The other day I wrote some dry ass bullshit about data mining and then followed it up with some boring assed statistics I've been collecting but suddenly something changed and the amount of thieves hitting the site is suddenly going off the charts compared to a month ago.

Let's put this into perspective:

If my website were a grocery store and these assholes were looters or shoplifters the shelves would be bare, the clerks would be guarding the doors with shotguns, the police would have the place surrounded and the parking lot blocked off and a riot squad would be firing rubber bullets and cracking skulls of people trying to get away with th loot.

The local jails, needless to say, would be overflowing.

I realize there is no physical theft involved like the grocery store example above, and I also realize putting something out on a public network invites a certain amount of risk, and lunatics from the fringe, but when you install barriers and roadblocks to stop that activity and they just keep coming it's beyond and above what falls under 'normal access' and is well into some serious realms of abuse and harassment.

But this is just 'copyright infringement' and 'bandwidth theft', right?

Well, maybe it is when you identify yourself as robot and behave in an acceptable way that allows me, the webmaster, to stop you with reasonable efforts. However, when you mask to conceal the nature of your visit, change the identity of the crawler, and then attempt to crawl undetected and bypass mechanisms such as firewalls put in place to stop your activity, then TECHNICALLY this becomes hacking.

Isn't hacking to gain unwanted access a FUCKING CRIME?

I'm thinking someone needs to make a test case on this and just bypass copyright altogether and try filing a criminal complaint against them for hacking and see what happens.

I have a simple case I could use to test this theory already, as initially I put simple roadbloacks in place for humans to get past just waiting to see when the bandits would code around it, and sure enough they did a few months later. Then of course I made it harder and I'm waiting to see if they'll take a shot at bypassing this roadblock as well. Don't worry, there are deadbolts I can install to keep them out if needed, but my current cat and mouse game is more fun and useful to learn from as they expose the level of their technology.

OK, since I put a lock on the door that stopped the unwelcome technology that was hitting my site and someone deliberately programmed to bypass it, isn't that technically hacking?

I'm thinking it's just about time to drop some money on a lawyer to see where this idea stands within the definition of the U.S. laws on hacking as a crime as it would be nice just to scare the shit out of the local scrapers.

Not to mention it would sure be a cool to put a logo on my site that basically explains to these assholes:

"Crawl my site, go to jail, it's the law."

Who am I kidding?

Some asshole in some foreign country would just start a whole new business selling scraping services or pre-scraped websites if they aren't doing it already.

Then we're right back to fighting copyright infringement.

PicGrabAss Image Theft Technology

Some fuckers on a website far, far away have a product called PicGrabber that let's you quickly and easily crawl a website and steal every picture, movie, and any other goddamn thing that isn't bolted to the walls and nailed down to the fucking floor.

Their slogan is:

PICgrabber - The software that finds millions of free movies and free pictures for you!

And catch this "feature" list:

• Scans the web with Keywords
• Scans any given start URL
• Downloads all found Images / Movies / MP3 automatically'

Shows up in your log file as:

Mozilla/PICgrabber - (http://www.movies-free.net/)

Luckily their little grand theft tool got nothing but bitch slapped with error messages.

Hey PicGrabber, try my slogan: FUCK YOU!

Live Servers Update

The more IPs I block from this group, they just seem to move to a new block of IPs. All are being hosted by live-servers.net and they just using block after block and keep on coming.

The current ranges being blocked are as follows:

88.208.192.
88.208.193.
88.208.194.
88.208.195.
88.208.200.
88.208.202.

These ranges never send any traffic with the exception of multiple crawlers from the same block at the same time and all IPs appear to be pointing back to a server farm so I'm recommending you just block 'em.

More IP updates on this group as they become available.

Where are all the kick ass PHP programmers?

Now that I'm ready to actually build my bot blocker for commercial purposes it seems like it's next to near impossible to find any quality PHP programmers, at least none that aren't already booked thru 2010, to help build the damn thing.

For those thinking it's going to be some simple piddly-assed fire-and-forget script think again as we're talking a product of much larger scope, including a centralized server component. The scope is more that just the "bot blocking" that I rant about here as it will be a complete crawler management and protection system. It will allow webmasters to manage access to their site without having to deal with the technical aspects, nor keep up with the latest crawlers. Additionally, there is a security component that will harden web sites against a variety of exploits.

I was prepared to just pay someone to convert it to PHP, or even be an equity partner if they wanted some of the back end action, but it's starting to look like I may just have to roll up my sleeves and do it all myself. That's going to throw a nasty monkey wrench into the time frame as there is still pending R&D that needs to be done.

So much to be done, so little time, and a planned summer BETA release not looking so good at the moment.

Blocked Spiders DO NOT Go Away

There are a few bold, albeit naive, statements by other so-called "bot blockers" that scrapers just go away after you deny them a few pages which is complete and utter BULLSHIT!

Some of the scrapers being blocked on my server have been set to BANNED for months now, haven't gotten a single page of value, yet they just keep coming over and over, attempting to get pages they remember regardless of the outcome.

Most bot blockers I've reviewed just set speed traps or page limits and then throw a captcha in their face to make them go away for a brief period of time, maybe a few hours, maybe a day or two, but many of them will come back over and over and get another chunk of pages when they return. The stakes are high and the scrapers want your content badly so putting silly little bandages on your website for short term solutions do not cure the long term problems.

The only way to truly stop them is to profile their behavior over time as my bot blocker throws all first-time suspected bot IP's into QUARANTINE. Once an IP makes it into quarantine they are immediately suspended for 24 hours and then challenged immediately when they return to the web site after 24 hours. This stops repeat offenders from getting any pages whatsoever when they return and also protects against permanently blocking a DHCP address by accident that is used to scrape only once. After a couple of repeated scrape attempts without breaking thru the challenge, which a human can easily do, the site is escalated from quarantine to BANNED which no longer presents challenges and just gives error messages on repeat visits.

Not rocket science but it has a lot more finesse than some of the more simplistic methods others employ and better hardens the site against repeated attempts at scraping.

Gosh Goes Wild

I blocked one IP address and they came back with a new IP address 72.51.37.210 doing their sneaky masked crawl bullshit.

Go away little crawler, stay away little crawler, or this WILL get ugly when I start cloaking 'YO MAMMA' snaps into your fucking search engine for major keywords like: "Pool Cleaning Services - Yo Mamma so stupid she saw shit in the pool and thought the anal porn queen was trying to teach her baby to swim!"

BTW, fix your dumb fucking crawler so it can properly parse a goddamn web page as hitting my server for "GET / <font" is about as bright as asking for 30 pages in a minute while claiming to be "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Stupid fucking shit really gets on my nerves.

Alexa Hiding in the Shadows

What the hell's going on with Alexa trying to crawl my site with NO user agent string!

Had an attempted anonymous crawl from 209.237.238.224 which is:

whois 209.237.238.224
Alexa Internet ALEXA-INTERNET (NET-209-237-237-0-1)
209.237.237.0 - 209.237.238.255

They still read the robots.txt file, but no clue who it was:

209.237.238.224 - - "GET /robots.txt HTTP/1.0" 200 111 "-" ""

What the hell's going on Alexa?

Someone break the crawler or you just trying to sneak under the radar after everyone blocked your ass?

You claim to crawl as "ia_archiver" but I don't see any ID here whatsoever!

Please explain, inquiring minds want to know!

Cloaked Bots Snared, Charted and Graphed

Since improving the profiling algorithm of my bot blocker so that cloaked bots could be trapped more accurately the statistical data has been piling up and now that data has been compiled and graphed for your viewing pleasure.

Hopefully this will explain to those that think I'm trying to stop beneficial spiders that those spiders aren't what I've been going on about whatsoever.

For those new to the blog we'll quickly define a couple of terms.

• Cloaked Bot - a web crawler that uses the user agent string of a browser like Internet Explorer, Firefox or Opera
• Blocked Agents - those crawlers that plainly identify themselves, such as Googlebot, but are unwanted and typically blocked via robots.txt, .htaccess, or other methods.
  

The chart below shows the amount of requested pages by cloaked bots pretending to be browsers (in purple) compared to blocked agents that have a plainly identifiable user agent string. One thing immediately obvious is that page requests by the cloaked bots far exceed the number of page requests made by all the other blocked agents combined.

The trend analysis of the cloaked bots reveal that, other than a couple of recent spikes, their page requests are on a slow but steady decline over the period charted. This could be a direct result of blocking them and sending them pages of error messages since they aren't getting what they need. Note that in the same time period the normal blocked agents keep attempting to request pages at a steady pace, regardless of the fact they're getting nothing but error pages, with no obvious trend evident at this time.

While some of this may look underwhelming at first, realize that many crawls would've been for hundreds or thousands of pages that are now being stopped before they ever start. Much of the attempted crawling appears to be repeat offenders that already have a complete listing of all the pages of the web site from prior visits. Some cloaked bots do manage to get the site map before being stopped and try to crawl all the links they know about, albeit fruitlessly.

At a minimum this information proved my theory, for my website anyway, that it's not the bots you know and can see that are the real problem, it's the bots you can't see.

With all this in mind, do you think you really know how many actual visitors and pages views your site gets?

Spy vs. Blog Cache

OK, this is getting ridiculous, spy on someone that's stupid and falls for your tricks.

I reported on blog spying the other day and the direct spying stopped showing up in my activity log but now I'm seeing information spies watching my site via Google cache.

Let me be the first to tell you sneaky snoopers that you've met your match this time, the gloves are off so keep fucking with me so I can add every trick in your book to my bot blocker product. Just keep showing off trying to spy on me as every time you play a new trump card you can never use that trump card against me, or anyone that will install my bot blocker in the future.

Talk about playing with fire, sheesh.

Did you think that you could stop hitting my blog and hit my cache and not be noticed?

Look at this listing from Google cache that was scrubbed of identifiable data to mask these people:

http://72.14.203.104/search?q=cache:incredibill.blogspot.com/ %22[THE IP OF THEIR WEB SITE]%22

Very clever, scanning Google for the domain name of my blog, and narrowing the search for LINKS to your site from my blog, by your IP address instead of your domain name hoping it would go unnoticed.

Maybe other people are stupid and fall for this shit but I noticed that IP address, did an NSLOOKUP, and VOILA!, it was the same block of IPs and spybot company name that I've blocked on my main website and discussed in this blog before and now this continuing surveillance is bordering on, bordering hell, it's harassment.

Just go away, we're done already.

UPDATE: After lunch I set the blog to NOARCHIVE so as soon as Google and all the rest update and kill the cache we'll see what they try next.

Beware Blog Spies

This isn't the first time I've noticed this but it's definitely the quickest that my blog has attracted spying operations that monitor information about specific topics.

The first time it was overtly obvious what was happening when I posted about corporate crawlers that spy on websites and the very next day my blog was under daily surveillance from these corporate clowns.

The second most obvious time started yesterday when I posted the blurb "Dubious from Dubai" which immediately started getting hits from what appeared to be automated tools performing automated queries for Dubai on search.blogger.com and one of them was from 217.164.235.140 which according to whois is Dubai: "P.O. Box 1150, Dubai, UAE". The other hits were from several places in New York which make me wonder who's camping on any information about these Arab countries and for what purpose?

I'm not a cloak and dagger kind of guy but sometimes you just see things that raise your eyebrows.

Yugoscrapia My Ass

The "Get a Fucking Clue" award for today goes to whoever the hell is sitting behind 212.102.136.25 hiding as Mozilla/5.0 (compatible; MSIE 5.0) from Yugoslavia that has spent over 8 hours now downloading 1,000+ error messages from my site and is currently still going strong.

Well pal, whoever you are, my automation is better than your automation because mine has a built-in BULLSHIT DETECTOR that stopped yours from doing it's job.

Dubious from Dubai

Just got hit for a couple hundred pages each from this triumvirate of IPs from Dubai:

195.229.241.181
195.229.241.180
195.229.241.187

Reverse lookup said NXDOMAIN and Whois didn't get much either so I'm just gonna block 195.229.241.* and be done with it.

Data Mining Kills the User Agent String

FACT: Your website is raw material for the Internet data mining industrial machine.

Much like the California Gold Rush there's a lot of free money on the table and everyone is scrambling to grab their share on the Internet. This time instead of sifting thru rocks looking for gold nuggets they're using bots instead of shovels and crawling websites instead of standing in a creek. The purpose of all this web crawling is to sift out information from a variety of websites looking for gold nuggets of content that will help get free money in the form of internet advertising. Everyone wants a share of the free money and your website could contain just the right couple of nuggets of gold that the Internet claim jumpers need to succeed.

Simplistic filtering of the User Agent string to block these claim jumping bots has definitely become obsolete because most undesirable bots already don't identify themselves as anything unique and try to hide their presence as the prize is too big to let a webmaster stop them. Don't think this behavior is limited to simple content thieves trying to capitalize on your hard work with AdSense as there are several corporations that I've caught in my snare and probably a bunch more lurking behind IPs that don't expose them with a simple reverse DNS lookup.

What kind of data mining happens on your site?

• Search Engines
• Data Aggregators
• Web Copiers/Offline Readers
• Copyright Compliance
• Branding Compliance
• Corporate Security Monitoring
• Media Monitoring (mp3, mpeg, etc.)
• Link Checkers
• Privacy Checkers
• Content Scrapers (pure theft)
• so on and so forth

Other than search engines which provide a valuable service bringing you traffic, many of these so-called services are just one-way bandwidth hogs that not only earn money off your back but you get to pay for the privelege!

Not all of the aforementioned services try to hide who they are and the more legit ones still check robots.txt and present a user agent string so you can opt-out (don't get me started) of their service. However, as the free money flows on the internet so does the desire not to get caught and stopped such as the spy services and scrapers.

More and more crawlers daily are pretending to be users than admit what they truly are to permit the webmaster to stop them, and that trend seems to be growing rapidly as the stakes are higher.

Use robots.txt and .htaccess while you can but you're only stopping the good guys as everything else has gone underground and there doesn't appear to be any reversal of that trend anytime soon.

ShitMoreBlueberries

Here we go again with another anonymous proxy site called EatMoreBlueberries.

On the site it goes blah blah blah about circumventing censorship then has the balls to strip my ads and shit, which censors MY ability to earn from my pages, might even be considered a tortious intereference with business, then slaps their ads on top of content that THEY DON'T FUCKING OWN!.

They appear to have a block of IP's so it looks like you can just block 69.49.99.* for starters and they're gone.

Well guess what fuckwads?

I've blocked your ass so you've gotten the ultimate censorship which is a big FUCK YOU from ME to YOU!

SeekOn Elsewhere

It appears that the people at SeekOn think that if "someone" submits your site to their service, just anyone according to them, they have the right to crawl your site. Better yet, they don't even bother to look for ROBOTS.TXT, oh fuck no, they expect you to use META TAGS on all your pages to control robots instead of that one handy little robots.txt file.

Just block "SeekOn Spider" in your .htaccess file and they can kiss your ass at that point.

JetBrains Damaged

The product claims to be an RSS Reader and Newsgroup Aggregator but I caught something claiming to be JetBrains Omea Reader 2.1.4 looking at pages, well, attempting to look at pages, that were neither RSS or Newsgroup.

Never seen this before and I'm not sure if it's a fake or this thing can be used for other purposes but you might want to keep an eye on this thing just in case.

Scraping Gold Medal

Most scrapers only try to get a few hundred pages at a shot and some of the more aggressive ones attempt to get a couple of thousand pages but this scraper gets the gold by attempting to extract 12,935 pages in one shot.

What he got was 12, 935 pages of error messages.

Way to go dumbass!

Brazilian Bot Busted, Oh GoshME!

Well some AdSense monetized bullshit search engine or something called GoshME masking as a BROWSER of all things has been very slowly extracting 56 pages for a total of 58,054 seconds or 16 hours from the address of 64.34.172.77.

BTW, what kind of happy horseshit is a Brazilian website that's all in English?

What's best is this fucking thing is POSTing to my web site, not using GETs like a normal crawler, so God only knows what these assholes are stealing and he's not telling.

FWIW my website shows up as #1 on their site for my category but I don't give a flying fuck.

MASKING AS A BROWSER?

FUCK YOU!!!

Le Tub and Le Return From Florida

Back from Florida so no more vacation, birthdays or weddings in my near future, done with all that shit and not a moment too soon.

While in Hollywood, that's Hollywood FLORIDA you morons, I decided to try what GQ magazine claimed was the #1 choice of "The 20 Hamburgers You Must Eat Before You Die" and gave Le Tub a try.

For starters, it should be called Le Dump as it makes every dive I've ever been in look like a sparkling palace by comparison. Everything including the tables and benches, if you can charitably call them that, look like they were all made of the same rotting aged dock lumber. The place is cluttered with bath tubs, sinks and toilets used as planters and stuff. Some would call this shit "decor" but overall it looks like you're dining in a fucking junkyard. How this place isn't condemned is fucking amazing.

Don't worry, we read several reviews of this place before going over there and knew what we were getting ourselves into but reality cramps are still a bitch.

We walk in and head toward what looks like a half empty bar and the cook shouts from the cloud of smoke in his bird cage sized kitchen "Those seats are taken and it will be an hour to hour and a half for a burger!" so we head outside and plop down on a table where my bench looks like it's about to collapse but surprisingly didn't wobble at all.

That's correct, you're reading this properly, 1 to 1 1/2 hours for a burger, which we also expected going in the door as it's the Disneyland of hamburgers and it will take that long to get your turn at the hamburger rollercoaster.

Why does it take so long?

All of the burgers are 8 ounces (1/2 pound) which makes them huge fucking piles of beef to start with and takes a long time to cook even to medium. The grill is tiny too, about a 3x3 grill so it doesn't hold very many of these gut bombs in the first place.

The menus are printed on crappy paper and of course mine had mustard stains all over it and might've been slightly soggy but there was only one item on the menu of any interest and that was the hamburger. It's only served one way, and that's the 8 ounce charbroiled way for $10 with the optional cheese costing $0.50 more. Comes with lettuce and tomato but you have to pay an extra $3.50 for small fries or $5 for a basket to share and the fries are well done, nice and crispy the way we like 'em.

Whoever wrote the review claiming Le Tub had a "good if not stellar selection of beer" wouldn't know a good beer if the bottle was smashed upside their head. Good beer is not named Bud, Bud Lite, Coors, Coors Lite, Heineken, Amstel Lite and Becks for fuck's sake!

Our skinny-assed waitress, and I mean skinny as my dick is bigger than her thigh, was pretty attentive and after taking our order it only took 4 Beck's before the burger arrived. While we were waiting we noticed so was everyone else with every table full of hungry zombies just staring at each other wondering when or if their food would ever arrive. Once the orders started flying out of the kitchen it was pandemonia at all tables with squeeze bottles belching out ketchup and mustard in a chorus culminating with the starved gobbling up a generous chunk of scorched cow on a bun.

The burgers had a nice charbroiled flavor and since I've now eaten the #1 hamburger that I must eat before I die, according to GQ, I can now peacefully pass knowing that Le Tub probably contributed to the clogged artery that will most likely kill me.

If you're ever in the neighborhood of Hollywood, FL you simply must try Le Tub as you can't even imagine this place until you see it first hand as there are simply no words that can truly describe this place.

Bon Appetit.