Wednesday, November 29, 2006

Dear Amazon AWS Group

To whom it may concern,

Your bot crawled my site today as shown below. Please notify your engineers, and I use the term loosely, that "Java/1.5.0_09" is not a valid bot name. Being that Amazon sells books on how to program Java, I'm sure you can find at least one book in your warehouse that will explain how to set the User Agent string when making web requests.

Additionally, would honoring ROBOTS.TXT be too much to request or do you feel justified not checking the robots file since your programmers can't figure out how to tell us what your bot name is in the first place?

216.182.236.241 [domU-12-31-34-00-00-B9.usma2.compute.amazonaws.com.] "Java/1.5.0_09"

216.182.236.142 [domU-12-31-34-00-01-1E.usma2.compute.amazonaws.com.] "Java/1.5.0_09"

216.182.236.177 [domU-12-31-34-00-00-F9.usma2.compute.amazonaws.com] "Java/1.5.0_09"

216.182.236.110 [domU-12-31-34-00-00-2A.usma2.compute.amazonaws.com] "Java/1.5.0_09"

216.182.236.167 [domU-12-31-34-00-01-07.usma2.compute.amazonaws.com] "Java/1.5.0_09"

216.182.233.105 [domU-12-31-34-00-01-D3.usma2.compute.amazonaws.com] "Java/1.5.0_09"

216.182.230.187 [domU-12-31-33-00-03-55.usma1.compute.amazonaws.com.] "Java/1.5.0_09"

216.182.237.9 [domU-12-31-34-00-01-B5.usma2.compute.amazonaws.com.] "Java/1.5.0_09"
It makes me weep for the future when a big web conglomerate, one that has a name that is synonymous with buying things online, one that should know better, starts to slide down that slippery slope of being a bad netizen.

Signed,
Get A. Clue

5 comments:

Anonymous said...

Hahhahahahhahahhahhaah !!!!

very funny! Cool cool cool!

Java/1.5.0_09-amazon-bot also visited us yesterday. He straight ahead decided to visit our quarantine bot trap. I will try to get and interview with him but I doubt I will get some valuable information from somebody hiding behind a name like "Java/1.5.0_09".

Digitally
TheHidingWebBot/1.5.0_09

Anonymous said...

Who needs a book?

http://java.sun.com/j2se/1.5.0/docs/api/

Right on the Web site is information that anyone with, additionally, a) some knowledge of HTTP and b) a clue can use to figure out that

connection.setRequestProperty("User-Agent", "FooBot/1.0+http://www.foo.com/crawler.html")

and Bob's your uncle. :)

Anonymous said...

Even more specific link

http://java.sun.com/j2se/1.5.0/docs/api/java/net/URLConnection.html#setRequestProperty(java.lang.String,%20java.lang.String)

CarlBrannen said...

I haven't seen Amazon, but 20 different DNSs have done this to me, too. It's not a problem, I'm just curious what's going on. And some are "Java/1.4.1_04".

Scott Allen said...

LOL...totally agree. :)