Maybe this is how Amazon responded to my tongue-in-cheek request to set the user agent on their crawler.
I have no clue why SpecificMedia would be attempting to crawl my site, or why they are coming from an Amazon IP address. Maybe it's possible when you hire the AWS for a specific task they just plug in the customer name as the UA. Perhaps Amazon just auctioned off the user agent to the highest bidder for some viral marketing thing, who knows.
Anyway, here's the IP's and the user agent seen crawling:
216.182.231.65Just what we need, more crap crawling the web.
[domU-12-31-33-00-03-EB.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.225.220
[domU-12-31-33-00-03-92.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.231.59
[domU-12-31-33-00-03-ED.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.228.145
[domU-12-31-33-00-02-53.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.230.236
[domU-12-31-33-00-03-26.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.225.180
[domU-12-31-33-00-03-02.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.231.86
[domU-12-31-33-00-03-D8.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.231.93
[domU-12-31-33-00-03-CF.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.228.139
[domU-12-31-33-00-02-55.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.230.163
[domU-12-31-33-00-03-6D.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
216.182.231.20
[domU-12-31-33-00-04-16.usma1.compute.amazonaws.com.]
"SMBot/1.1 (www.specificmedia.com)"
Joy.
3 comments:
I wish I'd seen this earlier Bill. It hit one of my sites last week using an Amazon.com IP Address from South Africa. Let me know if you want the details.
I just noticed these guys too about a month ago. I was banning them before they had an user-agent as well. Your observations parallel my own. Their bot SLAMS my sites, but now it's blocked, and my site will add any offending IP addresses to a blacklist automatically.
I just posted a new blog about this bot...SMBot hit me over 300 times in one day on 2 different sites, so I'm pissed. I have a great way to ban SMBot...Read my blog and spread the word. :)
Post a Comment