Saturday, October 14, 2006

No Referrer and Visitors vs Spambots

I've been struggling recently on better ways to handle requests to form pages, like a comments page, that are more secure yet more visitor friendly than just slapping up a "403 Forbidden" which runs off humans as well as bots.

After contemplating the issue and taking bookmarks and disabled referrers into account, I decided to simply redirect these potential bad hits to my home page instead of the old 403 error. This way a valid visitor that bookmarked the page could just navigate back, referrer intact and post as usual. So far I've seen a few humans that were redirected off the page for whatever reason navigate back to where they wanted to go, so it doesn't appear to be stopping determined people that aren't just there for malicious purposes.

Additionally, I only do this redirect after verifying the request isn't coming from the search engines as I obviously don't want to confuse the SEs by redirecting them to the home page.

The fun part is it seems to be confusing the shit out of the spambots, they're bouncing all over the place, quite hysterical to see them freak out.

For even more fun, verify that the referrer to your form page isn't a direct hit from a search engine to that page as many of the hand spammers from places like the Ukraine and India seem to like to use Google to find pages to post their clients' listings. For those reasons, now I'm also redirecting any direct hits to the posting page that comes from Google, Yahoo, MSN and ASK. Redirecting the hand spammers (aka SEOs) back to the home page seems to stop the hand spams as well since I've just made the job a little harder they just seem to move along instead of spending more time to find the page

One last trick, if you have the means to track these things like I do, is reject or auto-moderate anything with just a single page view to your site, which is just that form page. Some figured out I was looking for a specific referrer and plugged it in so the post looks 100% legit. Well bummer dudes, you need to view more than one page to make a submission so cleaning up the data to add a valid referrer was a nice try but you still have a bad bot profile by having no previous page views.

Gotta love all the fun and games with both sides escalating but so far I'm still spam free and winning this war.


John Andrews said...

Bill, had you heeded my earlier advice and hired a real PHP programmer, you would have known that *all* posts should check and verify a token before accepting data as valid. No need to track behavior/count page views/interrupt the stateless web. No tickie no shirty. Simple.

Is this the landmark post that finally acknowledges your assignment of SEO's to the spammer department?

IncrediBILL said...

Scuse me?

I know how to track data tokens but there is more going on than I posted about, I have data tokens as well.

Besides, if you know such good PHP programmers why aren't they offering a hand to these websites being spammed into oblivion to fix their shit?

FYI, I didn't say all SEO's, I just have problems with the ones from India.

Anonymous said...


When are your "secret spam stopping script" going to be public?

The rest of us needs something like that.