Stopping SpamBots With Apache Part II

primetyme writes: "To address some of the concerns brought up in the first article about stopping email harvesting spambots with Apache, I've written a follow-up article that details even more methods to keep email-sucking bots off your Apache based site.
Stopping Spambots II - The Admin Strikes Back continues the epic saga that pits Spambot vs. Administrator."

Restarting the server?

[epsalon]

The article suggests restarting Apache for every spam address detected. That could make DOSing your web server real easy. Spoof a bunch of IPs and request the honeypot dir. Watch as the webserver restarts over and over.
Also, this approach would easily block legitimate dialup users, and more problemaically - proxies. If the spambot is behind a proxy, you would block the entire user base of that proxy.
Maybe an X-Forwarded-For based approach? However, that is easily bypassed.

Re:Restarting the server?

[primetyme]

Valid point epsalon.. but to clarify, Apache only gets restarted for every *new* IP address detected.. As for the spoofing, it would take a lot of IP's to DOS the server, and anyone willing to go through that much trouble just to take down a webserver probably has better ways to do it. Point taken though :)

Re:Restarting the server?

[epsalon]

A simple improvement will be to send SIGHUP to the webserver to make it reload the config without restarting. This still can be used for DoS, but less efficiently.
A better way to do it is by writing (using?) an Apache module that does the logging in memory with no costy reloads or restarts.
However, this still does not prevent the proxy and dialup problems illustrated above. Also, you won't catch spambots that don't use robots.txt to find addresses.
Another improvemnt will be to deny addresses the moment they ask for robots.txt while identified as "Mozilla" user-agent, and to detect clients that do a websuck without requesting robots.txt first and deny them as well. You can detect a websuck by posting a "hidden" link in a place normal users won't see and stop any IP that requests it.

Re:Restarting the server?

[primetyme]

An actual Apache mod is slated for part 3 of the series.. stay tuned. Thanks again for the feedback!

Re:Restarting the server?

[beebware]

Why not just us a .htaccess file in your top-level htdocs directory?
Order by deny,allow
deny from spammers.ip.address.here, another.spammers.ip.address
allow from all
will _probably_ do it (ie this is an untested example!)

my favorite tactic

[augros]

here's one of the best tactics i've found: http://www.phpconsulting.com/php/hide-email.php

Re:my favorite tactic

[glyph42]

Right, well now that this cat is out of the bag, it's going to take the spammers three lines of code to get around this trick, that is if they haven't already.

Re:my favorite tactic

[beebware]

Best tactic I've see is just providing a web-to-email form for people to fill in. After all: if they've got their web browser loaded, do they really need to launch an email client to contact you? Keeps your address hidden, and as long as you don't use something like Matt Wrights formmail.pl script, quite secure. Get the outgoing mails tagged with the senders IP, browser details etc and it'll help track abusive messages as well...

My trick...

[Pathwalker]

I use this [ofdoom.com] little rxml widget on all of the email addresses on my web site.

If the client is detected as a robot, or the detection fails, the address is displayed as a randomly named graphic.

If the client is not detected to be a robot, then just a light entity encoding (which I change from time to time) is applied to the address, which is displayed as a mailto link.

Re:My trick...

[cluge]

The problem is that smart robot programs can make their robot appear as any client. From experience I can tell you that most of these critters running around harvesting e-mail addresses are telling the server they are WIN98 with IE 5.0.

I just have one e-mail address on my honeypot page, when you send an e-mail to that address it triggers a script that firewalls the sending IP with iptables/ipchains/ipfw (depending on the server) and logs it. Makes it easy to find open relays and spamhaus servers.

My technique...

[Anonymous Coward]

On my web page I convert email addresses to .gif *images* of email addresses. A real person will be able to see the address, but will have to type it in.

apache module

[po_boy]

I wrote an apache module in perl [mooresystems.com] to do a very similar thing. No restarting your webserver.

Cookies

[giminy]

Couldn't you just set a cookie, with a site-wide password in it? Then just require the cookie/password protect every page. Or do spam crawlers know what to do with cookies these days?