SpamAssassin 3.0 Released

davemabe writes "At long last, SpamAssassin 3.0.0 has been released. I've been using the release candidates for a month or so, and the results have been far improved over previous versions. Its use of SURBL along with Bayes auto learning make it seem like this solution is the one to beat. It looks like they've introduced a new logo as well. Snazzy!"

SURBL

[alatesystems]

For those not in the know, SURBL is really cool. It actually lets you scan the message(well, SA does that) and then look for urls that it links to. It compares this to a realtime BL of other people getting spam like you and if it is a known spam TARGET url then it blocks the message based on that.

It makes it really hard for them unless they just register countless domains.

Excellent technology, and I will be upgrading to the newest stable.

Chris

anto-spam

[Outsider_99]

Ive been playing with DSPAM [nuclearelephant.com] which seems very good. They claim a 99.991% accuracy. Apparently this is 10 times more accurate then a human. But Ive heard that most anti-spam solutions are very good.

3.0 New Features

[CleverFox]

Major feature list:

- SpamAssassin is now part of the Apache Software Foundation and has an
improved software license, the 2.0 version of the Apache License.

- SpamAssassin now includes support for SPF (the Sender Policy
Framework, http://spf.pobox.com/).

- Web site links contained in the message are checked against SURBL and
SBL. SURBL and SBL track sites that advertise with spam, known spam
sources, and spam services.

- The new 3.0 architecture allows third-parties to easily add plugin
modules.

- There is now SQL database support for both the Bayes and
auto-whitelist modules, allowing more large sites to easily deploy
SpamAssassin.

- A more accurate simulation of email client handling of MIME and HTML
improves our accuracy. In addition, there is better detection and
handling of spammer techniques that try to trick anti-spam software.

Important installation notes:

- The SpamAssassin 2.6x release series was the last set of releases to
officially support perl versions earlier than perl 5.6.1. If you are
using an earlier version of perl, you will need to upgrade before you
can use the 3.0.0 version of SpamAssassin.

- SpamAssassin 3.0.0 has a significantly different API (Application
Program Interface) from the 2.x series of code. This means that if
you use SpamAssassin through a third-party utility (milter, etc,) you
need to make sure you have an updated version which supports 3.0.0.

- The --auto-whitelist and -a options for "spamd" and "spamassassin" to
turn on the auto-whitelist have been removed and replaced by the
"use_auto_whitelist" configuration option which is also now turned on
by default.

- The "rewrite_subject" and "subject_tag" configuration options were
deprecated and are now removed. Instead, using "rewrite_header Subject
[your desired setting]". e.g.

rewrite_subject 1
subject_tag ****SPAM(_SCORE_)****

becomes

rewrite_header Subject ****SPAM(_SCORE_)****

- The Bayesian storage modules have been completely re-written and now
include Berkeley DB (DBM) storage as well as SQL based storage (see
sql/README.bayes for more information). In addition, a new format has
been introduced for the bayes database that stores tokens in fixed
length hashes. All DBM databases should be automatically converted to
this new format the first time they are opened for write. You can
manually perform the upgrade by running "sa-learn --sync" from the
command line.

The "sa-learn --rebuild" command has been deprecated; please use
"sa-learn --sync" instead. The --rebuild option will remain
temporarily for backwards compatibility.

- "spamd" now has a default max-children setting of 5; no more than 5
child scanner processes will be run in parallel. Previously, there
was no default limit unless you specified the "-m" switch when
starting spamd.

- If you are using a UNIX machine with all database files on local
disks, and no sharing of those databases across NFS filesystems, you
can use a more efficient, but non-NFS-safe, locking mechanism. Do
this by adding the line "lock_method flock" to the /etc/mail/spamassassin/local.cf file. This is strongly recommended if
you're not using NFS, as it is much faster than the NFS-safe locker.

- Please note that the use of the following command line parameters for
spamassassin and spamd have been deprecated and are now removed. If
you currently use these flags, please remove them:

in the 2.6x series: --add-from, --pipe, -F, -P, --stop-at-threshold, -S
in the 3.0.x series: --auto-whitelist, -a

- The following flags are de

Re:Release notes?

[AnotherScratchMonkey]

You can browse the version 3.0.0 Subversion repository [apache.org]. I'd suggest looking at the files UPGRADE and Changes.

Re:Release notes?

[Anonymous Coward]

The wiki included a link to http://issues.apache.org/eyebrowse/ReadMsg?listNam e=[e-mail]&msgNo=15 757 [apache.org] ; it would have been nice to have a list on the front page.

For those who may have forgotten

[numbski]

I'm building the latest on all of my clients' mail exchangers and our primary boxen. ;)

Here's the command to install/upgrade 3.0 via CPAN:

# perl -MCPAN -e shell;
cpan > install Mail::SpamAssassin

(many lines, type in the administrator's e-mail address, say no to network tests)

exit

#

Very difficult stuff. :) Keep up the good work.

Oh! Some link whoring as well:

SpamAssassin Milter for Sendmail [gnu.org] - Filters everyone without procmail

SpamAssassin Milter Quarantine [sourceforge.net] - Quarantines spam messages and sends summaries in digest for 1 or more times daily rather than simply delivering to the end user.

Re:Does it use IP's or URI's ?

[Ewan]

our Spamassassin 3 release candidate seems to filter on both IP addresses and URIs, seems very effective - our spamassassin now marks over 50% of incoming email as spam.

Ewan

Re:Does it use IP's or URI's ?

[NKJensen]

Sorry, I've found the question and some pros and cons here:

http://www.surbl.org/faq.html#numbered

Re:Does it use IP's or URI's ?

[krunk7]

Are you using Rules Du Jour [exit0.us]

Re:Does it use IP's or URI's ?

[Anonymous Coward]

This is just a small nitpick, Apple's Mail.app uses latent semantic analysis, not baysean filtering.

Re:For those who may have forgotten

[davemabe]

There's also sa-exim [merlins.org] which is a local scan plugin for Exim [exim.org] that does site wide spam filtering at SMTP time. It is quite nice.

Dave [runningland.com]

Re:fillters vs. stallers

[Anonymous Coward]

With spam stallers like sa-exim and tarproxy
your are stalling the spammers smtp connection
and the effect is that the spammer can't send
as much spam or that they drop you email from there email database.

Stalling has been proven NOT TO WORK. If the spammers were using a single threaded application, it would slow them down. But spammers are smarter than that. They have large distributed networks of spam zombies with multithreaded applications that aren't slowed at all.

And how do you determine if you should stall a connection? Blocklists only go so far - you have to scan anyway to determine if you should stall.

Spammers NEVER DROP YOUR ADDRESS FROM THEIR DATABASE. My company still gets spam addressed to employees who left years ago.

McAfee SpamKiller based on SpamAssassin no good

[aardwolf204]

A lot of closed source software has open source counterparts, (i.e. MS and Open Office) but its always interesting to see closed source commercial software based on an open source project.

McAfee has a product for Exchange servers that is based on Spam Assassin called Spam Killer. I found out about it from the Spam Assassin site when I was looking for a windows version. Spam Killer isnt free yet its not as expensive as some of the other solutions out there.

The major problem I've been having with it is it creating zero byte emails which cannot be downloaded via pop3. When a user gets 30 messages, and message 10 is a zero byte email the client will constantly download the first 10 over and over, creating duplicates, until the user logs into outlook web access (webmail) and deletes the zero byte message. This doesnt happen to the MAPI users but we have quite a few POP3 users.

The support people are useless, I'm about to try out Microsoft Intelligent Message Filter for exchange, and hopefully with some good RBLs it should be ok.

Re:Does it use IP's or URI's ?

[Kainaw]

I would rather extract the domain, look up the IP, and check the IP.

I wanted that a long time ago. At the time, I couldn't find a program written by anyone else, so I wrote my own. It works well for me and anyone who wants the script is free to use it. It is at my homepage [kainaw.com].

I no longer get spam

[Anonymous Coward]

The solution is extremely simple if you use OpenBSD.

rdr on $ext_if from any os "Windows" to any port smtp -> 127.0.0.1 port 8025

99.9% of all spam comes from compromised Windows boxen, and nobody with a clue would run a mail server on windows.

Turbo Smorgreff [www.des.no]

Re:Does it use IP's or URI's ?

[marmoset]

Best human-readable discussion of the techniques I've read is here [pacificavc.com].

Re:Improved Performance?

[Tim Macinta]

I must say, Python might be a nice language and all, but as it's making inroads everywhere it's also wrecking havoc on ones ability to convert older hardware into a competent server.

Spamassassin is actually written in Perl, not Python. I'm not saying your point about certain languages making it difficult to maintain older machines isn't valid, I'm just clarifying what Spamassassin uses.

Re:fillters vs. stallers

[Anonymous Coward]

Stalling has been proven NOT TO WORK
Do you have a link to the prove ??
If only one person run a staller it would
not have a big effect but if 0.01% would run
a staller it would have a big effect.

My hope is that the debian/gento/freebsd
would have a spam stalling MTA as the default MTA
at some time soon.

And how do you determine if you should stall a connection?
Please, see sa-exim or tarproxy.
Spammers NEVER DROP YOUR ADDRESS FROM THEIR DATABASE.
Why should the spammer drop your address if
your are not stalling them ???

Re:Performance

[SCHecklerX]

It scales just fine. We are using it in comination with Mimedefang's multiplexor here. Mimedefang is great, since you can totally reject stuff that fails basic tests without having to even bother invoking your spam filters.

We have 3 linux dual-processor mail servers, and have basically maxed them out with memory so that we can use ramdisks for Mimedefang processing. CPU Utilization is currently <10%.

Some stats from yesterday:

Spam stats:
Total Mail In and out: 39320
Processed by MimeDefang: 24210
Spam: 1519
Non-spam: 22691

Rejected stuff:
Completely discarded: 6453
Reject 554 total: 31343
Reject 550 total: 18151
Reject 501 total: 4376
Rejected pre-greeting total: 1273
Suspicious Header total: 1
Partial MIME type total: 0
Non-multipart total:
Forbidden File attachment total: 60

Re:Antispam Gateway Distribution?

[mortonda]

This is exactly the direction we are planning to go with Maia Mailguard [renaissoft.com], plus features such as tarpitting, network reporting, and p2p associations. It's going to take a while to get there, though.

TMDA

[Gudlyf]

I've used SpamAssassin for quite some time now, but I was still getting spam through, mostly because I'm a paranoid freak and figured I'd be missing out on mails that it mistakenly tagged as spam.

What I use now (alongside SpamAssassin) is TMDA [tmda.net]. This is basically an "approval queue" for messages. If someone not in your approved list send you mail, they get a reply telling them they need to send mail to a specificly generated address in order to allow the mail to pass through to me. Eventually mails that don't get approved time-out and get added to a blacklist for the future. I also quickly review the queued items every morning in case someone didn't see the approval mail (it has a tool that allows you to easily peruse the list with just subject and sender info). So far I've gotten NO spam through this method -- NONE. I used to get hundreds a day, and now I have a spam-free INBOX because of TMDA.

While I highly recommend using TMDA, it may not be for people running businesses or waiting for mail from clients. The auto-reply message can perhaps strike some as inconvenient, even though they only have to do it once (once they've sent mail to the approval address, they're added to the whitelist for all future mails). So far spammers haven't found a way around TMDA it seems...so far.

Re:Does it use IP's or URI's ?

[pqdave]

SpamAssasin is for email, and won't affect anyone trying to browse to your site. At worst, a properly-configured SpamAssasin would see a mention of your URL in an email, resolve it to the same IP as a spammer, and give it a few more points towards the spam threshhold. SpamAssasin (at least as used by my mail admin) scores messages based on various factors rather than giving pass/fail tests, so a suspicious URL in an otherwise non-spammy message wouldn't necessarily send it over the spam threshold.

Re:Does it use IP's or URI's ?

[pjrc]

I would rather extract the domain, look up the IP, and check the IP.

That won't help against "bulletproof hosting", commonly used by spammers, where a nameserver in a country like Russia or Poland resolves the name to one of thousands of zombie machines hosting the site.

The SURBL approach does.

Yes, I know that servers many host many domains: ... This will only increase pressure on the spamheaven server admins to get rid of the people who use spam to spamvertize their sites.

Spammers don't use $10/month shared virtual hosting for their websites.

Re:SURBL

[Darby]

I'm not aware of any (modern) mail clients that execute javascript,

Well, Kmail and thunderbird to name 2 off the top of my head.
Granted it's off by default, but the functionality is there.

Want to learn more about SA 3.0?

[felicity]

At ApacheCon 2004 [apachecon.com], there will be 3 presentations about SpamAssassin: a 3-hour tutorial covering SpamAssassin as a whole, a 1-hour talk about the new features in 3.0, and a 1-hour talk about using the new SQL features.

Sorry for the plug, but I thought may be interested. :)

Re:Antispam Gateway Distribution?

[Anonymous Coward]

sorry - I missed this post and alrady commented about the product that we use ... forget Barracuda. Sure, it's cheap, but iit's the same old SA crap. We use Mailfoundry - about the same price, but WAY better. They have the same 30 day free trial - try it and you'll HATE Barracuda. Period.

disclaimer - we have two of their boxes. What can I say, it's made my admin duties a lot easier since I don't have to deal with that silly spam crap anymore.

Re:Installing on Windows....you're kidding, right?

[Clover_Kicker]

SpamAssassin on Win32 is an afterthought at best.

The Win32 stuff is provided as a courtesy. I don't think they really expect anyone to use it, since it is so much easier to install on *nix.

Your ISP sucks then. :P

[EvilStein]

I'm using it on a dual 1.6ghz Xeon box with Gentoo here in the office - the box processes over 70,000 emails per day (spamassassin, amavisd-new and clamav/f-prot) and the load average barely goes above 0.02.

Your ISP just didn't want to take any time to actually learn about it. :)

Re:Installing on Windows....you're kidding, right?

[Anonymous Coward]

Normally I'd agree, but CPAN for Win32 users hasn't been a good option generally. It works ok with cygwin, but

- the make tests have failed too often (and thus confuse users) for anything but a manual install. There are long turgid reasons for this but the upshot is that few builds of SA have completely finished all tests via CPAN successfully. (This has greatly improved recently).
- Net::DNS until recently was very problematic and it was generally not safe to run any but an old copy - this tended to muck up CPAN newbies.
- Also, SA via CPAN assumes an installed C compiler, which is rarely the case for Win32 folks.

As you can see, most of this has improved rapidly, but it's still true that it's meant to be an MTA integrator - and outlook integration etc, require more work

People can (and have) created self-contained spamassassin.exe installs with the Perl Dev Kit. So Far no one has regularly maintained such a setup for the community and the PDK is not free.

See http://www.openhandhome.com/howtosa.html for detailed instructuctions on a standard Win32 setup of SA

Re:Installing on Windows....you're kidding, right?

[Christopher_G_Lewis]

First of all, there is no install. This is a pure source release. Quite common, and after a little bit of testing, (you wouldn't blindly put this on a production box, would you :-) it's quite easy.

Your points...
1. Extract it where ever you want.
2. So? PPM and CPAN are simple.
3. or you could use the docs on the web site you were looking at.

4. Step 10 does *not* require a D drive, the -D is for Debug mode. It spits out everything that SpamAssassin is doing, i.e. what config files, what db's what tests are being run. Actually quite usefull.

5.

Re:Installing on Windows....you're kidding, right?

[sidney]

I did a lot of the work of getting SpamAssassin to build and run on Windows. My goal was to have SpamAssassin build and install on Windows using the unmodified sources before version 3.0 was released. It does that now.

SpamAssassin was written in Perl on Unix and Gnu/Linux, for use in high volume server environments. The installation for an ISP or for anyone running a *nix mail server is a piece of cake. Their users get their mail filtered without having to install anything on their own PCs.

The fact that it works on Windows at all is a bonus. It is an open source project. Would anyone like to volunteer to help with the next steps of getting the server daemon, spamd, working properly in Windows as a service; writing or adapting an existing mail proxy that would integrate SpamAssassin with mail clients such as Thunderbird, Mozilla Mail, Eudora, Outlook Express; packaging it up in a standard Windows install package?

Addressing the 5 points in the parent post:

1. Nothing has to go in the root directory. The instructions show an example of Perl having been installed in C:\perl and configuration going in directories underneath a C:\etc\mail directory.

2. Yes you have to install Perl. And a recent enough version that doesn't have certain bugs. And the required modules. SpamAssassin was written in Perl, which makes it useful on systems that have Perl, such as most Unix and GNU/Linux systems. If you install Perl and the modules on your Windows system then you have a system that meets the minimum requirements. If you have a Palm Pilot or or an Xbox or Windows without Perl then your system does not meet the minimum requirements and you are not going to even try to run SpamAssassin on it. In that case install SpamBayes, or get an ISP who uses SpamAssassin for your mail, or any of many other alternatives.

3. Making the doc files is easier in *nix. I'll file a request for enhancement suggesting that generating the HTML be made part of the Makefile and that it be made to work under Windows. The doc files are generated from the sources as part of the build, so they are not included in a source distribution, which is what we are talking about here. If someone built a binary distribution they would include the doc files.

4. That -D command line option stands for Debug, not D drive

5. The whole install proces consists of 13 steps, some of which are things like "download SpamAssassin", some of which are "if you are installing the old version 2.6x do this extra step", and some of which have to do with getting the required Perl and Perl modules. The actual installation pretty much happens in three lines of step 7. It really is quite easy for a build and installation starting from source files. A binary installation package would be a lot easier. Does anyone know how to package perl plus modules plus a built SpamAssassin into a Windows install package? If you do, feel free to volunteer.

The focus on usability and user friendliness is where it should be in this particular project, on the sysadmin who installs SpamAssassin on a server and on their end users who don't have to install anything at all.

If you have the ideas and the expertise to also make SpamAssassin more useful and friendly to the end user owner of a PC running Windows, please volunteer to help.