A Look At Google's Email Spam Prevention 176
CNet has a story about the security measures Google employs to protect their email systems and fight the never-ending war on spam. Their Postini team, acquired two years ago, has a variety of monitoring tools and automated response systems to find and block undesirable messages. Quoting:
"The system scores each message on numerous combinations of criteria, assigning a weight to each and then comparing the score to those in a database of several hundred thousand message types that have been flagged as good or bad from Postini honey pots and customer spam reports. ... To block fresh spam attacks not covered by existing heuristic technologies and viruses not covered by existing signature databases Postini relies on proprietary Zero-Hour technology to identify new outbreaks that show up in the traffic patterns and quarantine them for later rescanning. Customers can also create and build out their own white lists of message senders they trust and blacklist others they don't trust. It takes an average of 150 milliseconds for a message to be scanned by the antivirus engines that Postini licenses from McAfee and Authentium.
Comment removed (Score:3, Interesting)
Praise Gmail (Score:3, Interesting)
But what about spam from "me"? (Score:3, Interesting)
Re:"Postini"? (Score:5, Interesting)
Google is the only mail service that I know of who still just won't accept my emails. They make it very difficult to contact them. There is a form buried somewhere in their help system, but it says that they won't respond unless they need additional info from you, which leads me to believe that they never actually read anything submitted through that form. (I have tried a few times.) They also specifically say they don't take whitelist requests. I have SPF records, I have correct reverse DNS, I'm not on any blacklists, etc.
This means when I send emails to my friends who use Gmail, or comparies who use Postini, I get blocked without cause. Then I have to use a different server. It's kind of annoying.
(Why do I use my own email server? Because I can. This is
Re:"Postini"? (Score:3, Interesting)
That said, since we turned on greylisting, I've seen a massive reduction in spam. The number hitting my spam folder has gone from about ten a day to one every few days. I assumed spammers had worked out how to get around greylisting by now, but apparently not.
Re:"Postini"? (Score:4, Interesting)
Have you noticed? GMail gives one no way at all to sort the captured spam. Since I still endure false positives from the system and there is NO way to disable or bypass it, having means to sort all of it by From:, To:, and other criteria would make it easier to identify the false positives and rescue them from the trash bin.
Well, I'll take that back, in part: that applies to the Webmail interface, but if ones uses IMAP with a local IMAP client, then the spam folder could be subscribed and sorted within the client. God only knows how GMail's system interprets the dragging of a message from Spam to Inbox via IMAP: does that automatically whitelist that sender in the future, or do I have to still log into the Web site and identify it as Not Spam manually?
McAfee (Score:5, Interesting)
Re: I do care how it works (Score:3, Interesting)
Re:"Postini"? (Score:5, Interesting)
I had a similar [wordtothewise.com] experience [slashdot.org]; I run my own mail server, send no bulk mail whatsoever, and both Postini and GMail independently decided I was a spammer. No DNSBLs had me listed, ReturnPath was happy, etc. Meanwhile, I was blocked from sending mail to my lawyer, my financial advisor, my chiropractor, etc., all of whom turned out to be downstream from Google. Despite Google's claims that the customer is in full control of filtering, none of them were able to get at my e-mail without getting their sysadmins involved - which often required discovering that they had sysadmins at all.
Worse, Postini's spam filtering takes its own output as input. Once it's scored a message of yours as spam, future messages will be more likely to score as spam - which of course makes any subsequent messages even more likely to score as spam. Brilliant. At one point, my spam score from a triple-signed (SPF/DK/DKIM) server was 98 out of a possible 100.
Google's philosophy of "we don't do it unless we can automate it" works horribly when it comes to customer service. There's no feedback loop, no whitelisting, no channels, no nothing. It's SPEWS all over again, or perhaps the Kafka International Airport [theonion.com].
But Google has no reason to worry about false positives; the more messages they call spam, the more spam they can say they blocked. Perverse incentives.
Re:Don't care how they do it.. (Score:5, Interesting)
I've set up GMail to filter my email and by comparison I'd say one or two spams get through. So I'm very happy with GMail's level of coverage. It's not perfect but it makes things tolerable. I'm not at all happy with Yahoo's level of coverage. Yahoo allegedly also has spam filters, but I've yet to see they actually work. It's not uncommon to find my email box filled with Nigerian and other scams.
Re:"Postini"? (Score:5, Interesting)
For what it's worth, Gmail has been just the opposite for me. It's Yahoo and AOL which randomly decide to block me -- sometimes with some cause, sometimes just because it's on a residential connection.
Yet Gmail never so much as greylists me -- everything goes straight through, every time.
Re:It was me! (Score:3, Interesting)
I signed up with Postini just as it was acquired by Google. Before that I'd used SpamSoap, which worked great but was declining in effectiveness (more false negs) but not in price ($30 per month is a lot for a small business). Postini and then Google were far more reasonable at just $3 per year per address (for the less-flexible controls). I get maybe one or two delivered spam per week, usually when I also see a corresponding spike in filtered spam which indicates a new attack of some kind. I get only one or two false positives a month.
The biggest thing I have noticed lately is that the spammers have started collating domain name "from" lines. I now routinely get a lot of spam (in the quarantine) listed as coming from the other valid e-dresses in that domain. This is new as of a month or so ago.
The real problem with Google/Postini is that, as others note in this discussion, they don't answer tech support AT ALL. You either take what they offer, or you don't. The control panel (for the $3/month option) is rather limited, and you have no blacklist features. There seems to be no way to tweak things, ask for assistance with filtering issues, etc. You just get what they offer.
For me, for a savings of $27 per address per year, that's a tradeoff I'm willing to make.
And by the way, I provide filtering for my family for free... it costs a few dollars extra per year, but I figure it's money well spent since Mom and Dad and the less geeky in my family don't get infected and I do less tech support than before.
Re: I do care how it works (Score:3, Interesting)
Re:Don't care how they do it.. (Score:3, Interesting)
20% on a Bayesian filter is ridiculously low; so low in fact I believe you are stretching the truth to make or point, or you're not training it.
My gmail account is quite old (gotten when only google employees were giving out beta requests), using an extraordinary common firstname.lastname account name, and since Jun 17, I've gotten 2247 spams. So that's what, 19 days? Gmail has *let through* probably fewer than 10 actual spam in that time frame (0.44%), and I haven't checked for any false positives.
So many spammers -from- gmail (Score:1, Interesting)
I run a moderate sized community and last month alone we banned over 50 throw away gmail spammerbots. That might seem to be a small number, but we're currently blocking .cn and .ru, and most other free email providers. Gmail addresses account for over 95% of our spam problem.
Be nice if they did something about that.
Re:Don't care how they do it.. (Score:2, Interesting)
I've been told by some people that part of the reason of the recent suckage of gmail's spam filter are people who think they're smarter than google and automatically mark all their messages as ham so they can get via pop or smtp to their computers and then run their own spamassassin/razor/bla tools on the mail. Thus, messages that are _obviously_ spam get marked as ham and are forwarded to the rest of users. I don't think it's the main reason, but worth sharing anyway in case somone knows more about this 'trend'.
Re:Don't care how they do it.. (Score:3, Interesting)
Spam Assassin is a great compliment to GMail's spam filters.
1) I use IMAP Spam Begone [rogerbinns.com] to check my google inbox and mark stuff as spam/not spam.
2) I use DMZ's remote SA-Learn [dmzs.com] to learn spam from my google spam folder (after I check it for false positives) and I use it to learn ham of stuff that IT marked wrong.
Result, I haven't had any spam make it through since I started using it.
(Both scripts do require editing isbg.py hasn't been updated in 5 years, so to work with newer python I fixed some things and sa-learn.pl needed to be edited to work with GMAil).
Just enable IMAP in gmail and go.