A Look At Google's Email Spam Prevention 176
CNet has a story about the security measures Google employs to protect their email systems and fight the never-ending war on spam. Their Postini team, acquired two years ago, has a variety of monitoring tools and automated response systems to find and block undesirable messages. Quoting:
"The system scores each message on numerous combinations of criteria, assigning a weight to each and then comparing the score to those in a database of several hundred thousand message types that have been flagged as good or bad from Postini honey pots and customer spam reports. ... To block fresh spam attacks not covered by existing heuristic technologies and viruses not covered by existing signature databases Postini relies on proprietary Zero-Hour technology to identify new outbreaks that show up in the traffic patterns and quarantine them for later rescanning. Customers can also create and build out their own white lists of message senders they trust and blacklist others they don't trust. It takes an average of 150 milliseconds for a message to be scanned by the antivirus engines that Postini licenses from McAfee and Authentium.
Re:Don't care how they do it.. (Score:5, Informative)
Pfft.. the internet became sentient sometime ago and used to babble like a baby. Since whatever it said was pretty much garbage, it was impossible for anyone to correctly figure out whether the noise was the baby's (spam) or from the tv (non-spam?). Now that the internet speaks more coherently it is far more easier for Google to figure out stuff that is coming from the internet - spam that is. It is rather obvious actually.
I wonder why yahoo has a miserable spam filter though; maybe Yahoo is like the careless parent who never gave a shit to figure out when the baby stopped babbling. And judging by the kind of spam I get in my hotmail box (it is all from microsoft), probably MS would be like those parents who insist on babbling themselves when the baby is around.
There, mystery solved! Now no one has to RTFA. Now if only someone made this into a car analogy for the greater good.
Re:Don't care how they do it.. (Score:3, Informative)
I get loads more spam than I used to. Something broke in Google's spam prevention about 4 months or so ago, and it's not been fixed yet. I redirect my email to my phone, where I get a notification of new email, and I've had to turn the sound and vibrate alert off because I got too much spam coming through.
Re:Don't care how they do it.. (Score:3, Informative)
Wow, what Bayesian filter are you using that is only giving you a 20% catch rate?
I'm using spambayes [sourceforge.net] (a pop3 proxy) and I would estimate it catches well above 95% of my spam. My inbox would be utterly unusable without it.
It requires some training - the more training you give it and the more religious you are, the better it works. I've trained it on around 3000 ham and 3000 spam messages and it is incredibly accurate (almost scarily, sometimes) at catching spam. False positives are extremely low - here's the stats it reports:
SpamBayes has processed 114790 messages - 56469 (49%) good, 54032 (47%) spam and 4289 (3%) unsure.
2328 messages were manually classified as good (2 were false positives).
2483 messages were manually classified as spam (829 were false negatives).
34 unsure messages were manually identified as good, and 1583 as spam.
SPAM volume patterns (Score:3, Informative)
What I find telling is how my SPAM volume rises and falls according to the American holidays. Whenever the Yanks have a holiday, SPAM drops to a trickle.
That to me is a clear indication that most SPAM originates in the US even though it mostly gets relayed through Asian proxies.
Re:"Postini"? (Score:3, Informative)
I helped a customer get off AOL's blacklist a couple months ago.
It was a straightforward process with an immediate automated reply.
In order to complete the process you must be able to receive an email at abuse@, postmaster@, or the technical or administrative contact for your domain.
The final email was from a human. It was completed the day following.
GMail Spam Filter != Postini (Score:2, Informative)
From the article:
"Google's Gmail antispam efforts are separate from those of Postini, which Google acquired two years ago, although it follows similar computerized operations and the teams have started to integrate the processes."
I've had email at an ISP that uses Postini, and I have email at Gmail. IMHO, Gmail > Postini.