Stories
Slash Boxes
Comments
typodupeerror delete not in

Comments: 176 +-   A Look At Google's Email Spam Prevention on Sunday July 05, @03:28PM

Posted by Soulskill on Sunday July 05, @03:28PM
from the click-here-to-report-as-spam dept.
google
business
internet
communications
security
CNet has a story about the security measures Google employs to protect their email systems and fight the never-ending war on spam. Their Postini team, acquired two years ago, has a variety of monitoring tools and automated response systems to find and block undesirable messages. Quoting: "The system scores each message on numerous combinations of criteria, assigning a weight to each and then comparing the score to those in a database of several hundred thousand message types that have been flagged as good or bad from Postini honey pots and customer spam reports. ... To block fresh spam attacks not covered by existing heuristic technologies and viruses not covered by existing signature databases Postini relies on proprietary Zero-Hour technology to identify new outbreaks that show up in the traffic patterns and quarantine them for later rescanning. Customers can also create and build out their own white lists of message senders they trust and blacklist others they don't trust. It takes an average of 150 milliseconds for a message to be scanned by the antivirus engines that Postini licenses from McAfee and Authentium.
story

Related Stories

This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • by Finallyjoined!!! (1158431) on Sunday July 05, @03:31PM (#28588401)
    I now get a couple of shed loads less spam. I used to check the apam directory for false positives. Don't bother doing that either.

    Go gmail :-)
    • by hansraj (458504) on Sunday July 05, @03:44PM (#28588493)

      Pfft.. the internet became sentient sometime ago and used to babble like a baby. Since whatever it said was pretty much garbage, it was impossible for anyone to correctly figure out whether the noise was the baby's (spam) or from the tv (non-spam?). Now that the internet speaks more coherently it is far more easier for Google to figure out stuff that is coming from the internet - spam that is. It is rather obvious actually.

      I wonder why yahoo has a miserable spam filter though; maybe Yahoo is like the careless parent who never gave a shit to figure out when the baby stopped babbling. And judging by the kind of spam I get in my hotmail box (it is all from microsoft), probably MS would be like those parents who insist on babbling themselves when the baby is around.

      There, mystery solved! Now no one has to RTFA. Now if only someone made this into a car analogy for the greater good.

      • Okay, I hope the the moderation to my post was an instance of meta-humor. Just in case someone who is ignorant about spam filtering techniques and believes the moderation that my post is actually informative or insightful: STOP! The internet is not really sentient (yet).

    • by jo42 (227475) on Sunday July 05, @04:37PM (#28588809) Homepage

      Don't care how they do it..

      Then I suggest that you don't really belong on /. ...

    • ...because it's actually not working - Gmail spam filter recently became very ineffective - i have to classify about 5-10 Viagra spams daily. (Google, have you heard of it? geez!) then it occurred to me that a while ago [slashdot.org] Gmail captcha was cracked, so I imagine spammers send themselves hundreds of spams only to classify them as "non-spam". - as a consequence, spams are now slipping through the crowd-sourced filter because the crowd is infiltrated. c'mon google this can't possibly that hard to fix!
      • Re: (Score:3, Interesting)

        I counter your anecdote with my anecdote! No, seriously - not to be an a$$ or anything, but I haven't gotten a single spam in GMail in over two years. There is none in the inbox, and none in the spam folder (label) either. I'm not sure why you are getting them, but it is clearly not everyone who is so afflicted (thankfully!). I'm not sure if it has something to do with accounts on different back end systems or what, but mine hasn't gotten any spam in one heck of a long time.
    • Re: (Score:3, Informative)

      I get loads more spam than I used to. Something broke in Google's spam prevention about 4 months or so ago, and it's not been fixed yet. I redirect my email to my phone, where I get a notification of new email, and I've had to turn the sound and vibrate alert off because I got too much spam coming through.

        • Spam Assassin is a great compliment to GMail's spam filters.

          1) I use IMAP Spam Begone [rogerbinns.com] to check my google inbox and mark stuff as spam/not spam.
          2) I use DMZ's remote SA-Learn [dmzs.com] to learn spam from my google spam folder (after I check it for false positives) and I use it to learn ham of stuff that IT marked wrong.

          Result, I haven't had any spam make it through since I started using it.

          (Both scripts do require editing isbg.py hasn't been updated in 5 years, so to work with newer python I fixed some things and sa-l

    • I used to have 20,000+ in my spam folder every day for years. Recently it dropped to the low 400's.

      But because there's much less spam, I actually check the spam folder quite often to see if there are false positives, and I almost always find a few. Makes me wonder how much mail I missed all this time?

    • by DrXym (126579) on Sunday July 05, @05:07PM (#28589005)
      Spam is now so bad for me on my home account that I reckon for every 100 messages, only two or three are legitimate contact. I literally get 200-300 spams a day. Bayesian filters will get rid of about 20%, and rules I've added such as deleting any email with cyrillics or other foreign characters still leave me with 100 or so to delete manually.

      I've set up GMail to filter my email and by comparison I'd say one or two spams get through. So I'm very happy with GMail's level of coverage. It's not perfect but it makes things tolerable. I'm not at all happy with Yahoo's level of coverage. Yahoo allegedly also has spam filters, but I've yet to see they actually work. It's not uncommon to find my email box filled with Nigerian and other scams.

      • Re: (Score:3, Informative)

        Wow, what Bayesian filter are you using that is only giving you a 20% catch rate?

        I'm using spambayes [sourceforge.net] (a pop3 proxy) and I would estimate it catches well above 95% of my spam. My inbox would be utterly unusable without it.

        It requires some training - the more training you give it and the more religious you are, the better it works. I've trained it on around 3000 ham and 3000 spam messages and it is incredibly accurate (almost scarily, sometimes) at catching spam. False positives are extremely low - here's the

      • Re: (Score:3, Interesting)

        20% on a Bayesian filter is ridiculously low; so low in fact I believe you are stretching the truth to make or point, or you're not training it.

        My gmail account is quite old (gotten when only google employees were giving out beta requests), using an extraordinary common firstname.lastname account name, and since Jun 17, I've gotten 2247 spams. So that's what, 19 days? Gmail has *let through* probably fewer than 10 actual spam in that time frame (0.44%), and I haven't checked for any false positives.

  • "Postini"? (Score:5, Insightful)

    by John Hasler (414242) on Sunday July 05, @03:34PM (#28588425)

    My previous ISP switched me over to Postini with no advance notice (we got a cheery note from marketing after the deed was done). Blocked half the spam and half the ham. They told us how to disable the filtering "features" but it turned out that all the filtering could not be turned off.

    I'm not with that ISP any more.

    • Re: (Score:3, Interesting)

      I don't take this to be anything more than single data point, but I've had the opposite experience with Postini. The university where I works may have it configured really nicely, since only rarely (i.e., 2-3 times per year) does spam make it to my inbox and slightly less rarely does ham get detained (less than one email a month).

      Sadly, the one thing that Postini DOES tend to do on occasion is relegate email I send to others on campus to their spam folder . . . .

      • Maybe it knows your email won't be that interesting to other people =). That's the next evolution I want to see, start blocking useless email (like chain letter jokes, etc.).
        • I wish that my ISP's filter would find Snopes candidates from my mother-in-law and relegate them to the bit bucket. But it never learns. Bayseian filter? No.... it learns only when a user spanks it.

          • by veganboyjosh (896761) on Sunday July 05, @06:15PM (#28589331)
            I used to get Snopes candidates from my mother-in-law a few years ago. I used to delete them without saying anything. Then I figured I'd try to teach her about the internet, and trusting things you receive in your inbox. I made an effort to track down whatever outrageous story she forwarded on snopes or wherever else, so that she'd see they weren't true, and stop sending them.

            Now, instead of getting emails from her with "I wonder if this is true. It sounds so amazing!", I get "I already checked Snopes, and while this one isn't real, it makes for a good story!" MLIA.
    • Re:"Postini"? (Score:5, Interesting)

      by icydog (923695) on Sunday July 05, @04:21PM (#28588703) Homepage
      I have had a similar experience with Postini, but from a different point of view. I usually use my own mailserver to send emails, and in the beginning I was greylisted and occasionally blocked by a few servers here and there, but after just a few quick emails here and there to ask why I was blocked, I was always promptly unblocked. I just use it for personal email so I'm not sending commercial or bulk emails. And before someone asks, no it's not on a dynamic IP or anything, it's in a fairly large colocation facility.

      Google is the only mail service that I know of who still just won't accept my emails. They make it very difficult to contact them. There is a form buried somewhere in their help system, but it says that they won't respond unless they need additional info from you, which leads me to believe that they never actually read anything submitted through that form. (I have tried a few times.) They also specifically say they don't take whitelist requests. I have SPF records, I have correct reverse DNS, I'm not on any blacklists, etc.

      This means when I send emails to my friends who use Gmail, or comparies who use Postini, I get blocked without cause. Then I have to use a different server. It's kind of annoying.

      (Why do I use my own email server? Because I can. This is /., after all.)
      • Re:"Postini"? (Score:5, Interesting)

        by Jay L (74152) * <`mf.yaj' `ta' `hsals+yaj'> on Sunday July 05, @04:54PM (#28588927) Homepage

        Google is the only mail service that I know of who still just won't accept my emails.

        I had a similar [wordtothewise.com] experience [slashdot.org]; I run my own mail server, send no bulk mail whatsoever, and both Postini and GMail independently decided I was a spammer. No DNSBLs had me listed, ReturnPath was happy, etc. Meanwhile, I was blocked from sending mail to my lawyer, my financial advisor, my chiropractor, etc., all of whom turned out to be downstream from Google. Despite Google's claims that the customer is in full control of filtering, none of them were able to get at my e-mail without getting their sysadmins involved - which often required discovering that they had sysadmins at all.

        Worse, Postini's spam filtering takes its own output as input. Once it's scored a message of yours as spam, future messages will be more likely to score as spam - which of course makes any subsequent messages even more likely to score as spam. Brilliant. At one point, my spam score from a triple-signed (SPF/DK/DKIM) server was 98 out of a possible 100.

        Google's philosophy of "we don't do it unless we can automate it" works horribly when it comes to customer service. There's no feedback loop, no whitelisting, no channels, no nothing. It's SPEWS all over again, or perhaps the Kafka International Airport [theonion.com].

        But Google has no reason to worry about false positives; the more messages they call spam, the more spam they can say they blocked. Perverse incentives.

      • Re:"Postini"? (Score:5, Interesting)

        by SanityInAnarchy (655584) <ninja@slaphack.com> on Sunday July 05, @05:10PM (#28589029) Journal

        For what it's worth, Gmail has been just the opposite for me. It's Yahoo and AOL which randomly decide to block me -- sometimes with some cause, sometimes just because it's on a residential connection.

        Yet Gmail never so much as greylists me -- everything goes straight through, every time.

          • Re: (Score:3, Informative)

            I helped a customer get off AOL's blacklist a couple months ago.

            It was a straightforward process with an immediate automated reply.

            In order to complete the process you must be able to receive an email at abuse@, postmaster@, or the technical or administrative contact for your domain.

            The final email was from a human. It was completed the day following.

    • Re: (Score:3, Interesting)

      My publisher uses Postini. Whenever I send my editor an article, I need to also send an IM so he can check it isn't in his spam folder (happens a good 10% of the time). Meanwhile, SpamAssassin has been giving me no false positives and very few false negatives for years. I'd much rather have false negatives than false positives in a spam filter. A false positive means I can lose (or have delayed) an important email. A false negative just means that I have to waste a second or two clicking the 'spam' butt
      • Re: (Score:3, Insightful)

        Tell him to look up the definition of "whitelist".

        My guess is the system runs much more optimally when your entire address book is whitelisted.

    • Re:"Postini"? (Score:4, Interesting)

      Have you noticed? GMail gives one no way at all to sort the captured spam. Since I still endure false positives from the system and there is NO way to disable or bypass it, having means to sort all of it by From:, To:, and other criteria would make it easier to identify the false positives and rescue them from the trash bin.

      Well, I'll take that back, in part: that applies to the Webmail interface, but if ones uses IMAP with a local IMAP client, then the spam folder could be subscribed and sorted within the client. God only knows how GMail's system interprets the dragging of a message from Spam to Inbox via IMAP: does that automatically whitelist that sender in the future, or do I have to still log into the Web site and identify it as Not Spam manually?

      • Re:"Postini"? (Score:4, Insightful)

        by rm999 (775449) on Sunday July 05, @04:38PM (#28588819)

        "there is NO way to disable or bypass it"

        Have you looked into filters? They added an option to "Never send it to Spam" about a year ago. You can create custom white lists with this, or just include everyone in the filter and totally bypass the spam filter.

      • Have you noticed? GMail gives one no way at all to sort the captured spam. Since I still endure false positives from the system and there is NO way to disable or bypass it, having means to sort all of it by From:, To:, and other criteria would make it easier to identify the false positives and rescue them from the trash bin.

        I haven't noticed - filters make this pretty trivial:

        in:spam from:blah

        • Re: (Score:3, Insightful)

          That's irrelevant: you'd have to KNOW who it was from in order to employ a SEARCH like that. That's not useful at all when you aren't looking for something specific.

          • Take a deep breath dude, was trying to give you info that I thought might help. Now it seems that you've presented a moving target. You first said:

            having means to sort all of it by From:, To:, and other criteria would make it easier to identify the false positives

            Now you say:

            That's irrelevant: you'd have to KNOW who it was from in order to employ a SEARCH like that. That's not useful at all when you aren't looking for something specific.

            If you don't know who it's from, to ,etc how is sorting by these fields going to help you filter out false positives? Since you now posit that you don't know who it's from, then that won't give you any information that you can use. In addition, you don't need to be searching for something specific to use the filters that are available.

    • My previous ISP switched me over to Postini with no advance notice

      Who uses their ISP's email/webspace anymore anyway? It makes switching ISPs much more difficult, unless you don't mind breaking all your old addresses or feeling stuck with your current ISP.

      • > Who uses their ISP's email/webspace anymore anyway?

        Not me any more, and that was one of the reasons. I pay Newsguy for Usenet and email service and also have another address provided by friends. All I want CenturyTel to do is handle packets.

  • ...has a quick look and goes back to catching up with news on the MailScanner mailing list.

  • by Anonymous Coward on Sunday July 05, @03:44PM (#28588487)

    but what I really want to tell you is that I've inherited a great deal of money and I need someone to help me transfer it to the US. I live in Nigeria. You all seem to be great gentleman, so I will pay appropiately.

    Contact me.

    • Re: (Score:3, Funny)

      by Anonymous Coward

      You all seem to be great gentleman

      You must be new here.

  • part of gmails phishing filter seems to do this

    if(hyperlink in email ends in .exe)
    {
        isphishing = true.
    }

    Even if this is an email from someone in your whitelist and is merely quoting text from your own message you sent them.
    And there seems to be NO way to prevent a message with .exe in it to be marked this way :(

  • by Pessimist+Cynic (1587497) on Sunday July 05, @03:58PM (#28588573)
    They can filter out the obvious spam mail, but some spammers are so clever and so well hung - because they've taken some DrMaxMan to acquire an enlarged sexual wand with which you can perform better and be bigger for f.r.e.e - that they can actually embed their spam offers inside real messages in such a way as to be completely undetectable by filters.
  • It was me who submitted that old /. story. I'm still with Postini, and it still does the job. The problem? Well, no spam filter is 100% effective.. and just about every time Postini lets spam through (very rarely), then they phone up the helpdesk irately and say "Postini should have stopped this!".

    So, the product is still great. Tech support has gone downhill though. Anyone who has tried to deal with Google tech support for anything will know how it feels..

    • Re: (Score:3, Interesting)

      I signed up with Postini just as it was acquired by Google. Before that I'd used SpamSoap, which worked great but was declining in effectiveness (more false negs) but not in price ($30 per month is a lot for a small business). Postini and then Google were far more reasonable at just $3 per year per address (for the less-flexible controls). I get maybe one or two delivered spam per week, usually when I also see a corresponding spike in filtered spam which indicates a new attack of some kind. I get only one o

  • Praise Gmail (Score:3, Interesting)

    by zhilla2 (1586095) on Sunday July 05, @04:10PM (#28588633)
    This is great for business mail too... small company where I work was literally BURIED with spam until we moved to gmail. Since their mail addresses were "in the open" on our website for years, some of them get 200+ spams a day. Now, if 1 in 1000 passes, it's a bad day. Also, in my private inbox, I had an VERY old mail address still redirected to gmail address... turned out that was the source of 1/2 spams (100+ / day). But those were filtered too without problem. So far so good... not a single false detection for ham. Nothing but praise so far. Disclaimer: I do not work for gmail. I am the genuine satisfied customer with smile on my face, from "after" picture, as seen on TV!
  • by Peaquod (1200623) on Sunday July 05, @04:17PM (#28588673)
    At least 75% of my spam is addressed as though it was sent from *my* gmail account. Of course, it's easy to set up a filter to reject all such spam, but then I lose the ability to send reminder messages to myself. Seems like it would be extraordinarily simple for google to outright reject messages that claim to be sent from their servers that in fact were not. I sure wish they would!
    • I believe e-mail spoofing (where the spammers spoof the header to make it look like it comes from you) is completely different than sending e-mails to yourself, and gmail knows this. That said, when is the last time a spoofed e-mail actually made it to your inbox?

    • by hidden (135234) on Sunday July 05, @04:46PM (#28588879)

      Keep in mind:
      It's a perfectly legitimate (and common) for non-webmail users to have their outgoing server be their local ISP. So if google did what you're suggesting, all those people that use an IMAP client to receive their gmail, and send via their ISP wouldn't be able to send to other gmail users

  • McAfee (Score:5, Interesting)

    by contrapunctus (907549) on Sunday July 05, @04:43PM (#28588845)
    So by using gmail, am I indirectly making money for McAfee?
  • In my humble and largely anecdotal experience, Postini works well. We send out e-mail that can often be flagged as SPAM when we perform penetration testing, and Postini seems to be the toughest to get around. We see in-house devices such as IronMain, and outsourced services such as MXLogic and FrontBridge/hosted Exchange, but Postini seems to do the best at stopping illegitimate messages. The company I work for uses this it as well, and logging into my Postini inbox I see a lot of spam but no false positive

  • Don't know about anyone else... I've been with gmail since it really *was* a beta, and it has been pretty good about not letting spam through. Past couple of months though and I have been getting three to four a day through. Are the spammers getting better or is the filter getting worse?
  • SPAM volume patterns (Score:3, Informative)

    by flyingfsck (986395) on Sunday July 05, @06:54PM (#28589533)

    What I find telling is how my SPAM volume rises and falls according to the American holidays. Whenever the Yanks have a holiday, SPAM drops to a trickle.

    That to me is a clear indication that most SPAM originates in the US even though it mostly gets relayed through Asian proxies.

    • One of my complaints about Postini (and whatever it is that CenturyTel uses) was that the "virus" filters cannot be turned off. I have no Microsoft or Apple software.

      • Re: (Score:3, Insightful)

        As an email administrator - I wouldn't give a user the ability to disable virus filtration on their email account - even if I knew they weren't a direct threat to any known virii. Too many stupid people out there know how to use the FWD button.

        I know what you're saying, but since you're probably the smartest user out of the tens of thousands that use your email server - they're not likely to give you a one-off option.

    • by Ron Bennett (14590) on Sunday July 05, @05:56PM (#28589261) Homepage

      150 milliseconds sounds fast, but equates to only 7 messages per second.

      Sure that may be faster, presuming it's a deep intensive scan, than what one can do on their home PC, and yes Google has zillions of boxes ... but anyways, my point is that 7 messages per second illustrates the very real, high cost of dealing with spam; scanning of just a million messages, which is a fraction of the spam volume, at 7 messages per second, takes well over a day of computer time.

      Ron

      • Re: (Score:3, Insightful)

        Did you have an easy to guess username?

        Just because you didn't send email from "robogun@gmail.com" doesn't mean your robogun@att.net isn't on a spam list somewhere. How do you increase the size of a spam list exponentially? strip all the domains from the addresses and find common names... then generate one email address for each domain you want to hit.

        Ta-da... spam email sent to accounts that were never used. This could indicate that google's directory harvest attack identification methods need some fine tu

"This generation may be the one that will face Armageddon." -- Ronald Reagan, "People" magazine, December 26, 1985