Some Anti-Spam Vendors Blocking and Slowing Gmail

fiorenza writes "Google's Gmail (and corporate mail) are being throttled and sometimes blocked by some anti-spam services, including MessageLabs and Antigen. Ars Technica reports that the blocking is a result of the Google CAPTCHA crack, which has allowed a deluge of spam from Gmail's clusters. Most users won't get blocked mail, but Ars confirmed with MessageLabs that Gmail delivery delays are to be expected."

Gmail and others blocking legit domains, so hey

There were number of times where my emails are silently deleted from Hotmail or even gmail, so hey. Welcome to the world of screwed up SMTP protocol. And all thanks to spammers.

Today email is less reliable message delivery medium than regular mail which is quite sad considering all transactions in SMTP were considered to be, well, transactions. An acceptance of email by destination means it is delivered, not going to /dev/null. Want to filter spam? Reply with 5xx codes instead - not accept with 2xx and then bin it (unless mailing list headers found in mail, there you can drop spam)

Re:Gmail and others blocking legit domains, so hey

I definitely agree with you, if a mail server accepts my mail with a 200 code, then the mail *should* be delivered. Even if its put in someone's spam folder, the message should get there. That's one of my pet peeves. That being said, from my experiences when setting up my mail server, Gmail was probably one of the best about not blocking legit mail (I've had an SPF record since the beginning though). I had lots of problems with Hotmail, and I think my mail was usually marked as spam by Yahoo until I enabled DKIM signing. With SPF records and DKIM, I don't think I have any major problems (though my mail server handles a pitifully small amount of mail, so its not like we're going to get marked as a bulk sender).

Re:Gmail and others blocking legit domains, so hey

Really? Do you have any idea the resources this would take for some organizations?

Based on stats from my frontend SPAM filters 80 - 90% of ALL mail receive in a day is SPAM.

On my reports some individual users are targeted with between 1500 and 2000 SPAM messages a day. There storage quotas would probably be exceeded over night from SPAM alone.

I would need to increase my storage capacity immensely if I allowed every spam message to get to the users Junk folder. Not to mention the extra bandwidth of allowing all those mail delivery connections to complete OR to send NDRs to forged senders that are going to bounce back at my system and cause even more load.

Re:Gmail and others blocking legit domains, so hey

True, and it is really not necessary to pass all mail. In my experience weeding out mal-configured mail servers (i use postfix rules and greylisting) takes care of over 90% of spam. The rest gets caught by an RBL or tagged by spamassassin and sent to the users spam folder. Things may change any day though depending on future strtegies by spam senders, but at the moment it works quite nicely.

Re:Gmail and others blocking legit domains, so hey

Here's what happens. On SMTP level. << is for what server sends. Rest is what is sent to server.

<< This is your great SMTP server. Yo!
<< 220 super.server.net ESMTP
HELO srv.my_super_subnet.server.net
<< 250 srv.my_super_subnet.server.net
MAIL FROM: <[e-mail]>
<< 250 2.1.0 Ok
RCPT TO: <[e-mail]>
<< 250 2.1.5 Ok
DATA
<< End data with <CR><LF>.<CR><LF>
Subject: Yo
From: Bob Superman <[e-mail]>
To: My Buddy <[e-mail]>

Want some viagra?

.
<< 554 5.7.1 Rejected, id=sdsada - SPAM
QUIT
<< 221 2.0.0 Bye

See?? No backscatter. The pre-queue filter runs *before* the message is accepted after the . is on the new line indicated end of message.

What happens now is they get,

<< 250 2.6.0 Ok, id=fsffs FROM blah Ok: queued as foo

or similar response. Then the filter runs and junks the mail! *That* runs email. I send out email, and it get junked. It gets delivered 50% of the time because some wise guy runs some new magic filter - no spam gets through, and 50% real messages get binned. Then people that should get mail complain that they never get mail (and not just from me).

If you reply to message from a post queue filter you get backscatter. This is wrong way of doing things. If you reject mail in pre-queue, there is NO backscatter.

The *only* reason to run after queue and drop silently is for mail marked as Bulk, like mailing list software marks mail as Bulk precedence. Any other mail should be treated as a *no* mail lost priority. Otherwise we may just abandon SMTP altogether.

Re:Gmail and others blocking legit domains, so hey

Just to add something, the problem with 5xx replies is filter is *before* queue so some mail may be delayed and servers need to be contacted a few times before they get a delivery slot. For example, say gmail can filter 1 million messages at a time. That means 1 million open connections. So, if you are connection 1,000,040 you get 4xx response - temporary failure due to no available resources. So try again later.

This is not a problem, really. You can wait a few days until you can deliver the message as long as it is *delivered* eventually. /dev/nulling spam while accepting it with 2xx code is like burning unopened envelope at post office because it was typed instead of handwritten indicating possible spam.

Pre-queue filter with only 1 unique IP connection at a time to mail server. Problem solved.

Huge email servers get reasonably constant and predictable amount of mail per day and per hour and even per minute. They can plan pre-queue filtering with some margin for any spikes. And if there is a huge bomb and your mail doesn't get there for 7 days and your server gives up, hey, at least you get a "Could not deliver the message because destination was not available". Much better than "err, never got any mail from you" from the destination party.

Google wins

The missing part of this story really is that Google`s Gmail client has very effective anti-spam filtering. I can see why companies who earn their keep protecting typical client-side email systems, would want to make Gmail obsolete or ineffective. Spammers might use Gmail as a tool to spam, but with good filtering it really doesn`t cost that much compared to the loss of time spent weeding out ham from spam.

Gmail should go back to cell phone authentication

Gmail should go back to their old scheme, where you had to have a cell phone to receive your password, and you could only have one gmail account per phone. That would slow the spammers down.

If you don't have a phone, you're probably not a good candidate for an advertiser-supported service anyway.

Re:Gmail should go back to cell phone authenticati

Expect to see a technological solution, this isn't a company full of middle managers or people who are used to losing technical battles.

If I were a betting man I'd say Google will either A) release a new authentication/authorization scheme for creating new accounts, or B) they'll evolve their current system to be resistant to delivering false negatives on bot provided responses.

Because honestly, isn't this just graphical/visual acuity based Turing test that needs to be treated as "passed" by the industry? The reasoning being: the equivalent of Alicebot now exists for the graphical world, so the test needs to be re-engineered to test another (currently) unpassed Turing style evaluation.

Based on that realization: the whole reason capcha's are stupid is that if you keep the existing design but try and make it "harder" to break, the designer of the Bot need only account for that change and not an entire redesign.

All this sounds like a great technical challenge: think up a new Turing test... When in reality those posting go back to invite only are absolutely right but it's likely we won't see that come out of Google.

Re:Gmail should go back to cell phone authenticati

Gmail should go back to their old scheme, where you had to have a cell phone to receive your password... If you don't have a phone, you're probably not a good candidate for an advertiser-supported service anyway.

Since when does cell phone == phone? Tons of people don't have cell phones, and most of them are consumers of various goods just like people who do have cell phones. It's amazing how the 'net culture makes it easy to write off huge swaths of the population just because they don't have or want the latest gadgets.

Re:

With the current state of the world economy, no one will be willing to pay for something that they get used for free.

Re:It's ok though...

Umm... I have used a number of commercial email systems (in-house for major companies and institutions) and none of them could provide a service that was even remotely close to what Gmail does for free.

Re:It's ok though...

Then you haven't used Exchange, etc.

Exchange trumps Gmail easily. No Contest.

Regards,

Re:It's ok though...

I have yet to see Exchange work well in any environment over a few dozen people; certainly not without investing large amounts of money on duplicate servers and hardware. Included is my favourite Exchange analogy: If the same method that exchange/outlook uses to store email were used in the real world as a paper filing system: Every document is translated into Greek, and the original is burned. Then they are all glued together into one solid block and stuffed into a magic box with a tiny slot, through which you can talk to a little gnome who somehow gets each message for you as needed. Sometimes the gnome gets confused and it takes hours (sometimes days) for him to sort things out; meanwhile he can't find your documents until he is totally finished becoming unconfused again. As an added bonus the gnome costs several thousand dollars and when he dies every few years you need to buy a new gnome. Oh and if the first box gets (arbitrarily) full you have to buy another special gnomebox, which of course costs $$$

Re:It's ok though...

Wow, if you're having that many problems with Exchange, your sysadmins need to do a better job. Exchange is generally a pretty good mail/groupware server for corporate environments. If you throw an Exchange server together in five minutes, then yeah, you might have some problems, but as long as you think it through beforehand (and like with anything computer related, have a good backup strategy) it should work pretty well unless you have some really unusual requirements.

Re:It's ok though...

It's funny how a good backup strategy involves NOT using anything from Microsoft to backup exchange. I mean seriously, how long has been exchange been out and Microsoft can't make a backup program that can backup and restore individual mailboxes? WTF! Oh, and I don't want to take the store offline to do it.

Re:It's ok though...

That is *completely* incorrect. In Exchange 2003 prior to SP2, the limit for the mailbox store was 16 GB. In SP2 they upped that limit to 75 GB, which really is probably enough for most of the small organizations that probably just have a single server running Exchange Standard.

Here is a document about it, scroll down to the part where it says Licensed Database Size Limit. http://technet.microsoft.com/en-us/library/aa998066.aspx [microsoft.com]

In Exchange 2007, Standard Edition can have up to five mailbox stores in each of five storage groups. And there is no limit on size. http://www.msexchange.org/tutorials/Exchange-2007-Store-Related-Changes-Improvements.html [msexchange.org]

Its also mentioned on the Microsoft Exchange page on wikipedia.

I'm not saying that Exchange is a perfect mailserver for (or worth the cost in) every situation, or denying that Microsoft does some really annoying things, but please try to get your facts straight before you complain about a piece of software.

Re:It's ok though...

In Exchange 2003 prior to SP2, the limit for the mailbox store was 16 GB. In SP2 they upped that limit to 75 GB

16GB per mailbox should be enough for most users for a few years to come, what is all the fuss about?

No 16GB to for the mailbox store, which houses ALL of the mail boxes, IIRC. Thus, the reason for having 10M-20M limits on peoples inbox.

Re:It's ok though...

>Exchange trumps Gmail easily. No Contest.

As a source for spam, and a plague of server-generated 'automated' notices, Exchange beats EVERYTHING.

Exchange is fine if you keep it where it belongs: inside a workgroup or protected by a SMTP-protocol filter (which is not running on the same box).

Recently I had to defend a customer who was the target of a DDOS... 80% of which were "bounces" from Exchange (forged From: undeliverables, permanent Out Of Office, DSNs, Mailbox full emails, etc). Exchange is pathetic in terms of controlling what gets "onto" the server.

By comparison, Google mail is a VERY good Internet citizen. They may have had Captcha compromised, but they'll plug it up. I'll them over their competition anyday.

Re:It's ok though...

And YOU definitely haven't used Gmail, which trumps Lotus Notes by a large margin... Oh, wait...

Re:It's ok though...

And you certainly haven't used a telegraph. I only get spam about once every month or so, and I can usually ignore it by about the 15th beep or so.

Re:It's ok though...

Did you seriously just post an unsolicited commercial message in a thread about how unsolicited commercial messages are watering down legitimate communication mediums?

Re:Crack down

I think the safest thing they can do right now is return to their invitation only registration in an effort to close the breach. Then they have to start deleting spam accounts quickly before the spammers adapt to inviting themselves. If they are lucky they will be able to delete spam accounts faster than they multiply.

Re:They will, eventually, be cracked again.

What they need to do is have a process for detecting when an account is spamming. Now, you and I would just say "when an account is sending 10,000 messages a day" and that would be correct for about 99.9% of the cases.

No, that's the whole point of defeating captcha. Instead of sending 10,000 messages from 1 account, send 10 messages each from 1000 accounts.

Re:They will, eventually, be cracked again.

welcome to [e-mail] (just kidding, but donotreply.com gets a lot of interesting e-mail, I just wondered what they'd do if they started getting 'spamtrap' addressed mail)

well, making special spamtrap e-mail addresses and putting them in the clear on usenet, message boards, or even on social networking sites owned by google, and making sure the content is boring drivel no one would e-mail that person about. well, i mean how could you decide how to make boring drivel that would still put their address out on sites? 'first post' messages?

wouldn't someone notice that google got 'first post' every time on 123 consecutive front page articles? wouldn't they? though and e-mail them a congratulation and get spam busted?

i mean i know i can post boring irrelevant information, but i can't guarantee that if an e-mail is tied to that identity that someone won't e-mail me....

so spam traps are harder to implement than one would think, unless they're in 'hidden' code. EG: you go to a website, the e-mail is in the html, but never shows on the page... and if you do that, then they might make a scanner that nullifies those addresses... once the realize what's happening.