Researchers Claim "Effectively Perfect" Spam Blocking Discovery 353
A team of computer scientists from the International Computer Science Institute in Berkeley, CA are claiming to have found an "effectively perfect" method for blocking spam. The new system deciphers the templates a botnet is using to create spam and then teaches filters what to look for. "The system ... works by exploiting a trick that spammers use to defeat email filters. As spam is churned out, subtle changes are typically incorporated into the messages to confound spam filters. Each message is generated from a template that specifies the message content and how it should be varied. The team reasoned that analyzing such messages could reveal the template that created them. And since the spam template describes the entire range of the emails a bot will send, possessing it might provide a watertight method of blocking spam from that bot."
Is there the checklist for why this won't succeed? (Score:1, Insightful)
Is it coming?
"Perfect"??? (Score:5, Insightful)
effectively (Score:3, Insightful)
"effectively" = "not quite good enough to actually work"
Seems to make sense (Score:5, Insightful)
Reactive only (Score:5, Insightful)
"Perfect" (Score:5, Insightful)
Headline tomorrow (Score:5, Insightful)
A team of hackers from Russia are claiming to have found an "effectively perfect" method for countering spam blocking technology. The new system deciphers the templates Spam Blocker is using to filter spam and then teaches spam generators what to write.
Calling BS (Score:4, Insightful)
I don't believe any spam filter that advertises 100% accuracy, especially one claiming to do it by figuring out the spam email 'templates'
Halting problem (Score:2, Insightful)
and then the researchers discovered the Halting problem and pretended it didn't exist.
Re:"Perfect"??? (Score:5, Insightful)
Worthless. Completely Worthless (Score:5, Insightful)
I've said it before, and I'll continue to say it - spam is an economic problem. Until something is done to address the money that spammers make, they will continue to find ways around these "effectively perfect" "discoveries".
Re:"Perfect"??? (Score:5, Insightful)
Fine with me. Most spam I get is obviously a template, since I get the same one for weeks. This would stop those additional sent copies. The false positive rate on this kind of thing is effectively 0%, so I'm willing to have it be an additional check on my email.
If it can stop a lot of this kind of spam, that's fine with me. Let it be an arms race. If the spammers have to make up new templates every 4 hours, that's going to make things a lot harder.
This isn't a cure for all spam, it's a fantastic filter for one (of the biggest) kinds of spam. Only headline makes it sound like it will solve all spam.
Information Security Puffery (Score:4, Insightful)
As a researcher in the academic side of the Information Security field, I can't help but notice a significant increase in the level of puffery and misleading promotion of research results. Self-promotion obviously isn't new, it's just that as the amount of newspaper-assisted promotion increases, the level of accuracy has dropped significantly. And more importantly, researchers seem much less apologetic about it. It's generating some real blowback.
The best recent example I can think of is Vanish, a cryptographic system for "destroying" data that was proposed out of University of Washington. It's not just that the system was broken [utexas.edu] a few days after it was presented, it's that this relatively minor result got more press than all of the perfectly legitimate crypto-systems research that was going on at the time. In fact, during the same time period a guy named Craig Gentry solved [techtarget.com] a major open crypto problem --- namely, how to compute on encrypted data --- and it got a fraction of the press coverage.
Not that I'm saying these researchers specifically asked to have their invention described as an "effectively perfect" solution to preventing spam --- which I guarantee you 100% it is not --- but that by going out on a University-encouraged PR junket, they've more or less encouraged this kind of coverage. This kind of stuff is damaging; people should describe their work as what it is. They've developed a technique that is highly effective at filtering /current-gen/ spam generators, in the lab. It won't stop all spam, and it's not effectively perfect, since spamfiltering is by nature an arms race. But of course that's not how it's going to be presented. In the long run this'll just make people more jaded with our field.
Re:What about changing the templates (Score:2, Insightful)
Man, building spamming systems and finding ways to vary the content but not the message seems like a fun cat-and-mouse game. Too bad it's so evil. Can I cut off my Guilt Lobe?
Re:Is there the checklist for why this won't succe (Score:5, Insightful)
Note that the "good guys" revealed their methods immediately after discovery, which means the "bad guys" can start looking for a workaround. The "bad guys" won't make the same slip.
Re:"Perfect"??? (Score:4, Insightful)
There is a final solution: make sending spam more expensive. Spammers will only spam so long as it's mind-blowingly wealthy. If you can raise their operating costs and bump them down from "mind-blowingly wealthy" to only "obscenely wealthy", they might switch to other lucrative immoral industries like manufacturing printer ink.
What this does is increase the computational power required to generate a spam email. The method they described sounds like it's self-learning (just hook it up to a spambot "oracle" and it'll figure out the new template), so spammers will likely have to abandon the use of templates altogether. If you increase the amount of computational time required to generate spam, you decrease the amount of spam sent and really decrease the profitability of it.
We keep pushing the requirements for spam further and further up the computational totem pole (or Chomsky hierarchy, if you will) and you get closer and closer to a point where spammers are going to have to create strong AI to write spam. If they fail, we don't have spammers anymore and if they win, well we have spam, but we also have strong AI! Win-win, I say.
Re:Again with the stupidity (Score:4, Insightful)
Had there been no spam filters, we'd all receive about the same amount of e-mail spam as we receive in the postal mail world.
I can't imagine what you base that statement on. Real-world junk mail is limited by the fact that it costs money to print and mail junk mail. Neither applies to spam.
Spammers aren't just competing with spam filters. They're also competing with each other for attention. Even in the absence of spam filters, the spammers would continually seek new ways to get more of their spam into your inbox than their competitors.
In fact, they might well invent the spam filter, with a deliberate back door so that their spam sails through while their competitors are dropped.
Re:Is there the checklist for why this won't succe (Score:5, Insightful)
However, the reason you use templates, rather than word salad or the first 100kb of
Re:Calling BS (Score:3, Insightful)
I don't believe any spam filter that advertises 100% accuracy, especially one claiming to do it by figuring out the spam email 'templates'
Yeah, and calling this a discovery stretches credulity. Who here thinks that Google, Yahoo, Hotmail, and your favorite big mail service provider, don't already do some version of this?
Re:Is there the checklist for why this won't succe (Score:4, Insightful)
Asshatitude always applies because you can never anticipate the next step in asshatitude evolution. They will always find new and innovative ways to be asshats.
Re:"Perfect"??? (Score:3, Insightful)
Re:Worthless. Completely Worthless (Score:3, Insightful)
Spammers send spam because it makes them money.
Agreed.
It makes them money because people are stupid
Not directly. The spammers themselves are paid by moderately smart people who are selling products online that are often of questionable legitimacy. While some of those customers are stupid, there are generally fairly crafty individuals making money off of the customers along the way.
The question is: why are people stupid, and how can we make them smarter?
You could ask the same question in the light of why 419 scams work, why old-school pyramid schemes work, etc. Money can make smart people pretty dumb at times.
I would argue that spam is an educational problem
You will not succeed in educating the problem away. Unless you want to impose some sort of requirements for users to access the internet (a la driver's licensure), you won't succeed at educating all the users and getting it to stick. It's like trying to design a better mouse trap; nature will just make a better mouse and then you're back to square one.
If you want to make a meaningful difference in the spam volume, you need to stop the money from flowing to the spammer.
Re:Is there the checklist for why this won't succe (Score:4, Insightful)
Now suppose my account were compromised and you got this exact message from my personal email, where the jpeg is a Viagra ad. There is absolutely nothing there for your spam blocker to latch on to, unless it parses the content of the jpeg itself. Anyway, blocking stuff like this would lead to unacceptably many false positives.
Re:Not our claim... :-) (Score:5, Insightful)
You mean a Slashdot editor posted something sensational, and people didn't RTFM and believed the summary/headline? Never!
Re:Is there the checklist for why this won't succe (Score:4, Insightful)
(X) technical ( ) legislative ( ) market-based ( ) vigilante
Has anyone ever suggested all of these? The government offers a contract and clears the legislative barriers to a company making vigilante robots which would hunt down and kill the families of all spammers while making the spammers watch?
Assuming these robots can fly, have powerful metal claws, and cannot be stopped, I can't see any problems on your checklist.
( ) Spammers can easily use it to harvest email addresses
() Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
() Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business
Nope. None there.
( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook
There are currently laws expressly forbidding the construction and operation of mass murder machines, but that's why I suggested we get rid of those laws.
( ) Ideas similar to this are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
(X) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
I do realize some wouldn't trust the company controlling the deathbots, which is why -I- would be the governing authority once they were operational. You can trust me because I promise to only kill you if you're related to a spammer.
Re:obvious (Score:3, Insightful)
and don't forget (Score:4, Insightful)
Spam isn't kosher anyway!
Re:What about changing the templates (Score:4, Insightful)
I think you're forgetting that the criminals who run botnets aren't as worried about damaging the normal operation of the Internet as the rest of us might be.
We start detecting their templates; they start making their templates more and more flexible. We chase, giving our filters broader and broader definitions of "bad" email. Clever spammers start sacrificing the percentage of thier mail that's coherant just to increase the output range of their templates, forcing the template-recognition filters to get looser. Eventually the filters become useless because they can't pick out every variation that could come from a template without also capturing a lot of legitimate messages.
Or something else happens that renders the filters useless. THe point is - yes, it's a win in that it fights techniques used today. No, it is not the grand victory proclaimed by the headline.
Re:Is there the checklist for why this won't succe (Score:4, Insightful)
The truth is that spam has been successfully fought by filters without compromising legitimate email. Furthermore as Paul Graham had stated, spammers have been forced to yield in smaller text-based messages or in-line images.
In particular,
(X) Mailing lists and other legitimate email uses would be affected
Possibly but the probability of losing legitimate email by modern heuristics is (proven) smaller than the probability of accidentally deleting it when it is mixed with spam.
(X) Users of email will not put up with it
They do, sometimes without their knowledge
(X) Many email users cannot afford to lose business or alienate potential employers
They would lose more without filtering. See 1st argument.
(X) Asshats
How ?
(X) Eternal arms race involved in all filtering approaches
(X) Extreme profitability of spam
And also extreme profitability in having a working e-mail address.
(X) Bandwidth costs that are unaffected by client filtering
This isn't the mid 90s anymore.
(X) Ideas similar to this are easy to come up with, yet none have ever been shown practical
The practicality of heuristic filtering (SpamAssassin etc) is proved by its transparency. Even old e-mail clients such as Outlook 97 can filter out email marked by X-Spam headers. Gmail and the rest of the privacy traders do it for you automatically.
(X) Why should we have to trust you and your servers?
Run it locally. Mozilla Messaging does.
(X) Feel-good measures do nothing to solve the problem
Age old forms copied from the newsgroups can't be used as arguments anymore. Time to be creative again!
(X) Killing them that way is not slow and painful enough
But cutting down their profit is.
Re:Worthless. Completely Worthless (Score:3, Insightful)
As long as there is money to be made in spam, spammers will continue to send spam.
But if the US government was to threaten the US based credit card companies that process every single one of these transactions there would be no more money, and no more spam.
Which transactions should they block?
It's also important to keep in mind that spammers don't make money from selling V1AGRA. Spammers make money from other people who want to make money by selling V1AGRA. The distinction is important because it doesn't really matter whether money can be made by selling shady products or not. As long as there's a sucker who *believes* they can make money by selling the shady products, the spammer has a customer. When that one wises up, there are 10 more waiting.
Re:Recognizing spam is easy, if you see enough (Score:3, Insightful)
Amen to that.... we moved our email accounts to Gmail a few years back.
Currently I get maybe two or three spam emails a week across three accounts, two of which have been in active use on the Internet for more than a decade.
Of course if I look in the spam folder, I see that in actual fact anywhere up to 50-100 a day per account. Not my problem. Possibly a problem for Gmail. But they seem happy to undertake to offer the service and remove it for me.
I do have to deal with it elsewhere.. I manage various Google Groups and it is an on going battle to audit membership requests so that you don't get spammed that way.
Re:Is there the checklist for why this won't succe (Score:1, Insightful)
What we need are "Moles";
Computers that interact with spam botnets as if they were zombies, but really arent.
EG, they either run from behind a VERY closely monitored network, or they run a very specially crafted version of the botnet's remote control software:
The purpose is to use the botnet's own rollouts of "enhancements" to be detected, and implemented immediately in the filter (with some sanity checks to avoid this being exploited A-la apple and it's "Leaks" policies.)
This way if the spam botnet rolls out a new template every X hours, the filter will automatically update right along with it, thanks to the Mole.
Since the botnets have to have an automated system for deployment of templates, and many botnets have already been compromised for the "how" of this being done, it makes sense to capture the update process, then compute every possible message that the template can produce, then MD5 hash them, and then check incoming emails to see if they match any of the hashes.
This would stop pretty much all but handcrafted spams, and make the utility of using botnets much lower.
It would also be a good proof of concept for how one can use the knowledge of how a botnet works to make a zombie work for you, without going on the offensive and assuming control over computers you dont own, and the legal quagmire that this entails.
Granted, "major" updates of the botnet to better obfuscate its update process and communications protocols would cause momentary lapses in coverage while the Mole gets analyzed, but this *IS* an arms race, afterall.
Re:Halting problem (Score:3, Insightful)
The Halting problem only exist for theoretical computers with infinite memory, for real computers with finite memory its trivial to solve (wait till a memory state repeats, done).