Catching Spam by Looking at Traffic, Not Content

AngryDad writes "HexView has proposed a method to deal with spam without scanning actual message bodies. The method is based solely on traffic analysis. They call it STP (Source Trust Prediction). A server, like a Real-time Spam Black list, collects SMTP session source and destination addresses from participating Mail Transfer Agents (MTAs) and applies statistics to identify spam-like traffic patterns. A credibility score is returned to the MTA, so it can throttle down or drop possibly unwanted traffic. While I find it questionable, the method might be useful when combined with traditional keyword analysis." What do you think? Is this snake oil, or is there something to this?

sounds good to me

[seanadams.com]

I realize most of us here would ordinarily prefer for our ISPs to just move bits around, but it seems like they are in a pretty good position to curb spam if they were to start look at traffic patterns like this. If some DSL customer suddenly starts opening hundreds of outgoing SMTP connections, that would be a pretty reliable sign that his machine is pwned. Just block or throttle port 25, and send the customer an email telling him to fix his computer, and keep it blocked until he does - or he contacts abuse@ with a legitimate explanation. Not filtering based on the contents of the data should let them maintain plausible deniability and common carrier status.

We can't do this on our personal or company internet connections because we only see individual messages coming from many different IPs, but on the other end of the connection, or even at the backbone level, this strikes me as a pretty solid solution. They could even just tag the packets with the evil bit [faqs.org] and let us decide if we want to filter them or not.

Re:

[Anonymous Coward]

That may be just another tool to circumvent Spam. My primary email spam filtering is Spamd @ openbsd.org/spamd. The service-based spamd is known as Spam Assassin. This is a daemonized version that was ported for Openbsd by the gods. It can be troublesome to configure if you are a first timer. But remain vigilant with google groups and documentation provided by openbsd.org and the man pages within spamd.

Re:sounds good to me

[GreggBz]

The new bread of zombies have wised up to port 25 blocking / throttling and like to funnel everything through the MTA for the domain to which they are connected.

A combination of policyd, postfix, spamassain and ids/bandwidth accounting software has turned it into something manageable, at least where I work. Customers are allowed say, 100 e-mails in a 30 minute time span. If they complain and have a real reason, we can adjust. This also makes finding users with pwned machines a lot easier.

Some of them now (the spam zombies) seem to be moderating their outgoing connections so that it's not so obvious but their volume is still substantial. It just never ends...

Re:sounds good to me

[GreggBz]

It's blue! It's moldy! It's the The night of the living Bread.

Re:

[webdragon]

I'm sure they could do that fairly easily but with how everyone is sue happy their going to have to change the terms of use contracts first to reflect that they can and will do it so they can cover their rear from being sued.

your specific idea sounds damned good to me

[Penguinisto]

"If some DSL customer suddenly starts opening hundreds of outgoing SMTP connections, that would be a pretty reliable sign that his machine is pwned. Just block or throttle port 25, and send the customer an email telling him to fix his computer, and keep it blocked until he does - or he contacts abuse@ with a legitimate explanation."

...locking down port 25 outbound from the client would cure most of the bots out there (though not all - some jackass could set up a couple of open relays to listen on port {so

Re:

[fifedrum]

I work for an email hosting company and our standard with ISP customers is they use IMAP or SMTP auth, worst case, POP before SMTP. It's amazing how much spam is blocked going from an open relay for an ISP to authenticated-only.

spambots are bad, but my biggest problem is with fraudsters, both 419ers and standard credit card fraud types.

These sleazebags cause more trouble than the bots, and it's illegal to kill them. I'm not sure why they cause more trouble, they send out less email than the bots, perhaps th

Re:

[ronanbear]

Or examining patterns might be a less resource hungry way to look for spam. Anything that does get flagged can be assessed in more detail. The more mail you send the more it gets checked to see if it's spam. Most people send valid bulk email (where they send it) to similar lists of people. That's easy to block using blacklists if a spammer tries the same method. But if you're sending 100s of messages to different people and it's a different 100 people and a large number of them are invalid addresses then

Has been done for a long time.

[MadTinfoilHatter]

My (previous) ISP did this several years ago. I found out when I was making a computer for a friend. At the time (this was a few years ago) I didn't yet know just how quickly an unprotected windows-box is owned by viruses. I thought I'd be okay for the time it takes to download a firewall. 20 seconds later I got a popup that I recognized as an infection, so I shut down the machine, and tried to get the firewall / AV-software with my other machine instead - only to be greeted by a screen where my ISP informs me that "By the look of your outgoing traffic, it would seem that your machine has been turned into a spam-bot by a virus, and your account will be automatically unblocked 1 hour after the suspicious traffic stops." This was followed by some generic instructions for virus removal.

Re:sounds good to me

[kripkenstein]

Sounds good? Don't major email providers already do something like this? What else are Google doing when lots of people click on "This is Spam" for a particular email - surely they notice such things? The same should be true of email traffic patterns. Yet, perhaps some minor detail in TFA is the new bit. Obviously any improvement in this area is welcome.

While this will not stop spam, it will be reduced dramatically. The STP value of a spam source will grow proportionally to the number of junk messages sent. The first several thousands emails will get to unlucky recipients when spamming starts, but the rest hundreds of thousands will not.

Actually, webmail can do one better: if a message is marked as spam at some point in time, the system can retroactively remove it from the Inboxes of the 'first few thousand unlucky recipients' (or mark it 'this may be spam', gray it out, etc., at the least). I don't know of anyone doing this, but I wish they would.

Re:

[jgc7]

I agree, but you would have to be careful, because every newsletter/promotion might get marked as spam by a couple of people. The algorithm would need to be pretty sophisticated.

Re:

[djtack]

Not filtering based on the contents of the data should let them maintain plausible deniability and common carrier status.

This is a popular myth, ISP's are not common carriers [wikipedia.org], nor do they want to be.

Re:

[nine-times]

If some DSL customer suddenly starts opening hundreds of outgoing SMTP connections, that would be a pretty reliable sign that his machine is pwned.

But what if the machine isn't "pwned"? Maybe the DSL customer just started a mailing list on his home server about... whatever.

This is part of what makes spam such a problem, that the Internet really needs to be a bit of a free-for-all, or else people will be prevented from doing reasonable things that they technically should be able to do. We could end spam

Re:

[nuzak]

> Maybe the DSL customer just started a mailing list on his home server about... whatever.

Then he asks to get port 25 unblocked. Or he's serious enough about his hobby mailing list to drop 8 quid a month for a dreamhost account (which isn't itself spam-free, but you know at least DH's nets aren't full of zombies). Or he switches to a web feed. There are solutions, but giving random strangers the benefit of the doubt isn't one of them.

If SPF and Domainkeys ever got any traction, then Challenge-Response

Re:

[Mr. Underbridge]

I implemented a somewhat similar system (based on statistics such as TLD, time the email was received, whether it had attachments, how many recipients on the recipient list, etc) and it was decent. It's not as good as a good system that does consider text, but it wasn't as bad as many of the systems on the market.

Re:

[Miseph]

Makes sense, since:

spam = bad
torrents != bad

Anyway, you're comparing apples to socket wrenches... Torrent is a file transfer protocol which can be used legitimately. Spam is a specific abuse of the various e-mail protocols, and by definition cannot have any legitimate use. For your comparison to make sense, it would either have to be between using torrent to distribute virii and spam, or between torrent and SMTP/etc. traffic.

I don't need no steenkin' introduction

[DrSkwid]

Thanks for you misguided rant, quite amusing.

My ISP (and I mean mine, I'm a shareholder) doesn't give a flying fuck what I do with the bandwidth I paid for (and yes, I do pay). The fixed IP of my 2Mb ADSL suits my needs, and many of the needs of other business users we have as customers, extra QoS not required

Get off your high horse and suck it's cock.

This is painfully obvious and hopelessly naive but

[Recovering Hater]

I am going to say it anyway. Why can't people stop responding to spam in the first place? Is it too much to ask? If spammers made absolutely zero dollars for their efforts would they stop? Will underdog be able to escape from the burning rubble in time? Tune in next week to find out in our next exciting adventure!

Re:This is painfully obvious and hopelessly naive

[jimicus]

As soon as you've found a way to get that message through effectively to 100% of the population, do let us know.

Re:This is painfully obvious and hopelessly naive

[Grey Ninja]

We could try mass mailing them. I've had some success with that in the past. =)

Re:This is painfully obvious and hopelessly naive

[Pontus_Pih]

I was going to say... What would happen if we all started replying with the same auto generated mails? How would the spammers tell the difference from legit spam replies?

No!

[Penguinisto]

We have enough problems with idiots who leave all their backscatter-inducing defaults on @ their mail servers - coupled with the common joe-jobs, it would quickly turn the Internet into a gelatinous mass choked with bounces.

Thx in advance,

/P

Re:

[Tim C]

What would happen if we all started replying with the same auto generated mails?

The time it takes me to deal with the 2000+ spams I get each day would increase unmanageably?

Re:

[hackstraw]

I was going to say... What would happen if we all started replying with the same auto generated mails? How would the spammers tell the difference from legit spam replies?

That too has been implemented. Its an invited DDOS attack on the spammer. I love it :)

Regarding the article, this is no big deal. Blacklists, whitelists, and greylists already exist. There is no additional market value with those techniques to eliminate spam.

Re:This is painfully obvious and hopelessly naive

[CohibaVancouver]

>What would happen if we all started replying with the same auto generated mails?

Generally there's nothing to 'reply to' - To order the viagra you've got to go to a web site, or fax in an order - and all the latest 'pump and dump' stock-selling emails don't sell anything at all. They buy some stock, spam out their messages, then dump the stock when the price goes up. Often the company in question knows nothing about it.

Re:

[cmeans]

Except that, maybe, once the spammer constantly sees bogus (usless) email coming from your "legit" email account, it's in their best interest to stop sending you spam...just to reduce the noise at their end. I guess they could black-list your email address, but then they'd still be getting tones of junk mail to filter...serves them right!

I'll never stop

[diskofish]

Where else would I get my Viagra from?

Re:I'll never stop

[El_Muerte_TDS]

You shouldn't. Impotence is nature's signal that you are not fit for reproduction. Your reproduction will only result in more people responding to spam, which is ofcourse a bad thing.

So do the world a favor... please...

Re:This is painfully obvious and hopelessly naive

[stavrosg]

I am going to say it anyway. Why can't people stop responding to spam in the first place? Is it too much to ask?

People will stop buying from spam when they stop forwarding every hoax or urban legend they recieve through their company e-mail to everybody else on their address book.

When someone finds a way to do it, please ping me.

Re:

[Daniel_Staal]

Easy, first you start a nuclear war...

...Then once all the humans are dead, there will be no more spam problem. Except for the kind in cans. Those will last forever.

Re:

[Archangel Michael]

I know of a way, but it is distasteful to too many people.

Death

Re:This is painfully obvious and hopelessly naive

[the dark hero]

That's the problem. this world is full of stupid people. They might not make money off of most people the spam gets to, but if you cast a big enough net you're bound to catch something(including some dolphins). Millions of pennies still add up to thousands of dollars.

Re:

[jdunlevy]

That's the problem. this world is full of stupid people.

The problem isn't so much stupid people as it is naive people. One big reason there are suckers ready to be taken in by spam is that every day, there are still a great many people experiencing spam for the first time. (The internet was "growing at an annualized rate of 18%" as of December 2005 [useit.com] according to one source just found in a quick Google search [google.com].) There are still a lot of people out there who've never read e-mail; they haven't yet learned about

Re:This is painfully obvious and hopelessly naive

[KKlaus]

Complaining that people are frequently bad decision makers is usually not worthwhile. Much better to recognize the truth that they are, and then work to try and take the decisions out of their hands.

Its similar to a pretty interesting conceptual innovation in medicine, when people realized that even excellent doctors will at some point make grossly negligent mistakes simply due to the shear amount of work they do (i.e. operating on people with paralytics but not analgesics). So the innovation is to make them make fewer decisions - machines that check settings before running, labels that a four year old could understand, arrows and other reminders liberally applied.

So similarly here, yes it's annoying that people continue to "fund" spammers, but education is not the answer. Because, unfortunately, the spammer's target market of "everyone in the world" will always contain enough people to make their trade profitable if all we rely on is good decision making on the parts of spam recipients. So the solution has to be technical or legal. And in that regard, another small step for man here.

Even if no one ever responds, it won't stop

[MarkusQ]

Even if no one ever responds, it won't stop as long as the people paying to have it sent think it works. It's like burning candles to St. Balderdash for scam marketing morons. As long as there is a steady supply of rubes who think that sending spam is their road to riches, and are willing to pay some brighter but no more honest spam lord to send their dreck to a bazillion hapless victims for them, spam will contine to flow.

This is true even if no one ever responds to, falls for, or even opens a spam message ever again.

--MarkusQ

Re:This is painfully obvious and hopelessly naive

[cdrguru]

The money in spam isn't from people buying stuff - it is from the silly advertiser thinking they can send their ads to millions of people for $1000. They do this and get a report back that says only 0.8% of the people opened the email.

The spam-sending organization then shows them that they need to revise their message with a better subject line so more people opened the email. Another $1000 and more spam is sent, this time 0.7% of the people open the email.

Continue this until the advertiser runs out of money. If you have enough contracts for sending spam it matters not a whit if anyone buys the stuff at all. It is only important that people pay for it to be sent.

Re:

[rel4x]

You sir, have no idea what you're talking about. They get paid by the sale for products, by the lead for mortgages, or a percantage for stocks. Go to bulkerforum.biz and look around.

Re:

[account_deleted]

Comment removed based on user account deletion

Spam is desensitizing people for other reasons

[scottsk]

No one cares if anyone responds to spam or not. Spammers are the lowest of the low on the food chain. What people who peddle immorality want is a steady stream of junk mail coming into your inbox hour after hour, day after day, and year after year. Eventually, it's going to weaken you - being exposed to all the viagra, penny stock, erase your credit without paying, etc etc etc - pretty soon you'll think this stuff is normal, and when you have to make a moral decision someplace else (someplace more profitabl

Re:This is painfully obvious and hopelessly naive

[bcrowell]

Why can't people stop responding to spam in the first place? [...] If spammers made absolutely zero dollars for their efforts would they stop?
First off, if people stopped responding to spam, it wouldn't have any effect on phishing spam, since phishing is based on tricking the user into thinking it's legitimate mail rather than spam. Also, once you have control over an army of zombies, the incremental cost of sending one spam is zero. Even if the spammer thinks he's unlikely to make any money at all by s

unlikely indicators

[Speare]

I think the question raises an interesting point: spams *behave* differently on the network than most legitimate emails. It may not be a perfect discriminator, but it sure might be a corroborative scoring aid. This reminded me of the controversy when Slashdot started using text compressibility as a metric for "lameness." I was a disbeliever, and still have my reservations about it, but as a part of the overall toolbox for filtering lameness, the technique seems to have value.

And yet likely...

[paladinwannabe2]

Bayesian filters sometimes find weird words to do filtering on. Obviously there is 'Viagra' and 'Manhood' but there are also words like 'Republic' that have very high correlations with phishing spam- because any email that from the 'Democratic People's Republic of $Country' is likely to be as bogus as the countries name. If a country needs to add 'Democratic' or 'Republic' to its name, you know something's wrong.

In a similar way, any easily compressed text (like boing
boing
boing
boing
boing
boing
) is most

Re:And yet likely...

[Zocalo]

because any email that from the 'Democratic People's Republic of $Country' is likely to be as bogus as the countries name. If a country needs to add 'Democratic' or 'Republic' to its name, you know something's wrong

• Central African Republic
• Czech Republic
• Democratic Republic of the Congo
• Dominican Republic
• Former Yugoslav Republic of Macedonia

And that's just the common names and not the official ones like "Republic of Ireland". Given that this is precisely the kind of verbose terminology that you would find in a genuine official email from a government body in such a country, I don't think that's going be suitable for anything other than a minor nudge towards spamminess.

Places you don't want to be

[paladinwannabe2]

Democratic Republic of the Congo- Welcome to the land of warlords, genocide, and more genocide.
Central African Republic- Less than half the genocide of its neighbor in the congo.
Dominican and Czech Republics, and Macedonia- actual democracies.

So two of your five examples help prove my point- and when you start stacking adjectives together- like 'People's Democratic Republic of Korea' you know you've got one of the worst places to live on Earth.

Also, why on earth would you get an 'official governm

The best and worst places to be

[paladinwannabe2]

Looking at Wikipeida [wikipedia.org] we find that out of the 14 freest places to live, 'Republic' is part of the title on 4 of them. Looking at the 8 worst places to live, 'Republic', 'Democratic', and 'People's' are part of the title of 6 of them, and they appear a total of 10 times in the name of 8 countries. So it seems that my point has some factual backing, and there's a strong correlation between having 'Republic', 'Democratic', and 'People's' in a countries name and it being none of the above.

greylisting works

[grub]

OpenBSD's greylisting [openbsd.org] in spamd works wonders.

Re:

[ivan256]

Greylisting is great and all, but I'm left wondering what OpenBSD has to do with it... Can you name a single operating system that can run an MTA that can't do greylisting?

I didn't think so.

Re:

[ivan256]

I read the docs, and I still don't see how it's any different than the greylisting deamons (written in a variety of languages, and some probably sharing code with this one) that run on Linux or Windows.

The problem with this

[wiredog]

Mailing lists. How does it not tag a server that sends out mail to a list as a spammer?

Re:

[crossmr]

I'm betting the average bit of spam set out doesn't have anything on 99.9999999999% of mailing lists out there. Anyone with a mailing list which would approach the levels of spam one would expect from a compromised computer can speak with their ISP and give them the details to get an exemption.

Re:

[gmuslera]

Mailing list are a problem, but is something that could have a sustained ratio of sent mails, and maybe more important, a sustained ratio of received mails, if you count mails coming from and going to that host can lower the score as the mix of both traffics can hint a mailing list server there.

But what about announcement lists? You know, you sign up in a site, company, etc, and want to receive a mail when something big changes, a new product, whatever. That are usually unidirectional, targets a lot of peop

Re:

[kevin_conaway]

Indeed.

To expand on your idea, think about the small business owner who sends a monthly newsletter to a few hundred of his customers from his home pc.

Re:

[Tom]

Solveable problem. The traffic pattern of a mailing list is different from the traffic pattern of spam. Just for starters: Very few mailing lists have 50 million different subscribers.

The devil in this doesn't lie in the concept, the concept is sound. Implementation will be tricky.

Greylisting

[Daemonstar]

This is similar to greylisting [greylisting.org] that has been around for a bit.

Greylisting is a simple method of defending electronic mail users against e-mail spam. In short, a mail transfer agent which uses greylisting will "temporarily reject" any email from a sender it does not recognize. If the mail is legitimate, the originating server will try again to send it later, at which time the destination will accept it. If the mail is from a spammer, it will probably not be retried, however, even spam sources which re-tra

Re:

[Gary W. Longsine]

No, greylisting can be applied to RFC-compliant MTAs as well.

Re:

[Daemonstar]

*sigh* I'll bend my rule of not responding to AC's this time, because you obviously do not understand the reasoning behind greylisting; it has nothing to do with "buggy MTA's", but with intentional misconfiguration (taken from the TAMU website):

According to the internet specification, when a mail server receives a "400-level" error, it must queue the e-mail message and try later to deliver it. For legitimate e-mail, this process is standard and mandatory. Properly configured mail servers will redeliver t

request

[illuminatedwax]

please put obligatory Standard Spam Form joke below here please

we've got to keep this place organized

My version

[Valdrax]

Your post advocates a

(X) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the mone

Yes and no - but a suggestion...

[Penguinisto]

I like the idea of gathering and using statistics on traffic patterns, but what they're looking for in many cases can be too easily defeated (e.g. "Junk messages are small"... now we get to watch MTA's spend more time trying to sort spam messages packed to the gills w/ random ASCII, necessitating a look through the message body all over again).

OTOH, As part of a larger array of spam-fighting tools, okay - there's bits in there I actually like and which can be used as part of other solutions, if not used in the way suggested. As someone who runs a couple of MTA's on top of everything else I do around here, I always like to find new and interesting ways of stopping spam.

N.B., all that I ask is this: Please make it useful w/o sucking down resources or requisitioning another server. I detest external RBL's - please don't suggest anything that may have an overly-subjective and/or an overly-dependant basis like that. If it isn't RFC-compliant (yes, Verizon, I'm talking to YOU when I say that!), I won't go near it.

Satisfy those, and yes, I'm interested, as would lots of other SMTP-monkeys out here.

/P

Obligatory

[teslar]

Your post advocates a

(x) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(x) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
(x) It is defenseless against brute force attacks
(x) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
(x) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
(x) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(x) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(x) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

Re:

[hlh_nospam]

"( ) Sending email should be free"

Interesting canned response letter; although I've seen similar posted on the usenet email abuse lists.

However, the assertion that sending email should be free is questionable. First of all, email is NOT free anyway -- it ALWAYS arrives postage-due, i.e., the recipient pays the majority of all cost either directly or indirectly for all email. That is the ONLY reason that spam exists in the first place. The marginal cost of sending spam is very nearly zero, so even a fou

Re:

[Tom]

(x) You didn't understand before you hit "reply".

In theory, this would work and mailing lists would not be a problem. If the implementation sucks, though...

Same for "why trust your servers"? - you don't have to. If the method works, there will be multiple services offering similar products, and you can choose which one to trust.

Re:

[Peter La Casse]

This form is for ideas that have been thought of before and have been discredited, but I'm not convinced yet that this idea wouldn't work. Here are the biggest objections you raised:

(x) Mailing lists and other legitimate email uses would be affected

How? The method specifically mentions whitelisting, and only mailing lists or other "legitimate uses" (can't think of any myself) that involve thousands of recipients would be noticed by the proposed algorithm.

(x) It is defenseless against brute force attack

OPPOTUNITY. == DISCRETION REQUIRED ==

[Anonymous Coward]

SIR,

OUR TECHNOLOGY DEPARTMENT HAS COME UP WITH A GREAT OPPUTUNITY TO STOP ALL YOUR SPAM. THIS TECHNOLOGY IS CALLED source Trust Prediction (STP). IT WORKS BASED ON identifying patterns and trends in real time AND IN THIS WAY PREVENT SPAM. HOWEVER TO MAKE PROFIT FROM THIS NEW TECHNOLOGYY WE NEED TO DO A PATENT APPLICATION. YOUR NAME CAME FORWARD AS AN EXCELLENT INVESTOR FOR THIS. WITH THE CURRENT RISE OF SPAM THIS TECH WILL BE REQUIRED QUICKLY BY A LOT OF PEOPLE.

I am only contacting you as a foreigner, I will use my influence to
effect legal approvals and onward transfer into your account At the
conclusion of this business, you will be given 50% of the total
PROFITS, 50% will be for me and my family AFTER DEDUCTION OF THE PATENT COSTS
. I await to hear from you.

Yours truly,

Mr.Barry Leoard.

FNB OF SOUTH AFRICA
THIS
IS MY PRIVATE EMAIL ADDRESS, YOU CAN SEND YOUR REPLY HERE:-
[e-mail]

Re:OPPOTUNITY. == DISCRETION REQUIRED ==

[Archangel Michael]

Source Trust Detection (STD)

There, fixed your spelling ...

Dead on the money.

[WindBourne]

3 years ago, I was working developing some software for sale to the feds and commercial world. For the commercial world, I proposed the same idea. The only way to stop spam is have cooperating servers. More importantly, they need to have a lot of servers where fake addresses can be sent to. Load these into outlook and let the spammers harvest them. Now, you have a decent service that can be offered for free or sold.

Problem

[Billosaur]

What about legitimate mass marketers. The company I work for contracts with advertisers to send out bulk mailings to our opted-in users. Now, we don't spit out emails by the millions, but we certainly do send out large chunks of emails from a common source. Is this kind of thing going to interfere with legitimate mailings to opted-in customers?

Re:

[radja]

probably, but since the use of hexview's product is also opt-in, that's not a problem.

Acceptable loss.

[Kadin2048]

I'm not sure I care. Those "legitimate" "opt-in" lists tend to get reported by users as spam eventually anyway. Meaning even if they did originally 'opt in,' it's basically nothing but a nuisance eventually. (Usually people opt in, allegedly or actually, and then can't figure out how to opt out, or don't want to spend the effort to do so.) The effect is the same as spam, even if the intent isn't.

I would consider the elimination of commercial mass email a very small price to pay for the elimination of spam.

Re:

[ynohoo]

What about legitimate mass marketers.

Dont be silly - they are all bastards.

Just because your business model is (currently) legal, does not make it defensible outside of a court of law. Around here, you are still vermin.

Ditto. Big mass marketers will benefit.

[giafly]

This system relies on whitelisting to handle companies like yours. Hence you'll need to spend more on ISP relations. Big bulk-mailers can more easily afford this so they will gain at the expense of competitors.

BTW this system won't work because the author's assumptions are wrong. Botnet senders can easily afford all the following suggested countermeasures. I expect they'll carry on as normal. Then, if blocklisted, switch over to DDOSing the the STP servers until the blocklisting is removed again.

How sp

Re:

[Animats]

The company I work for contracts with advertisers to send out bulk mailings to our opted-in users.

And did they opt in by specifically requesting your mail, or implicitly as part of some other transaction? If it's the latter, you're a spammer. Die.

If people really want your content, offer an RSS feed. If nobody subscribes to your feed, they didn't want your content.

Re:

[Billosaur]

We have a new world's record in the Jump to Conclusions!

Anybody got a "Troll" mod point to spare?

Its not snake oil, but...

[popo]

... and its not disimilar from greylisting from what I can tell, but I don't think its going to be
effective in the long term. Getting around this type of filter (or delay) seems relatively simple
compared to the task of defeating the bayesian filters over the past couple years.

The lynchpin of greylisting is that legitimate mail will "try again" after being returned by the
server, while spam will not. The conclusion (which we hope is true) is that any mail that is
not re-sent was in fact spam. Never mind the danger that the assumption could be false and
legitimate mail gets lost -- how long will it be before spammers simply "re try" their spam --
or worse -- just send everything twice?

As with any attempt to modify behavior electronically -- behavior usually wins.
 

Re:

[stu42j]

just send everything twice?

Many already do but not enough to stop greylisting from being ineffective. Even if all you do is delay the message you still increase the chance that the message can be blocked by other means.

Re:

[raddan]

The nice thing about greylisting is that if spammers learn the "trick" of becoming RFC-compliant and thus retry their connections, the cost of their operation goes up. The cost may be small over several thousand messages per day, which is easily handled by a normal, behaving MTA, but for a spammer whose cost calculations depend on spewing out millions of emails per day, it may be a dealbreaker. Combine that with tarpitting and some way of feeding Bayes scores back to the tarpit/greylist (ala relaydb), and

Solution

[VincenzoRomano]

The SMTP protocol is showing all its age and weakness. It has not been designed to cope with today's use.
First of all it lacks authentication and authorisation mechanisms. The various anti-spam, white/black/grey listing look more like workarounds than solution.
Then you'd like to really know whether your message has been delivered or not and other nice details about the messages.
My personal feeling is that it's time now for a new messaging protocol.
SMTP is dead, long life to SMTP!

Re:

[John Bayko]

I see two possible replacements for email.

One is based on RSS (or similar, like Atom). Right now, RSS is used for what amounts to "mailing lists", by notifying the recipients there's something new, and they can pick up their copy - though it works by polling, no actual notification is sent.

One extremely important advantage of this is that you know exactly where the material is from.

I'm hoping that future versions will allow an RSS feed to be customised per user, which would basically amount to sender-

What about SenderBase?

[NtroP]

This isn't a new concept. Our mail gateways already participate in something like this with IronPort's [ironport.com] SenderBase [senderbase.org] reputation filtering. 90%+ of our incoming mail traffic is dropped based on poor reputations scores without looking at anything more than the sender's address. So far, we've never had a false-positive that we know of, and only once, after many customers were made a part of a bot-net and started spamming, did SenderBase throttle traffic to one of the local ISP's. A quick call to their mail admins pointing out the problem and they were able to block those customers from sending mail until they were cleaned up and the reputation score climbed back up again.

It has really taken the load off our mail servers by blocking millions of connections. The rest, we run through SpamAssassin and everything works great!

I am curious...

[localman]

Are any of you people still living with spam? Do we really need another solution? I've found that a personally managed baysean filter is plenty good enough. I'm down from 700+ per day to 2-3 per day. I still dislike the fact that spam is out there, but I haven't actually had to deal with it in years. Has this not worked for other people? I mean, I do have to continue to feed the filter, but it's very little work. Nothing wrong with new ideas in the battle, but I thought that for anyone who cared it was already won.

Cheers.

Re:

[Intron]

Why do you think it "works" when your server has to scan and reject 700+ emails/day?

Personally, I think that email should have a button that you can press if you don't like the email that adds a 0.1V charge to the sending PC. If one person presses it, the charge won't be noticeable, but if 1,000,000 press it...

Re:

[Phroggy]

Are any of you people still living with spam? Do we really need another solution?

Anyone who's a mail server administrator is living with more spam than you could probably imagine. During a four-week period, across two of the (very small) servers I manage, 38,728 connections were refused because of RBLs. Of the messages that were accepted, 8,102 were assigned a SpamAssassin score above 15 and sent to a system-wide quarantine folder that users never see. Another 13,619 messages were assigned a score between 5 and 15, and sent to a user-accessible quarantine folder for review. I use R

this and other effective weapons

[fifedrum]

yes, traffic shaping is effective in determining the nature of connections

I work for a small email company we process millions of emails an hour inbound, but only a few million a day outbound.

Our most effective filters are:

connect/HELO restrictions: you can only get email into the environment if your IP address resolves to a FQDN.

HELO restrictions: if you connect using X different HELO strings, you are blacklisted. Spambots often randomize the helos, this blocks those.

Spamassassin at the client side, filtering email into various folders based on the score.

antivirus server that filters the few viruses that make it in, and phishing is filtered too.

The problem? All this doesn't catch enough of the spam. We still have loads of CPU dedicated to filtering spam, but something like this technique at the border will help, and I'll predict (based on experience watching the traffic and spam filtering graphs) that we could cut spam another 30% just by watching the curves and tightening the restrictions during those peaks.

Re:

[TheRaven64]

connect/HELO restrictions: you can only get email into the environment if your IP address resolves to a FQDN.

Does this actually do anything? I just checked and my (residential) cable modem IP resolves forwards and backwards. Since most spam is sent by zombies on similar connections, won't they all resolve?

Re:

[macdaddy]

I'd suggest you look into Canit-Pro from Roaring Penguin [roaringpenguin.com]. It's from the author of MIMEDefang. Actually it's MD's commercial big brother. They make an appliance but I still run the app locally on Fedora boxes. They give you the full source code. It's extremely extensible. It makes Barracuda Networks' products look like child's play. Basically it will take the knowledge you already have and give you a platform to extend and build upon it. Canit-Pro is slick. The auto-tempfail by recipient and IP is g

Botnets?

[Jabrwock]

This wouldn't really work against botnets, would it? Because of the fact that they are distributed, you wouldn't really have a source trust issue... Not one that would trip any warning flags, anyway.

I can see it though, be a handy tool to aid against regular spammers, perhaps in analysing traffic to assist in maintaining SBLs...

All they have to do is slow down.

[edunbar93]

And this applies to botnets... how, exactly? If you can infect just a million computers with your spam bot, then you can send a million messages an hour by sending *one* message an hour per host! With a billion plus hosts on the net, you need to infect less than 0.1% of them to make that happen. The number of vulnerable computers at any given moment in time is easily more than 20%.

But hey, for every complex problem...

Whitelists

[Doc Ruby]

Everyone should prioritize their incoming email by who in their address book sent it, or it's unsolicited, probably commercial, email, "UCE", aka SPAM.

Won't really work

[jerseyjim]

I use a popular, public email service. My emails have been identified as spam at times. The reality is the everyone from the service uses the same IP email address. All it takes is one person from that service to send spam and all those using the service get flag...so volume along isn't a good indicator.

Would work if

[crossmr]

We made everyone who had a mailing list which contacts more than 100 people "register" with their ISP. They don't have to disclose the recipients or the nature of the list, simply a "I will be sending out a mailing list to x amount of users everyday in addition to my personal usage. Any customer who spits out more than some reasonable number of e-mails (who knows, maybe 200 per day is sufficient for most home users even on the upper ends of e-mail usage) will find their ability to use the outbound server re

The only real solution to spam.

[arthurpaliden]

Is to have the ISP charge for email usage in the same way as you get charged for your cell phone usage.

Won't work.

[Animats]

Won't work. It just means the owners of zombie PCs get big bills.

Re:

[metamatic]

It just means the owners of zombie PCs get big bills.

That's not a bug, it's a feature.

Right now, the costs caused by Windows insecurity are passed on to me even though I don't run Windows. Passing those costs on to the people causing them would be much fairer.

Follow the money and stop the source

[spectro]

1. Company offering product or service hires spammer 2. Spammer creates botnet by installing spyware in unsecured computers 3. Botnet sends spam Pretty much any solution so far involves stopping step 3, the delivery when the real problem relies in step 1, we need to find ways to stop step 1 from happening. Lets make hiring spammers a criminal offence, the same way "murder for hire" is. You can catch them by just having undercover officers order the product/service. I say let's make hiring spammer to adver

It already exists

[m0i]

Commtouch does this already:
http://www.commtouch.com/Site/Enterprise/e_technol ogy.asp [commtouch.com]
few false positives, >97% catch rate, 0.3s per message scan (on my system from live data, not marketing specs).

Great idea, just several years late ;)

[Thorizdin]

For everyone screaming that this isn't feasible, will kill mailing lists, and other wise render effective communication via SMTP impossible you might want to consider that about a quarter of global email volume is already flowing through a system very much like what the OP describes.

Ironport (recently purchased by Cisco for $830 million US) has been doing this kind of service for large providers for several years.
Their statistics site is publicly viewable, but using their stats requires a subscription fee.
http://www.senderbase.org/ [senderbase.org]
Its interesting to look at how well or poorly the MTA's you use are scored. All of the stats are gathered by the systems they sell to ISP's and enterprise customers. These boxes perform the spam filtering for that organization's customers and provide statistical data back to senderbase.org, which allows all Ironport customers to "know" about problems for all other Ironport customers.

The link to their PDF on their metric's is here:
http://ironport.com/pdf/ironport_wp_reputation_bas ed_control.pdf [ironport.com]

We evaluated their system last year as a possible replacement for a third party spam/virus scanning provider and may end up purchasing their equipment once everything with the Cisco purchase shakes out. Their solution, while not perfect, behaves far better than some of the things that large service providers *coughAOLcough* have tried and are (or were when we tested) comparable to most of the content based scanning systems in terms of spam filtering with a lower rate of false positives.

This technology has been around for years

[j0yc3]

IP Reputation filters are not a new idea by any stretch of the imagination.
CipherTrust TrustedSource [trustedsource.org]

Want to get rid of Spam?

[jstmehr4u3]

Get rid of HTML emails.. Spam isn't as cool when it doesn't have a bunch of fake links, pretty pictures, etc. You think the internet would cease to exist if we went back to text only?

Send a URL in your text-only email if you want to check the email out in HTML...

Just a thought

The problem is that it is still filtering

[cyberscan]

One can come up with all kinds of trick to filter spam, however, the problem still remains. Spam will continue as long as it is profitable. There are too many "Puppies in a barrel" for a spammer to choose. After many, many years of prodding, many people have finally gotten antivirus program, yet they neglect to download or purchase virus database updates. Many people spend time and effort to ensure that their computers are malware
free, yet their router retains the default username and admin password. Spammers have programs that allow people to try to log in to these routers and use their embedded telnet commands to send spam without the knowledge of the computer owner or any program residing on their computer. The point is that the Internet can be compared more to "swiss cheese" rather than the "series of tubes" that the politicians use. There are many, many points of attack for spammers to use.

Filtering spam is much akin to a person who holds hands in front of his or her face while a bully is pummeling him or her. The person is likely to fend off blows from the bully, but some of the blows will get through. Once a spam is sent, even if properly filtered, the damage has already been done. Until very recently, all I had in my area was dialup. My program successfully filtered about 99% of the spam received, however I still had to wait about 30 minutes before I was able to view my legitimate mail. I lost 30 minutes of time that I could have been working on a client's problem, while the spammer lost nothing. I also lost a client because a program that I previous used labeled his email he sent me as spam. Again the spammer who spammed me lost nothing. Spammers are like bullies, they will not stop until people HIT BACK!

It is only when spammers have to deal with the large amount of bandwith used, the processing power to handle complaints, and the loss of sales that result from efforts to filter complaints will spam be much less profitable. The idea is to punch back and deter the bully. Sending complaints to the spammers' websites get them at their weak point - the place where they make contact with potential buyers. Several program have attempted to hit back, and 2 of them were very successful in doing so. However, like spammers, these programs had a weak point, and that point was the fact that they needed a central server in order to instruct each individual program. Now things are different. There are several projects currently underway to trade complaint instruction files via peer to peer networks. What this means is that there is no central server which spammers can attack in order to silence complaints to their websites. One such project is called SpammerSkewer, and it is an open source GPL program that is in alpha. The program can be found at http://spammerskewer.sourceforge.net/ [sourceforge.net] .

It is also important to note that these new programs are not distributed denial of service programs. As for SpammerSkewer, it only receives instructions on how to complain. It does not initiate complaints. Only a user can initiate a complaint by either bringing up the complaint interface or by dragging an email into SpammerSkewer's spam directory. It is the Spammer who determines how many complaints are submitted to their websites. SpammerSkewer's author even provides a way for spammers to "opt out" from receiving complaints if they insert a header clearly labeling their email as spam. Another way they can opt out is by not sending spam in the first place. In a distributed denial of service attack, a person other than the one who controls a victim's website is the one that controls how many visits a site receives. With SpammerSkewer, it is the Spammer who sends out the spam that determines how many visits a site advertised via spam gets. The only sites that are put in SPammerSkewer's instruction files are those well known to be advertised via spam. Instruction files are also cryptographically signed in order to prevent tampering. I

Re:

[Southpaw018]

Damn, got beat to it. Sorry for the redundant spammy post!