Slashdot Log In
State Dept E-mail Crash After "Reply-All" Storm
Posted by
timothy
on Sat Jan 10, 2009 09:40 PM
from the forward-this-story-to-all-your-friends-in-triplicate dept.
from the forward-this-story-to-all-your-friends-in-triplicate dept.
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Bedlam... (Score:5, Interesting)
http://msexchangeteam.com/archive/2004/04/08/109626.aspx [msexchangeteam.com]
Again...
-Ghostis
Re:Bedlam... (Score:4, Insightful)
Parent
Re:Bedlam... (Score:4, Insightful)
Change that to 'that warns if an exceptionally large number of messages are being sent and smack the user over the head with a LART if they don't click cancel' and i'll agree with you.
A large company should have an internal mailing list and/or intranet system that individual users can post messages to. Letting individual users send email to more than a few thousand users in one hit is madness. Especially if they are anything like our customers where they think it is a good idea to send a 10MB attachment to 500 users...
Parent
Re:Bedlam... (Score:5, Funny)
I remember 10 or so years ago a disgruntled employee managed to send a heartfelt "Fuck You" to the entire 27,000+ employees as he was being given the heave ho.
That one tied up the network for some period of time. I always wonder who the bright star was how had composed the distribution list for the entire company directory.
Parent
Re:Bedlam... (Score:5, Funny)
That one tied up the network for some period of time.
Thats why I always use qmail for my Fuck You messages.
Parent
Re:Bedlam... (Score:4, Insightful)
That said, I think there actually should be a distribution list for the entire company - it can be useful for some stuff.
However the actual name should be hard to guess, and secret.
Then you set up the "everyone" list for people to send to which actually goes to a moderator.
If the moderator thinks the email should go out, it is sent out via the "secret-real-everyone-list", otherwise it isn't.
If the email indicates that the sender has significant lack of discretion or intelligence, the moderator may wish to pass it to the Bosses concerned so that they can take necessary measures.
In one of the places I worked for "everyone" actually went to the Big Boss(es), and I think it worked reasonably well.
Parent
Re:Bedlam... (Score:5, Insightful)
It doesn't need to be secret if there are controls on who can send messages to the list. It is so trivial to do for any competent email admin no matter what software they use.
Parent
Re:Bedlam... (Score:5, Informative)
This is a configuration error, not a newsworthy event.
For sendmail [sendmail.org], it would be a configuration directive in their sendmail.mc (or whatever theirs is:
confMAX_RCPTS_PER_MESSAGE("100") ... or a modified line in sendmail.cf:
O MaxRecipientsPerMessage=100
In MSExchange [microsoft.com] it would be a registry change
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\Max Recipients on Submit
DWORD Value 100
Parent
Re:Bedlam... (Score:5, Interesting)
Parent
Re:Bedlam... (Score:4, Funny)
You have tried to send an email. Do you wish to allow or deny?
Parent
Re:Bedlam... (Score:5, Insightful)
What the fuck? Don't do that. Reply all has a valid use case. In fact it's the way everyone at my company most commonly replies to email messages. Why? Because the CC list is there for a reason - those are people who are supposed to know what's going on in that email thread.
How about just educating your users on checking who they're sending an email to, every single time they send one.
Parent
Re:Bedlam... (Score:4, Insightful)
How about conducting discussions on discussion boards rather than email?
Great Idea!
"Dear Vendor,
Since several people will need to be in on this conversation please visit our website at:
www.webboards.com and create a user account and password.
When you've registered you will need to visit the topic "Vendor discussions/XYZ Project" and pay special attention to the topic "How do we implement X without breaking Y or can we live without Y?".
We would really appreciate your input.
Signed,
Employee who's about to be fired."
Brilliant idea! A webboard! Now... I just need to send a mass email to everyone involved to let them know there is a new topic open and how to get to it. That sounds far more intuitive than just sending an email to 5 people!
Parent
Re:Bedlam... (Score:4, Insightful)
If you were my admin and you did this to me I would hunt you down and kick you in the shins.
In the year 2009 we now often work in teams. We often communicate as teams. We often 'think' as... you guessed it... teams.
But by all means I'm sure whatever company you're working at people only talk to one person at a time. You have no group discussions and the only interaction that occurs between employees is by the watercooler and in meetings.
At our company however more than one recipient is the norm. Especially when you want to keep a project manager 'in the loop' of a conversation with a vendor. In fact our most common occurance is to have to say "oops - sorry looks like I dropped so and so from this conversation". Not "Ooops, I accidentally killed our mail server while talking to 4 people."
So go ahead and remove Reply-All in the classic System Administrator "I don't care how my users want to use my network. It's mine and I'll do as I please." dick move. Because that's what it is. It's a Dick move and expect irate emails from users who suddenly find their email doesn't work very well anymore.
Parent
Re:Bedlam... (Score:5, Informative)
Parent
Re:Bedlam... (Score:5, Funny)
Haha, sex change team.
Parent
Re:Bedlam... (Score:5, Funny)
What's the M Sex Change Team? People who still haven't gotten over Judi Dench playing M? Come on, folks, M is a title, not a person; it's not a sex change!
Parent
Re:Bedlam... (Score:5, Insightful)
And then they threaten disciplinary action if someone uses them the wrong way. Wouldn't it be so much easier to just lock them down? It's what most companies do.
Parent
Re:Bedlam... (Score:5, Informative)
dude, transsexualism has nothing to do with being gay. most homosexuals aren't transsexuals. they're just males/females who are attracted to their own sex.
the city you are looking for is Trinidad, Colorado [wikipedia.org], which has been dubbed the Sex Change Capital of the U.S. [cbsnews.com]
Parent
sigh (Score:5, Interesting)
What an irony that they decided to mass mail when they've warned their employees not to do so. What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response... However, judging by the nature of their work [it is the state department after all] I don't believe it unreasonable that there could be events in their future requiring such mass mailings again and having the whole system crash under the load would be no doubt very bad in emergencies.
Thsi is a test... (Score:5, Funny)
Parent
Re:Thsi is a test... (Score:5, Insightful)
The problem is the morons who send email with "everybody.all.everwhere" (or whatever) in the To: or CC: list. If they were smart enough to put them in the BCC: field, it would be impossible for people to clog up the system with Reply All. Alas.
Parent
Re:sigh (Score:5, Insightful)
Parent
Re:sigh (Score:5, Insightful)
What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response...
No. What they should have done was installed a mailing list manager, created a read-only list called "employees", and posted to it. Voila - n-thousand workers get announcements with no ability to reply to the whole list. Problem solved.
Parent
Incorrect Headline (Score:5, Insightful)
Whoever wrote the headline for this summary needs to have their slashdot editor privileges revoked.
TFA states "an e-mail storm nearly knocked out one of the State Department's main electronic communications systems", and "a major interruption in departmental e-mail". The problem is clearly spelled out as "e-mail queues, especially between posts, back up while processing the extra volume of e-mails".
This is simply the queues backing up, not the servers crashing. Nowhere does TFA state anything to suggest that there was a "State Dept E-mail Crash", which the summary's headline boasts. The proper headline should read "Large E-mail Queues at State Dept After Reply-All Storm".
No, I'm not new here. That's why I'm fed up with the sensationalist "journalism" that is getting worse and worse here.
President-Elect Obama Assassinated! (Score:5, Insightful)
This just in, President-Elect Obama Assassinated! Oh, don't take it so literally. I was just trying to capture your attention in a short amount of time. Obama wasn't killed, silly. There was just some CHARACTER assassination against him on a late night talk show.
Parent
Mail to 'everyone', [click] (Score:5, Funny)
I remember my first year of college when I wanted to send Xmas greetings to 'everyone'. I remember, the IT director of the college running from computer lab to computer lab looking for student number xxyz.
Fun times.
Re:Mail to 'everyone', [click] (Score:5, Funny)
I was supervising the call center at the time, and I saw hundreds of hands tentatively raising. The message probably went to two thousand people.
Parent
Two questions: (Score:5, Interesting)
a) Maintaining large list by copying all recipients into the hrader is a fucked up idea at best (because there is no way this list will be kept updated), and a informaiton leak at worst (because somebody eralier on a non-updated list may get information which he should not get - e.g. former employees). Why do governmental institutions still us it?
b) Why in the world do modern e-mail clients still allow reply all to hundreds of recipients without an additional safety question. I would expect my program would warn me before sending an emails to thousand people.
Re:Two questions: (Score:5, Insightful)
Are you sure you want to do this stupid thing ?
pops up, people universally click "OK" without a second thought.
People have just been blasted by too many of these warnings to take any proper note any more.
Parent
Mail list software anyone? (Score:5, Informative)
Dear state department
I'm sorry to hear about your recent trouble
There is a brand new invention on the internet which have the ability to ease the strain on your mailservers. it is called maillist managers. one is called mailman and can be found at: http://www.gnu.org/software/mailman [gnu.org]
There are several others, some free, and some non free, but they exist for most server platforms. If you don't have the expertice in house to set it up corrctly, you can get any number of consultancy companies to help you out.
Yours faithfull
Almost anonymous coward
Reply All isn't the problem (Score:5, Insightful)
The problem is the message replied to having - RTFA - several thousand addresses in the To: and CC: fields. This is what BCC is for . Allowing people to put several thousand addresses in to the headers will eventually result in a mail storm, whether someone hits Reply To All or not. The first time someone opens a virus laden attachment that goes through their (archived by law, this being a federal agency) emails, it will send itself out to thousands of equally clueless people. One of them will run the attachment, which will send another copy to several thousand people. And so on. This happened where I work once, by people who should have known better. Before it was done, I was getting two hundreds copies of the virus per day.
Whoever sent out the message replied to should be fired and criminally prosecuted for deliberately sabotaging the State Department's email system. But since the article doesn't mention this at all, I'm assuming it was some dumbass boss somewhree who is immune to any form of disclipline for anything, up to and including murder.
Re:Reply All isn't the problem (Score:5, Informative)
Having been a witness to the incident in question, here's what happened:
1) Around December 30th a blank e-mail (with receipt request) went out to almost all users. Apparently it was from a single user with some malware etc. (we didn't get any further details).
2) The next day, the same blank message was sent out again (from the same user).
3) As people came back from vacations, we got a few "Please remove me from this list", and "What is this message" send as reply-all.
4) Then, followed with a bunch of "Me Too".
5) Then, a bunch of "Please, don't reply all" (sent, of course, reply-all).
6) Followed by a bunch of "remove me from this list".
and so on, and so forth, with no end in sight...
The initial message didn't have any virus or other "payload"; just a blank message that caused a bunch of confusion. The whole incident was actually pretty hilarious to watch.
Parent
The Transit Authority Incident (Score:4, Interesting)
I am on a list of bidders for potential contracts with the Washington Metropolitan Transit Authority, which operates the Metrobus and Metrorail for Washington, DC and the nearby suburbs in Maryland and Virginia. The annual budget for the Authority is in excess of a billion dollars; it's larger than the budget of the entire State of Montana, for example.
One time I got a message with more than 25 recipients on it regarding a change in the way they were operating their procurement website. Well, I suspected that it was some spammer pretending to be from the Authority, because one of the "red flag" signs of being spammed is more than 10 recipients on the same messsage. But I discovered that it really was from the Transit Authority, it was simply an ordinary announcement with no url links and nothing but the announcement. But instead of simply either making the recipients BCC recipients, and sending it to an internal transit authority e-mail address as To:, or sending individual messages to each potential supplier, the contracting agent had simply sent it out To: listing all persons who were registered as bidders with the authority.
My e-mail address was one of these potential suppliers along with a few other people.
1,627 other people to be precise. This was the longest To: list on an e-mail message I have ever seen on a piece of e-mail that wasn't spam; 1,628 contacts. No, I didn't reply all, but I couldn't think of a way to refer to this incident as a "Send All" message and tie into this story. The other half of this incident was that the procurement agent had also just given all potential suppliers to the Authority, every other supplier's e-mail address, too.
In the same vein... (Score:5, Interesting)
I signed a confidentiality agreement with them.
I am not allowed to discuss ANYTHING about the product or reveal I am testing it or anything. I was never there, I am nobody.
Last year I got an email - From The President of The Testing Company - personally thanking me for all the help in the last year.
He also thanked everyone else who "helped" last year as well and I could see who they were because apparently the President (or the secretary) just put all our emails into the TO: field and let it fly.
Lots of Identifiable people on the list because they used their WORK email, like john.doe@largecorporation.com So it was easy to see who else was part of that big Butt Plug testing program.
I did a REPLY to ONLY the President and laid into him about the confidentiality agreement and told him if he didn't know how to use email to stay away from the computer.
Later that day we all got another email from the President, this time apologizing for revealing all our personal emails, never happen again etc etc. And apparently he figured out how to use BCC!
So yelling at someone does seem to work to change behaviour.
Also- this is a dupe comment, I posted this once before on Slashdot someplace, but since this is Slashdot I didn't think a dupe would be a problem.
Parent
Conspiracy Nut: What an easy way to "lose" mail (Score:4, Insightful)
Am I the only one to think that it is quite peculiar that it is happening 9 days before the Government turns over? I mean, how much difficult could it be to say that some sensitive/embarrasing mails got lost during this crash? I think this should be looked into in more detail and make double sure that no mail was "lost".
Wrong(?) (Score:5, Informative)
OpenNet, by a very quick look on google, seem to be their network name for the non-classified bits and pieces. Supposedly Microsoft + Cisco stuff.
Feel free to disagree, but please provide a URL reference to the OpenNet email server software vendor if doing so.. ;-)
Parent
Re:Wrong(?) (Score:5, Funny)
TFA mentioned the use of the recall feature that is only supported by Exchange servers and Outlook.
Of course not :
# cd /var/spool/mail
# rm *
see ? It's all been recalled !
Parent
YES it's Exchange and yes it crashed... surprised (Score:5, Informative)
yes its exchange internally
openNet is what they brand it as
feel free to correct me with evidance that it was not the case any more but I know 2 exchange servers there and this say's otherwise [74.125.45.132]
exchange has the recall ability and so does lotus notes
most other servers do not have this feature for very good reasons l
regards
John Jones
www.johnjones.me.uk my blog about email and digital communication [johnjones.me.uk]
Parent
Re:YES it's Exchange and yes it crashed... surpris (Score:5, Funny)
For a train your English is quite good.
Parent
Re:Exchange, huh? (Score:5, Insightful)
And next I expect you're going to try and teach them to edit their quotes and to stop top posting ?
I'm close to giving up on educating users with email, it's pretty hopeless I think.
Top posting is easier for most users to understand. For business users, its best to top post by default, unless you are going to counter a long email point by point. In that case please be sure to top post the words "see below."
On open mailing lists (anything not run by your employer where you decide to subscribe) I bottom post and edit posts. At work, I top post. It gives a complete linear history of a conversation, which is good because most outlook users just sort email by date.
Some people just can't handle reading properly formatted reply emails, let alone writing them.
Parent
Re:Exchange, huh? (Score:4, Interesting)
I recently (as in, within the last week) gave in to breaking netiquette and dumbing down my emails by top posting. Why? I used to respond through the email, making sure everything is properly indented (for HTML users) or prefixed with a >. I also edit out extraneous content with [snip].
And yet, every time I get into discussions with clients who aren't VERY computer savvy via email and respond to each point in order (as one SHOULD to make it EASIER to understand) they miss it.
Why?
They skim. They don't look for anything beyond the top of the email. If it's not contained in the top paragraph, it's obviously not important enough to worry about.
Proper netiquette saved my behind in corporate America (the company for the specific company in question shall rename nameless. Let's just say that it's a Waltham-based HR software company which was recently bought out. They're not SAP or Oracle so you may not know of them unless you're with Fidelity, State Farm, Sears, etc. HR departments). VPs used to come back to me and demand to know why my team didn't find certain obvious showstopper defects (I personally found the ones in question, analyzed the potential show-stopping, contract-voiding effects the bugs might have, and argued for their resolution and was vetoed by the COO and CEO personally). Well, as it turns out, they denied ever hearing about the defects, and the blame was on me since I was directing Quality Assurance. Fortunately for me (or perhaps unfortunately because I remained at that company for 2.5 years after that incident because I foolishly believed their lies about stock options, etc. - in the recent buyout "preferred stock" holders got NOTHING but the common stock holders (mainly the CEO and CIO) made millions - I could have moved on to any of the much better offers that came my way during that time) I archive ALL my email. I don't delete email unless it's spam or jokes, etc.
These were bugs I brought up to the director of client services, the COO, and the CEO (I went up the food chain properly) and while the director of client services wanted it fixed; she immediately saw the potential ramifications, the COO and CEO flat-out rejected it, citing the costs involved in fixing it (it was an architectural issue which would have required 3 to 6 days of dedicated time for the chief architect, myself, and two Sr. software engineers).
The issue blew up at a client site. The client spent months and months developing content (in our English-like business logic language), assuming that plans would display to the employee as our Sales and Support staff claimed it would, and didn't set up the complex tests I did to verify. Silly client, they assumed that the software works as advertised! They discovered after hiring many temps and contractors to develop their HR portal that data inheritance was completely broken. They had to reimplement 6+ months of work. Well, needless to say, the shit hit the fan at that point.
The CEO and COO came by my desk (I moved on to Release Engineering at that point, wanting to do more coding and and playing less political games since the executives were morons, in denial about our being a software company despite our sole product is software and our sole service is designed on selling seats of product and number of subjects and plans, not hours/days/resources for implementation. It was a very product-driven model and the customers were treated as a product-driven company would operate.). They wanted to know why these defects were not found under my watch. Fortunately while I was director not only did I personally read every single defect to classify and prioritize them properly and ensure they were assigned to the correct software engineer, but I also happened to be the one who discovered and analyzed that defect, suspecting it was broken when I was digging through some old spaghetti code we had in place driving that module.
So, I found the email thread in outlook in about one minute. I also happened to have follow up messages citing our con
Parent
Re:Exchange, huh? (Score:5, Informative)
Looks like the pathetic one is you, and the Submitter. If you RTFA, it clearly says
He said the result was "effectively a denial of service as e-mail queues, especially between posts, back up while processing the extra volume of e-mails.
Never says the actually crashed, merely that the high volume generated large queues, exactly what you would expect to happen in a properly engineered system. But hey, this is Slashdot, so making up reasons to hate Exchange (and there are plenty of LEGITIMATE reasons to hate exchange) is the norm.
Parent
Re:And in other news... (Score:5, Funny)
Good to know that rigorous competition in the marketplace has totally eliminated misuse of 'reply-all' in the private sector. I look forward to continuing to have a lower life expectancy and higher infant mortality than Canadians and Swedes.
Parent
Re:And in other news... (Score:5, Interesting)
And in even further news, corporations are not perfect.
I take it you're not familiar with how enterprises plan. They plan for regular load, not aberrant once-in-a-blue-moon load. This is bog standard behavior for a system responding to people doing stupid things. If you think this is restricted to the US government, you've never worked in corporate IT.
Parent
Re:'recalling' email - laugh! (Score:4, Insightful)
Why is that funny?
Exchange has a feature your email client didn't support. Ha ha ha!! IT'S HILARIOUS!!!!
Parent
Re:'recalling' email - laugh! (Score:5, Informative)
Message recall. Oh dear.
Years ago, I wrote the bulk of this feature. It is not an Exchange feature, but an Outlook feature. It works by sending a custom MAPI message that Outlook recognizes and processes. Of course, this only works if all recipients are using Outlook. It also, after we did some usability testing, only deletes unread email, or email that has not been moved to a subfolder (the original version was quite determined and would hunt down and kill the message even if it had been moved to a subfolder, renamed or entered the email protection program). In this way, it did not violate the UI dictum that the computer move things around when you haven't given it instructions to do so.
So yes, it is Outlook only. If sent to a non-Microsoft mail system, it degrades to a simple notification that the message is being recalled. And it does not a good choice for getting rid of flames you shouldn't have been sending. But within its expected use as a feature - correcting mistakes in email that should have been caught before pressing send, it works fairly well.
But because it is client based, rather than an Exchange feature, it does cause a new mail message to be sent to each original recipient and, combined with a send-all storm, could greatly exacerbate things.
And, preemptively, for those who have philosophical objections to me having written the code in the first place, I'll just have to live with your disapproval and hope my steady paycheck somehow sooths my guilty conscience.
Parent
Re:'recalling' email - laugh! (Score:4, Funny)
And, preemptively, for those who have philosophical objections to me having written the code in the first place, I'll just have to live with your disapproval and hope my steady paycheck somehow sooths my guilty conscience.
Sir, your conscience should enable you to sleep as a baby. This feature alone has provided me with countless hours of entertainment as people suddenly realize the error of their ways as they make bold statements (often of a derogatory nature) in an e-mail which happens to be sent to the wrong person in CC. As they hastily move to send out the recall of their mail only to receive a mail from someone who was offended.
This feature combined with the nature of some people to act before they think has proven quite entertaining indeed. Please enjoy your steady paycheck to its fullest, your code has amused me countless times.
Parent
Re:Ban the Reply All Function (Score:5, Funny)
This is the sort of thing that listservs seem to do pretty well.
I just wish I could convince more of my users to use them. I have one winner who sends a list using 300+ CC's. The anti-spam system on the mail server slows that list to a crawl (deliberately). They wonder why it takes 3 hours to send, and I tell them to use the list server that we set up, but it's different and they don't want to be bothered. I think I'll make it take 6 hours next time.
Parent
you have no idea how unemployment works (Score:5, Informative)
The guy who wants to quit but doesn't because he'll only get unemployment benefits if he's fired :)
Um...which goes to show how little you know about unemployment. At least in MA, you don't get shit if it is "termination with cause", ie fired. If you're laid off, great- but even then, your employer gets a phone call from the unemployment department asking whether you were fired or laid off. Nothing stops them from lying and saying you were fired with cause- and then you've got a legal battle on your hands, which you can't afford.
Other fun facts about unemployment in MA: you don't get paid for two full weeks after you FILED- not after you were laid off, but after you FILED. You get a pittance compared to your normal salary; you'd be lucky to make rent on a studio apartment in Boston based off an entire month's unemployment checks.
Any income is deducted from your UA check. Say for example you find a 2-3 hour consulting thing on CL and make $150 helping someone fix their computer. Guess what? Your unemployment check for that week will be $150 smaller. This basically means that you have no incentive to find any kind of income while you're on UA.
Last but certainly not least: you have to pay taxes, medicare, medicaid, etc on your unemployment benefits. It's not bad enough that you're basically on welfare- you have to fork over a portion of the money the government is giving you, BACK to the government. Cute, eh?
Parent
Re:Not the first time this has happened... (Score:5, Funny)
I saw a weird variant on that back in university.
One of the engineering departments had a room full of (at the time) fairly high end sun workstations, and these were used both interactively and for people running longer compute jobs overnight.
To facilitate overnight jobs, the admins had set up a round robin dns alias that updated every couple of seconds to point to the machine reporting the lowest load average.
One of the students in my class had the bright idea of "If put 'ssh lowest' in my bashrc file, every time i open a terminal window it'll automatically pick the least loaded machine".
Fast forward a few minutes and we've got 80 sun workstations which have all systematically ssh'd to each other and none of which will accept any new connections...
Parent