State Dept E-mail Crash After "Reply-All" Storm 384
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
Re:Bedlam... (Score:4, Insightful)
Incorrect Headline (Score:5, Insightful)
Whoever wrote the headline for this summary needs to have their slashdot editor privileges revoked.
TFA states "an e-mail storm nearly knocked out one of the State Department's main electronic communications systems", and "a major interruption in departmental e-mail". The problem is clearly spelled out as "e-mail queues, especially between posts, back up while processing the extra volume of e-mails".
This is simply the queues backing up, not the servers crashing. Nowhere does TFA state anything to suggest that there was a "State Dept E-mail Crash", which the summary's headline boasts. The proper headline should read "Large E-mail Queues at State Dept After Reply-All Storm".
No, I'm not new here. That's why I'm fed up with the sensationalist "journalism" that is getting worse and worse here.
Re:And in other news... (Score:3, Insightful)
If, like approximately 1/2 of the American population, you currently had no health care at all, your attitude would probably be different.
And you might want to remember that the current financial and industrial collapse was given to us by the finest and most highly educated examples of stupid, greedy, incompetent, short sighted, overpaid, negligent, and possibly criminal management that private enterprise has been able to produce and promote.
'recalling' email - laugh! (Score:1, Insightful)
I was at a company that used 'exchange' for their mail (sigh). most users did windows but I run freebsd and used imap and local ascii client for my mail.
one day a marketing person sent out mail and sent the wrong thing. they then sent some kind of 'recall' message.
the thing is, my ELM user agent didn't listen and neither did my IMAP puller ;)
recalling an email. yeah, right. pretty laughable.
Ban the Reply All Function (Score:3, Insightful)
To all the mods, please don't destroy all my Karma. I really do hate that Reply All button.
Re:sigh (Score:5, Insightful)
Re:Ban the Reply All Function (Score:2, Insightful)
It is like adding "Press this button to be fired" function to your corporate email system. You know someone is going to press the button
Yes: The guy who wants to quit but doesn't because he'll only get unemployment benefits if he's fired :)
Re:Bedlam... (Score:4, Insightful)
Change that to 'that warns if an exceptionally large number of messages are being sent and smack the user over the head with a LART if they don't click cancel' and i'll agree with you.
A large company should have an internal mailing list and/or intranet system that individual users can post messages to. Letting individual users send email to more than a few thousand users in one hit is madness. Especially if they are anything like our customers where they think it is a good idea to send a 10MB attachment to 500 users...
Re:Thsi is a test... (Score:5, Insightful)
The problem is the morons who send email with "everybody.all.everwhere" (or whatever) in the To: or CC: list. If they were smart enough to put them in the BCC: field, it would be impossible for people to clog up the system with Reply All. Alas.
Re:'recalling' email - laugh! (Score:4, Insightful)
Why is that funny?
Exchange has a feature your email client didn't support. Ha ha ha!! IT'S HILARIOUS!!!!
Re:sigh (Score:5, Insightful)
What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response...
No. What they should have done was installed a mailing list manager, created a read-only list called "employees", and posted to it. Voila - n-thousand workers get announcements with no ability to reply to the whole list. Problem solved.
Reply All Insanity (Score:3, Insightful)
I've seen it so many times over the years. I wonder why it's so hard to add an administrative setting that limits Reply All to a certain number of users? Set at 100, it would only send the first 100, then ask the user if they wanted to send the next 100. Or 300 or 400 or whatever.
I can't count the number of people sending a hasty and blistering reply to thousands of people. Not only committing public suicide but accounting for who knows how many unproductive man hours while the entire organization stopped to read their spew. It's just crazy.
President-Elect Obama Assassinated! (Score:5, Insightful)
This just in, President-Elect Obama Assassinated! Oh, don't take it so literally. I was just trying to capture your attention in a short amount of time. Obama wasn't killed, silly. There was just some CHARACTER assassination against him on a late night talk show.
Re:Bedlam... (Score:5, Insightful)
And then they threaten disciplinary action if someone uses them the wrong way. Wouldn't it be so much easier to just lock them down? It's what most companies do.
it's useful for some stuff (Score:3, Insightful)
I use the reply-all button frequently, for ad-hoc small group discussions. If I have a document I want two people to review, I send it to both of them, and they send their comments back to both me and the other person I sent it to with reply-all, so we're all on the same page.
If the same group of people is frequently collaborating you can set up a mailing list, but it's a real pain in the ass to set up a mailing list every time you want a group of 3 or 4 people to exchange 5-10 emails.
Reply All isn't the problem (Score:5, Insightful)
The problem is the message replied to having - RTFA - several thousand addresses in the To: and CC: fields. This is what BCC is for . Allowing people to put several thousand addresses in to the headers will eventually result in a mail storm, whether someone hits Reply To All or not. The first time someone opens a virus laden attachment that goes through their (archived by law, this being a federal agency) emails, it will send itself out to thousands of equally clueless people. One of them will run the attachment, which will send another copy to several thousand people. And so on. This happened where I work once, by people who should have known better. Before it was done, I was getting two hundreds copies of the virus per day.
Whoever sent out the message replied to should be fired and criminally prosecuted for deliberately sabotaging the State Department's email system. But since the article doesn't mention this at all, I'm assuming it was some dumbass boss somewhree who is immune to any form of disclipline for anything, up to and including murder.
Re:Bedlam... (Score:5, Insightful)
What the fuck? Don't do that. Reply all has a valid use case. In fact it's the way everyone at my company most commonly replies to email messages. Why? Because the CC list is there for a reason - those are people who are supposed to know what's going on in that email thread.
How about just educating your users on checking who they're sending an email to, every single time they send one.
Re:Bedlam... (Score:4, Insightful)
If you were my admin and you did this to me I would hunt you down and kick you in the shins.
In the year 2009 we now often work in teams. We often communicate as teams. We often 'think' as... you guessed it... teams.
But by all means I'm sure whatever company you're working at people only talk to one person at a time. You have no group discussions and the only interaction that occurs between employees is by the watercooler and in meetings.
At our company however more than one recipient is the norm. Especially when you want to keep a project manager 'in the loop' of a conversation with a vendor. In fact our most common occurance is to have to say "oops - sorry looks like I dropped so and so from this conversation". Not "Ooops, I accidentally killed our mail server while talking to 4 people."
So go ahead and remove Reply-All in the classic System Administrator "I don't care how my users want to use my network. It's mine and I'll do as I please." dick move. Because that's what it is. It's a Dick move and expect irate emails from users who suddenly find their email doesn't work very well anymore.
Re:Ban the Reply All Function (Score:3, Insightful)
Re:And in other news... (Score:1, Insightful)
You're suggesting that Congress somehow forced private enterprise to behave stupidly, greedily, incompetently, negligently and possibly criminally? That private enterprise was unable to resist and followed each other, like lemmings, over the cliff?
Keep in mind the fact that a recurring theme on /. (plus in most conservative, liberal, centrist, and libertarian publications) is that government in the US has been reduced to not much more than the paid-for pawn of special interests with big money.
A huge problem is that the thing which surprises us most about politicians is not that they're whores, but that they're cheap whores. Many seem perfectly willing to sell out their constituents, the country, and the constitution for relatively small campaign donations. I might be able to understand an expensive whore of a politician accepting payment of hundreds of thousands or millions of dollars in exchange for extending some favor. But the politician who takes a few thousand dollars in exchange for letting make millions or even billions of dollars be made at the ultimate expense of the rest of us is a cheap whore.
but... (Score:3, Insightful)
Nice knee jerk reaction there, spunkyfuck. (Score:1, Insightful)
Re:'recalling' email - laugh! (Score:3, Insightful)
When somebody sends out an email with incorrect information? Oops, I just noticed that I said 3 pm for the meeting, but I forgot about the timezone change so it's actually 2 pm. Or I just mistyped. Whatever.
Do a recall and replace - that way you don't have people thinking the second email was just a duplicate.
Re:Two questions: (Score:5, Insightful)
Are you sure you want to do this stupid thing ?
pops up, people universally click "OK" without a second thought.
People have just been blasted by too many of these warnings to take any proper note any more.
Re:Incorrect Headline (Score:3, Insightful)
Well, yes, but I can only think of one transmitting MTA that can be overloaded by an e-mail with a thousand recipients in the same domain. Most MTAs will not split up the e-mail into one incarnation per recipient when spooling it, but work on a single copy, and only transmit the e-mail once per unique MX. Thus the only real load is on the delivery agent once the mail gets to the final destination, and not on the sending mail server or intermediates.
And if the mail delivery agents are overloaded, that should not affect sending e-mail in any way; only the time before you get to receive new e-mail will be impacted. (Which is a self-adjusting system, because if it takes long for you to receive e-mail, you will answer them less often too...)
Re:Bedlam... (Score:4, Insightful)
How about conducting discussions on discussion boards rather than email?
Great Idea!
"Dear Vendor,
Since several people will need to be in on this conversation please visit our website at:
www.webboards.com and create a user account and password.
When you've registered you will need to visit the topic "Vendor discussions/XYZ Project" and pay special attention to the topic "How do we implement X without breaking Y or can we live without Y?".
We would really appreciate your input.
Signed,
Employee who's about to be fired."
Brilliant idea! A webboard! Now... I just need to send a mass email to everyone involved to let them know there is a new topic open and how to get to it. That sounds far more intuitive than just sending an email to 5 people!
Re:Bedlam... (Score:3, Insightful)
More problems with webboards:
Where do you host the webboard? If it's not exposed to the internet then it's publicly available to hacking. If it's not exposed to the internet then nobody can access it externally. An email server can be hardened much more easily.
How do you notify people of new replys? You send out emails.
How do you keep a conversation private and then open it up to other people as needed? Discussion boards with per topic passwords? Sounds really awkward.
Emails are around for a reason. You only need to check them when something is relevant to you, they're publicly accessible and easy to send to. Everybody has one already regardless of where you work. With conversation threading they can act just like a webboard. They're peer to peer without a need to decide which host will carry the project. It's a good technology.
The *correct* solution to this reply to all situation is to detect an action which is determined to take longer than X time and alert the user that they're about to do something horrible. If it's going to take Y time tell the user the server is unable to send the message and what they can do to fix it. This is a UI design problem. Not a feature problem.
Re:Bedlam... (Score:4, Insightful)
That said, I think there actually should be a distribution list for the entire company - it can be useful for some stuff.
However the actual name should be hard to guess, and secret.
Then you set up the "everyone" list for people to send to which actually goes to a moderator.
If the moderator thinks the email should go out, it is sent out via the "secret-real-everyone-list", otherwise it isn't.
If the email indicates that the sender has significant lack of discretion or intelligence, the moderator may wish to pass it to the Bosses concerned so that they can take necessary measures.
In one of the places I worked for "everyone" actually went to the Big Boss(es), and I think it worked reasonably well.
Re:Bedlam... (Score:5, Insightful)
It doesn't need to be secret if there are controls on who can send messages to the list. It is so trivial to do for any competent email admin no matter what software they use.
Re:Bedlam... (Score:2, Insightful)
Not good enough.
He will still expect the angry "where is my reply-all functionality", and this is still the wrong way to deal with the situation. Not all people know the shortcuts (I didn't).
Instead of using a scalable solution to deliver mail by wasting what, 1-2 admin workdays altogether, he's going to waste at least 1 hour per month per person to work out that his Majesty Dick has been screwing again with their mail system.
Re:Exchange, huh? (Score:3, Insightful)
Re:'recalling' email - laugh! (Score:3, Insightful)
[Corrected] Send a followup with a changed subject line so that people know something has changed.
Re:Exchange, huh? (Score:5, Insightful)
And next I expect you're going to try and teach them to edit their quotes and to stop top posting ?
I'm close to giving up on educating users with email, it's pretty hopeless I think.
Top posting is easier for most users to understand. For business users, its best to top post by default, unless you are going to counter a long email point by point. In that case please be sure to top post the words "see below."
On open mailing lists (anything not run by your employer where you decide to subscribe) I bottom post and edit posts. At work, I top post. It gives a complete linear history of a conversation, which is good because most outlook users just sort email by date.
Some people just can't handle reading properly formatted reply emails, let alone writing them.
Microsoft (Score:3, Insightful)
They used exchange and got screwed, just like -everyone- who uses exchange.
This happens all the time just most companies cover up stuff like this because it's not good for the share price.
Conspiracy Nut: What an easy way to "lose" mail (Score:4, Insightful)
Am I the only one to think that it is quite peculiar that it is happening 9 days before the Government turns over? I mean, how much difficult could it be to say that some sensitive/embarrasing mails got lost during this crash? I think this should be looked into in more detail and make double sure that no mail was "lost".
Re:YES it's Exchange and yes it crashed... surpris (Score:3, Insightful)
The very fact that there is a "recall" feature shows a lack of understanding of how email works.
What's the phrase? Trying to take something off the internet is like trying to remove piss from a pool.
One can argue that internally they should have complete control, but even then one is racing against time to delete the message *on the client* before the person reads it. What's the point?
Re:Not the first time this has happened... (Score:2, Insightful)
Good stuff, university being used for what it's there for - education. I'm sure that guy won't ever make that mistake again.