State Dept E-mail Crash After "Reply-All" Storm 384
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
Bedlam... (Score:5, Interesting)
http://msexchangeteam.com/archive/2004/04/08/109626.aspx [msexchangeteam.com]
Again...
-Ghostis
sigh (Score:5, Interesting)
What an irony that they decided to mass mail when they've warned their employees not to do so. What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response... However, judging by the nature of their work [it is the state department after all] I don't believe it unreasonable that there could be events in their future requiring such mass mailings again and having the whole system crash under the load would be no doubt very bad in emergencies.
Two questions: (Score:5, Interesting)
a) Maintaining large list by copying all recipients into the hrader is a fucked up idea at best (because there is no way this list will be kept updated), and a informaiton leak at worst (because somebody eralier on a non-updated list may get information which he should not get - e.g. former employees). Why do governmental institutions still us it?
b) Why in the world do modern e-mail clients still allow reply all to hundreds of recipients without an additional safety question. I would expect my program would warn me before sending an emails to thousand people.
Re:And in other news... (Score:5, Interesting)
And in even further news, corporations are not perfect.
I take it you're not familiar with how enterprises plan. They plan for regular load, not aberrant once-in-a-blue-moon load. This is bog standard behavior for a system responding to people doing stupid things. If you think this is restricted to the US government, you've never worked in corporate IT.
Re:Bedlam... (Score:5, Interesting)
The Transit Authority Incident (Score:4, Interesting)
I am on a list of bidders for potential contracts with the Washington Metropolitan Transit Authority, which operates the Metrobus and Metrorail for Washington, DC and the nearby suburbs in Maryland and Virginia. The annual budget for the Authority is in excess of a billion dollars; it's larger than the budget of the entire State of Montana, for example.
One time I got a message with more than 25 recipients on it regarding a change in the way they were operating their procurement website. Well, I suspected that it was some spammer pretending to be from the Authority, because one of the "red flag" signs of being spammed is more than 10 recipients on the same messsage. But I discovered that it really was from the Transit Authority, it was simply an ordinary announcement with no url links and nothing but the announcement. But instead of simply either making the recipients BCC recipients, and sending it to an internal transit authority e-mail address as To:, or sending individual messages to each potential supplier, the contracting agent had simply sent it out To: listing all persons who were registered as bidders with the authority.
My e-mail address was one of these potential suppliers along with a few other people.
1,627 other people to be precise. This was the longest To: list on an e-mail message I have ever seen on a piece of e-mail that wasn't spam; 1,628 contacts. No, I didn't reply all, but I couldn't think of a way to refer to this incident as a "Send All" message and tie into this story. The other half of this incident was that the procurement agent had also just given all potential suppliers to the Authority, every other supplier's e-mail address, too.
In the same vein... (Score:5, Interesting)
I signed a confidentiality agreement with them.
I am not allowed to discuss ANYTHING about the product or reveal I am testing it or anything. I was never there, I am nobody.
Last year I got an email - From The President of The Testing Company - personally thanking me for all the help in the last year.
He also thanked everyone else who "helped" last year as well and I could see who they were because apparently the President (or the secretary) just put all our emails into the TO: field and let it fly.
Lots of Identifiable people on the list because they used their WORK email, like john.doe@largecorporation.com So it was easy to see who else was part of that big Butt Plug testing program.
I did a REPLY to ONLY the President and laid into him about the confidentiality agreement and told him if he didn't know how to use email to stay away from the computer.
Later that day we all got another email from the President, this time apologizing for revealing all our personal emails, never happen again etc etc. And apparently he figured out how to use BCC!
So yelling at someone does seem to work to change behaviour.
Also- this is a dupe comment, I posted this once before on Slashdot someplace, but since this is Slashdot I didn't think a dupe would be a problem.
Re:Two questions: (Score:1, Interesting)
Simple solution:
Rename "Reply all" to "Reply to N recipients."
Compare:
Reply all
Reply to 5 recipients.
Reply to 154,261 recipients.
Re:Wrong(?) (Score:3, Interesting)
TFA mentioned the use of the recall feature that is only supported by Exchange servers and Outlook.
Re:you have no idea how unemployment works (Score:1, Interesting)
Yes, except it doesn't annoy you into changing your circumstances. Ok, so the amount you get is so small that it's hard not to exceed it, but the fact that any money you earn is subtracted from the amount you get means you don't have any incentive whatsoever to go after any job that will pay you less than the full value of your benefit plus the full value of the free time you'd be giving up for the job. Why mow a guy's lawn for $10 if you're going to come out the same at the end of the month? Unless you get a substantially better opportunity, it encourages you to maintain your circumstances.
By contrast, a scheme where, say, a quarter of the money you earn is subtracted from your unemployment benefit means that doing anything earns you money; all of a sudden mowing that lawn means you've got $7.50 more than you would have if you didn't do that. There's still the value of your time, of course, but all of a sudden the barrier of "job worth taking" is much, much lower.
Re:Bedlam... (Score:3, Interesting)
So you know enough about Exchange to know the Registry Key for configuring a max recipient count, but not enough to think that they were using DLs, which count as one recipient?
A DL would only be reply. The problem is with reply-all, meaning there was a list of addresses in the CC: or To: fields. Otherwise, we would not even be discussing this. If it were just a DL then reply and reply-all are essentially the same function, no?
So it sounds like they need to install decent mailing list software, not just an "everyone" address.
Re:Exchange, huh? (Score:4, Interesting)
I recently (as in, within the last week) gave in to breaking netiquette and dumbing down my emails by top posting. Why? I used to respond through the email, making sure everything is properly indented (for HTML users) or prefixed with a >. I also edit out extraneous content with [snip].
And yet, every time I get into discussions with clients who aren't VERY computer savvy via email and respond to each point in order (as one SHOULD to make it EASIER to understand) they miss it.
Why?
They skim. They don't look for anything beyond the top of the email. If it's not contained in the top paragraph, it's obviously not important enough to worry about.
Proper netiquette saved my behind in corporate America (the company for the specific company in question shall rename nameless. Let's just say that it's a Waltham-based HR software company which was recently bought out. They're not SAP or Oracle so you may not know of them unless you're with Fidelity, State Farm, Sears, etc. HR departments). VPs used to come back to me and demand to know why my team didn't find certain obvious showstopper defects (I personally found the ones in question, analyzed the potential show-stopping, contract-voiding effects the bugs might have, and argued for their resolution and was vetoed by the COO and CEO personally). Well, as it turns out, they denied ever hearing about the defects, and the blame was on me since I was directing Quality Assurance. Fortunately for me (or perhaps unfortunately because I remained at that company for 2.5 years after that incident because I foolishly believed their lies about stock options, etc. - in the recent buyout "preferred stock" holders got NOTHING but the common stock holders (mainly the CEO and CIO) made millions - I could have moved on to any of the much better offers that came my way during that time) I archive ALL my email. I don't delete email unless it's spam or jokes, etc.
These were bugs I brought up to the director of client services, the COO, and the CEO (I went up the food chain properly) and while the director of client services wanted it fixed; she immediately saw the potential ramifications, the COO and CEO flat-out rejected it, citing the costs involved in fixing it (it was an architectural issue which would have required 3 to 6 days of dedicated time for the chief architect, myself, and two Sr. software engineers).
The issue blew up at a client site. The client spent months and months developing content (in our English-like business logic language), assuming that plans would display to the employee as our Sales and Support staff claimed it would, and didn't set up the complex tests I did to verify. Silly client, they assumed that the software works as advertised! They discovered after hiring many temps and contractors to develop their HR portal that data inheritance was completely broken. They had to reimplement 6+ months of work. Well, needless to say, the shit hit the fan at that point.
The CEO and COO came by my desk (I moved on to Release Engineering at that point, wanting to do more coding and and playing less political games since the executives were morons, in denial about our being a software company despite our sole product is software and our sole service is designed on selling seats of product and number of subjects and plans, not hours/days/resources for implementation. It was a very product-driven model and the customers were treated as a product-driven company would operate.). They wanted to know why these defects were not found under my watch. Fortunately while I was director not only did I personally read every single defect to classify and prioritize them properly and ensure they were assigned to the correct software engineer, but I also happened to be the one who discovered and analyzed that defect, suspecting it was broken when I was digging through some old spaghetti code we had in place driving that module.
So, I found the email thread in outlook in about one minute. I also happened to have follow up messages citing our con
Re:Exchange, huh? (Score:3, Interesting)
That post wasn't rambling. It only took me a minute to read. I'm pretty sure his business emails contained just enough needed data to get the point across.
Perhaps people should learn how to read again without going "tl;dr" everytime they see more than 1 paragraph. Which is why he had to start top-posting, because that's what people do. More than one paragraph, they don't read it because that would tired their little brains out too much to spend 1-3 minutes reading.