Super Bowl Blackout Caused By Defective Protective Relay 210
New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."
Did someone lost his job? (Score:2)
Re:Did someone lost his job? (Score:5, Funny)
That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?
Presumably the person that receives the big end-of-year bonus when everything goes well?
Re:Did someone lost his job? (Score:5, Funny)
Yup, that's the way it goes in some parallel universe :)
Re: (Score:2)
That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?
Presumably the person that receives the big end-of-year bonus when everything goes well?
They probably get a higher end-of-year bonus for resolving the problem.
Upper level compensation works in mysterious ways - plenty of CEOs manage to get guaranteed bonuses that are completely detached from company performance.
Re: (Score:3, Funny)
No, they are not detached from company performance.
If the company performs well, the bonus becomes astronomical. If it performs less well, the bonus is merely unbelievable.
To regular people, it appears to be detached from reality.
Re: (Score:2)
We watch companies fail, then get bailed out by the government, after which CEO's still get astronomical bonuses.
Yes, the bonus system is detached from reality. Any CEO whose company fails as a result of his actions, inactions, and poor judgement calls should be sacked, then sued for dereliction of duty. Said CEO should never again have access to resources which he hasn't earned, himself. Let him push a frigging broom and mop for the rest of his life.
Those astronomical bonuses to which you refer have no
Re: (Score:2)
I suggest you get someone to read davester666's post to you, and then explain it. Someone from England would be a good choice.
Re: (Score:2)
I suggest you read Davestar's post, as well as my own again. I've disagreed with Davestar. He claims that the failures who ran banks into the dirt didn't get astronomical bonuses, merely unbelievable bonuses. I stated that they did indeed get astronomical bonuses.
I have little idea which language might be your own native language, but respectful disagreement seems pretty obvious to me here. Davestar basically split a hair, and I told him that execs always get the bigger half of the hair.
Who gets fired vs. who gets bonuses (Score:2)
.
hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels, so the firing most often will happen to those at the lower level who executed the commands, including putting in crappy materials that were ordered when the higher-ups want to save money. At least that seems to be the way of the USA; Japan's older way would have those responsible all the way
Re: (Score:2)
Well thanks for pointing that out, Miss tic-tac tits.
If you hadn't been here we'd have been grossly misinformed about how capitalism works! Shame on you, isorox, for lying to us so!
Re: (Score:2)
What you say is pretty much true. Sadly, that didn't happen because Shrillary is that good - it happened because congress are such pussified pretenders. Few congressmen seem to have any clue what life is like outside the United States. They are to busy sucking the corporate teats that keep them in power to pay attention to the real world.
Re: (Score:2)
Re: (Score:2)
It was a fail safe (Score:4, Informative)
Re: (Score:2)
It was too many people tweeting at the same time.
Re:It was a fail safe (Score:5, Informative)
It apparently did it's job. But apparently it was given the wrong job. It is accused (by the manufacturer, of course) that someone entered the wrong amperage that it should do its job at. Unlike home circuit breakers which come in specific amperage levels (and vary from unit to unit by plus or minus 10 percent which is considered acceptable), these relay devices, which are a component in an overcurrent protection system, cannot be made at fixed amperage levels due to economics. They are quite expensive to replace with another just to tweak the settings due to changes made elsewhere in the power distribution network, and the number of different amperage values needed would be very large. They can be expensive also because either they directly connect to current transformers that have high open circuit voltage potential, or operate from digital sensors on the current transformers. They are also expected to have accurate at better than one percent.
Re: (Score:2)
Wasn't there a practice run done? Really, if you're responsible for a multi-billion dollar event, it might be a good idea to fire everything up a few times and make sure it behaves as intended.
Re: (Score:2)
Re: (Score:3)
Just to add to this, the term "circuit breaker" refers to a switch that can open (interrupt) and close a circuit under load without arcing or damage. They are typically designed with springs or gas charges to very rapidly open the contacts without arcing or have channels around the contacts called arc chutes that extinguish arcs (older DC breakers had "blow out" coils, when opened the arc would be magnetically pushed out until it extinguished) . "Over current protection" refers to a device that guards again
Re: (Score:2)
Re:It was a fail safe (Score:5, Informative)
Apparently the circuit breaker failed even when there was no short-circuit event.
I work as an electrical engineer with an electric utility and it seems to me the circuit break perform as intended. The relay told the breaker to trip (open), so it did. After a series of check to make sure no equipment was damaged, electricians were able to close the breaker again.
The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.
Re: (Score:2)
The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.
Tested? Who tests anything?
Re: (Score:2)
The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.
Tested? Who tests anything?
In my line of work. If you don't properly commission your stuff, it has high consequences. Both in casualty and in economic consequences.
As a result, if someone works for me and not properly test/commission the equipment, I will make sure they don't work for me ever again, period. If they fuss then I will see to it that the rest of the company knows how incompetent you are and why your company/affiliate tolerate such unprofessional attitude and conduct. It's a fairly small circle in my line of work t
Re: (Score:2)
My only thought was, "The power went out during the Superbowl? I'm surprised I haven't heard more bitching".
That was about it. Although there are apparently millions of Americans who care, there are also millions of Americans would don't.
Re: (Score:2)
I was expecting millions of non-Americans to not care ... seems I was wrong, judging from news-sites etc.
The TL;DR (Score:3, Informative)
Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).
Re:The TL;DR (Score:5, Interesting)
Re: (Score:2)
.
I know that a fuse is thermally activated with
Re:The TL;DR (Score:4, Informative)
In my experience, most relays have a "Instantaneous" setting that goes off as fast as possible if you have like 20-30 times as much current as should be there, a "Short Time" setting that goes off in few seconds (a fixed time, exactly how long is settable) if the current is several times times what it should be (exactly how much current is settable) and the "Long Time" setting which follows $Fixed_value = [Current]^2 * time ("I squared T").
The "Long Time" setting integrates current squared when ever the current is above the "Pick-up" value which is typically around 20% over normal rated current. Exactly how much the integrated value has to reach to trip on "Long Time" is very complex and has to be coordinated all the other relays and systems. Generally, the lowest level of breakers are given time to trip first, in hopes that the problem is solved while only interrupting a single circuit. The upstream breakers are set with a higher value so they will trip after the downstream breakers had their chance.
Re:The TL;DR (Score:4, Informative)
The protective relay is not at all a part of the circuit breaker. The breaker is a separate device completely, it might not even be in the same cubicle with the protective relay. Also, one protective relay may be commanding several breakers to open on a fault, or it may not actually be commanding a breaker per se, but starting a chain of operations, opening the overloaded breaker, notifying a transfer switch to close tie breakers and go to an alternative power source, etc.
Electrical controls are complex and nuanced, that is why there are professionals to do it. I work in the industrial process control industry, and have programmed my fair share of protective relays, both for switchgear and for motors. (Schweitzer, GE, Square-D/Schneider and ABB specifically.)
Re: (Score:3, Interesting)
You are being pedantic to make yourself feel important. Sure they may not be part of the racked out breaker for high voltage breakers (4160, 13.8kv, and above), but in those cases the relays are associated with the breaker and include it in their designation (51-bkr designation, 86-breaker designation, 27-breaker designation, etc.). The control logic for those breakers will usually be in the breaker cubicle and the relays will usually be mounted on the front or with the control logic. For lower voltage brea
Re: (Score:3)
You are being pedantic to make yourself feel important. ... Why an engineer talking to the general public on Slashdot feels the need to point out the difference is unknown. Perhaps this engineer feels under appreciated?
Talk about projection.
Re:The TL;DR (Score:5, Insightful)
You sir, are a bit of an ass. He is giving relevant and interesting information, which is true. I know, I'm also a process control engineer. The protection relay is quite a complex device (normally approaching the complexity of a small PLC) and very easy to set up wrong.
Re: (Score:3)
But calling it a "protection relay" instead of a "circuit breaker" really is missing the point with pedantry.
As I understand it, such a protection relay is one component tells the switchgear and/or other components what to do, based on loading and other parameters.
The entire system, viewed as a black box, is a circuit breaker. And in practice, the entire system behaves in a manner not dissimilar to the circuit breakers in my own house: Detect fault (either current fault or ground fault or arc fault or som
Re: (Score:3)
You can have a breaker without a protection relay, but not a car without a fuel pump. A fuel pump is often mechanically driven, and a protection relay on it would be silly. Even the electric ones use a simple fuse. A car with a blown fuse isn't necessarily broken (and normally a spare is carried anyway), and can be up and running again in minutes. A breaker will work fine without a protection relay. It simply won't trip on fault conditions which is considered dangerous. I have seen a couple like that. You c
Re: (Score:2)
Of course you can: A simple circuit breaker is just a heater that is mechanically coupled to a switch that locks in the off ("tripped") state: Just like a fuse, but resettable -- no fancy-pants "protection relay" required.
But: If a protection relay exists in this context, it is part of the circuit breaker system. Thus the generalized term "circuit breaker" certainly applies.
"Often?" None of the modern cars I've own
Re: (Score:2)
But: If a protection relay exists in this context, it is part of the circuit breaker system. Thus the generalized term "circuit breaker" certainly applies.
Incorrect. In industry they are definitely separate items. The protection relay is in the panel. The breaker is a removable item that fits into the panel, and normally gets wheeled around a trolly. You clearly have never worked with this stuff. perhaps you should before you talk about it? Why do you insist that the breaker is the panel? I really don't understand how you are confusing the two?
let me help you out here:
This [weiku.com] is what a breaker looks like. Notice the distinct lack of a protection relay.
Re: (Score:2)
"Often?" None of the modern cars I've owned have had a mechanically-driven fuel pump: It is always in the fuel tank (with the fuel), with wires going to it, connected to some manner of switch or relay. (I've owned antique cars with mechanical fuel pumps, but they don't count for any meaningful quantity of "often.")
Diesel. Even modern ones are mechanically driven... Which is over half of vehicles in some places, which I'd call 'often'.
Re: (Score:2)
"Some places" isn't the US. TFA is about the US.
And being in the US, I do see a few diesel cars around every now and then. I do not see them "often."
Re: (Score:2)
Well put, sir.
Re: (Score:2)
I have never seen CTs as part of the breaker assembly. They'll be in the panel, or in the case of some large motors by the motor as well. Admittedly I haven't worked with all the breakers that you can get, but this seems normal so far... (Also for the mods, mod parent up. Nobody worth their salt thinks a protection relay is part of a breaker. It simply is not.)
CYA (Score:4, Insightful)
Re: (Score:2)
"What the hell's your problem? Just shove a penny in it so we can watch the game!"
Re: (Score:2)
Manufacturer of a over current trip relay says if you raise the high current trip point to above the amount of current you were drawing it wouldn't trip?
Say it ain't so!
Re: (Score:2)
I smell BS.
Scenario: this part 'failed', the relay was opened, and the power went out. Subsequently the power was restored.
Must be true: Either the relay was reset or the relay was bypassed.
Known fact: The HID lamps they have take 20 minutes for a full reset cycle, and power was out for 35 minutes. Lights could have come back on as soon as 20 minutes into the outage.
Conclusion: the faulty part was identified and handled just under 35 minutes into the power outage.
This all seems reasonable. But...
Now, a w
Seems like system failures (Score:3)
Are frequently caused by the devices installed to prevent them. Quite ironical.
Re: (Score:3, Interesting)
That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system.
I think the turning point was when journaling file systems came to Linux.
Re: (Score:2)
That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system.
I think the turning point was when journaling file systems came to Linux.
I think it was a turning point when journaling file systems came to be -PERIOD-.
Re: (Score:2)
I've never experienced any problems with my UPSs and they have definitely saved me from some unnecessary shutdowns and reboots. On a side note, I never install the included software.
Re: (Score:2)
So what happens when you have a power outage? :P
Re: (Score:2)
The computer turns off.
The only difference is that it's less graceful that if you have a UPS.
Re: (Score:2)
So you care not for data losses you were working on then. :( Are you sure all UPSes do that? I know one of mine did because it was very old.
Re: (Score:2)
I very rarely have data losses while working on a computer.
Back when I first started out I had the experience of using some quite dodgy software so I got into the habit of saving things quite frequently. It's served me well over the years.
Re: (Score:2)
How about online stuff? That part sucks.
Re: (Score:2)
Likewise, I find that the cheap-ass RAID controllers built into motherboards tend to cause more data loss than just running JBOD and taking your chances. Though nowadays there is software RAID on Linux, which is good enough for a lot of things.
Re: (Score:2)
I'm very sparing with RAID as well. Maybe the balance would tilt in favor of UPS use if I used RAID more often.
Re: (Score:2)
Wow, what kind of UPS's were you using? I don't recall ever having a failure caused by a UPS.
Re: (Score:3)
Have enough of them and the high chance of failure stacks up so you start seeing a few. Have them for long enough and the batteries will die and won't always die gracefully. A frequent battery replacement schedule instead of waiting for a warning could get you around the last one. The largest problem I had was when there were minor power fluctuations that would make lights flicker and make th
Re: (Score:2)
If you're not trying to play big badass games you can build a low-power PC and power it with something like a PicoPSU [mini-box.com], that one will do 160W sustained. Then you can build your own UPS trivially using a car battery and charger. Put the computer against an external wall and run power through it to deal with venting issues. It just sucks to have the power go out when you're in the middle of something. Or, of course, use a cheap inverter. I would have done this already but both my battery chargers are manual, t
Re: (Score:2)
I'm sure you could do something better than what APC uses with such a thing since there is a lot more to it than their monitoring hardware.
Re: (Score:3)
It is possible to buy a UPS for around $40, and all of these items are crap. CyberPower, DirectUPS, Tripp-Lite, Opti-UPS, Minuteman, Powercom...any of these can end up becoming the least reliable component to a Linux system.
Even with APC, who sells reliable units if you spend enough, you're looking at a few hundred dollars for one that is usefully reliable. And even there you have to be careful, do your research, and test to be sure. The biggest issue right now is how power is generated, which determines
Re:Seems like system failures (Score:4, Informative)
True, but there is a failure and then there is a FAILURE. Lights going out... that's an oops. Trunk line overheating and starting a fire during the Superbowl... that's worse. Transformer exploding during Superbowl... that's worse, too. So, yeah, the system failed - and maybe putting the circuit breaker in-line makes a problem more likely. But it almost certainly makes the failure less severe.
Re: (Score:2)
Re: (Score:2)
I guess if your replacing one SPOF with another, it better be more reliable. ;)
Re: (Score:2)
Re: (Score:2)
You need two failures - the box and the protocol/application to have a system failure.
So you are telling me that if your RAID card fails and writes bad data to all 5 discs in your array, they will not have corrupt data because they are a Redundant Array of Inexpensive Discs. You are telling me that if you botch the config of VRRP, your network will not go down, because you must have at least one box fail to have a failure.
An interesting theory, but reality has proven you wrong a few million times.
Re: (Score:2)
Re: (Score:2)
http://en.wiktionary.org/wiki/irony [wiktionary.org]
What you consider irony is not actually irony, as you can see in 4. If we started calling dogs "cats", it wouldn't make them cats, would it?
Re: (Score:2)
Irony includes when the actual is the opposite of the intended. But then these discussions always come down to word nazis like yourself being believers in proscriptive language, not descriptive, and English is officially proscriptive (even if that is an oxymoron).
Superbowl 47? (Score:2, Funny)
Bill Maher (Score:2)
"Just when you thought the NFL couldn't get any blacker"
"God punishes Beyonce for acting like a stripper."
-- Bill Maher
Re: (Score:2)
Is the maintenance engineer related to... (Score:2)
Re: (Score:2)
Re: (Score:2)
Could you elaborate? Was there a lighting incident in 1976 that only you remember?
The audio went out. http://www.youtube.com/watch?v=DrP5ZM0otP8 [youtube.com]
OOO WEE OOO (Score:2)
Alphas (Score:2)
Reminds me of an episode of the Syfy-channel show Alphas
Re:Alphas (Score:4, Funny)
Reminds me of an episode of the Syfy-channel show Alphas
Yeah, Superbowl reminds me of SyFy shows too.
Re: (Score:2)
Working just fine (Score:2)
From TFA
The relay device wasn't put online until December 21. Between then and the Super Bowl, the device functioned properly during three major events -- the New Orleans Bowl, a Saints-Panthers NFL game, and the Sugar Bowl -- Entergy said.
If the device tripped out because the load it saw exceeded its settings, then I'd say that the device was functioning perfectly fine at the Superbowl.
Now were those setting suitable for Beyonce's half time show? That would be the question to ask.
Slow electrons? (Score:2)
Apparently not, I was able to see and hear it 100%.
You do know that the outage occurred after the second half kicked off, right?
Raise your hand if you wanted it to be "Windows" (Score:2)
I'll start by raising mine. Seriously. It's just funny to me now.
Next Year... (Score:4, Funny)
The NFL just announced that next year, the Superbowl will be played at a Motel 6, because they'll leave the lights on for you.
TFA (Score:5, Informative)
You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.
Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.
Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.
The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.
tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.
Re: (Score:2)
Re: (Score:2)
Sensationalist Journalism? (Score:2)
"Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages."
Isn't the point of the protective relay to CAUSE power outages when the load is too high in order to prevent damage to equipment and fires from the line carrying more load than it should.
Re: (Score:2)
Adding complexity does not improve reliability (Score:2)
... in general. It is just the same as with security: If you do not design it in from the beginning, it does not really help and ha s a good change of making things worse.
As it looks like the circuit-breaker type device was configured wrongly, that holds even more: The more components that need a setting, the higher the probability somebody messes up or has his/her capability for understanding how the system works exceeded.
Re: (Score:2)
But if you are competent and get the settings right, those relays can prevent huge amounts of damage. Well worth it. I know, I work with them, and if I did not have such devices the one site I worked at would have burned down when the capacitor banks faulted.
Re: (Score:2)
I don't dispute that. I am merely saying that they need to be part of the initial design to be most effective and that later additions need to be done very, very carefully.
No good info yet (Score:2)
Even EL&P's coverage [elp.com] doesn't say anything substantive. When we start seeing articles with diagrams of the feeders, maybe we'll know something.
An enclosed stadium in Louisiana in winter shouldn't be anywhere near its electrical load peak. No air conditioning load.
Re: Explanation (Score:2)
Re: (Score:2)
yeah, some people thought it was suspicious because stopping Baltimore's momentum helped enable that San Francisco rally/near-comeback.
Re: (Score:2)
As a technical troubleshooter I can not believe how they try to say it was not caused by the grand surge of electricity needed.
Maybe the device saw the usage as abnormal and just did it's job a little bit late.
Just admit it, you didn't think it would draw that much amps/be a problem with stage over-hype.
First thing I did after seeing it was check solar activity. Once I saw that it was nominal and not CLOSE to a major power fluctuation event, I determined it was caused by Human Error. Don't fucking care what the error was - people will fix broken shit. It was just an error.
Re: (Score:2)
BS
Re: (Score:2)
Commentators talked about the momentum shift, and then that happened.
Yeah, if the spread was Baltimore by 3 or less, this would affect the betting outcome but not the game outcome, which makes sense as a lower-impact way to 'fix' it. However, most spreads I saw were San Francisco by 4. Maybe the attempt to 'fix' it went wrong, maybe it was about some side bet.
Re: (Score:2)
*if the spread was Baltimore by 3 or more*
Re: (Score:2)
I don't think Alanis Morisette nor too many of her fans read slashdot, but thanks anyway. :)
Re: (Score:2)
I like Jewel's music.
I like Alanis' music.
People will judge, they always do.
Both artists write good lyrics, and have some pretty infamously bad examples of using the wrong word at the wrong time.
So, to set the record straight:
Alanis: Sang "Isn't it ironic?" when, in fact, few, if any of the scenarios she mentioned were irony (implied meaning in opposition to the literal meaning)
Jewel: Sang "...with such casualty", when she actually mean casualness (i.e. not death, but with little re