typodupeerror

## Super Bowl Blackout Caused By Defective Protective Relay210

Posted by timothy
from the ravens-can-see-at-night dept.
New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."
This discussion has been archived. No new comments can be posted.

## Super Bowl Blackout Caused By Defective Protective Relay

• #### Did someone lost his job? (Score:2)

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?
• #### Re:Did someone lost his job? (Score:5, Funny)

on Saturday February 09, 2013 @04:57PM (#42845623) Homepage Journal

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

• #### Re:Did someone lost his job? (Score:5, Funny)

on Saturday February 09, 2013 @05:33PM (#42845839)

Yup, that's the way it goes in some parallel universe :)

• #### Re: (Score:2)

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

They probably get a higher end-of-year bonus for resolving the problem.
Upper level compensation works in mysterious ways - plenty of CEOs manage to get guaranteed bonuses that are completely detached from company performance.

• #### Re: (Score:3, Funny)

No, they are not detached from company performance.

If the company performs well, the bonus becomes astronomical. If it performs less well, the bonus is merely unbelievable.

To regular people, it appears to be detached from reality.

• #### Re: (Score:2)

We watch companies fail, then get bailed out by the government, after which CEO's still get astronomical bonuses.

Yes, the bonus system is detached from reality. Any CEO whose company fails as a result of his actions, inactions, and poor judgement calls should be sacked, then sued for dereliction of duty. Said CEO should never again have access to resources which he hasn't earned, himself. Let him push a frigging broom and mop for the rest of his life.

Those astronomical bonuses to which you refer have no

• #### Re: (Score:2)

I suggest you get someone to read davester666's post to you, and then explain it. Someone from England would be a good choice.

• #### Re: (Score:2)

I suggest you read Davestar's post, as well as my own again. I've disagreed with Davestar. He claims that the failures who ran banks into the dirt didn't get astronomical bonuses, merely unbelievable bonuses. I stated that they did indeed get astronomical bonuses.

I have little idea which language might be your own native language, but respectful disagreement seems pretty obvious to me here. Davestar basically split a hair, and I told him that execs always get the bigger half of the hair.

• #### Who gets fired vs. who gets bonuses (Score:2)

re: Presumably the person that receives the big end-of-year bonus when everything goes well?
.
hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels, so the firing most often will happen to those at the lower level who executed the commands, including putting in crappy materials that were ordered when the higher-ups want to save money. At least that seems to be the way of the USA; Japan's older way would have those responsible all the way
• #### Re: (Score:2)

re: Presumably the person that receives the big end-of-year bonus when everything goes well?
.
hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels

Well thanks for pointing that out, Miss tic-tac tits.

If you hadn't been here we'd have been grossly misinformed about how capitalism works! Shame on you, isorox, for lying to us so!

• #### It was a fail safe (Score:4, Informative)

on Saturday February 09, 2013 @04:52PM (#42845595) Homepage
Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.
• #### Re: (Score:2)

It was too many people tweeting at the same time.

• #### Re:It was a fail safe (Score:5, Informative)

on Saturday February 09, 2013 @07:48PM (#42846691) Homepage

It apparently did it's job. But apparently it was given the wrong job. It is accused (by the manufacturer, of course) that someone entered the wrong amperage that it should do its job at. Unlike home circuit breakers which come in specific amperage levels (and vary from unit to unit by plus or minus 10 percent which is considered acceptable), these relay devices, which are a component in an overcurrent protection system, cannot be made at fixed amperage levels due to economics. They are quite expensive to replace with another just to tweak the settings due to changes made elsewhere in the power distribution network, and the number of different amperage values needed would be very large. They can be expensive also because either they directly connect to current transformers that have high open circuit voltage potential, or operate from digital sensors on the current transformers. They are also expected to have accurate at better than one percent.

• #### Re: (Score:2)

Wasn't there a practice run done? Really, if you're responsible for a multi-billion dollar event, it might be a good idea to fire everything up a few times and make sure it behaves as intended.

• #### Re: (Score:2)

It probably doesn't help that electricity, especially at these levels, is a tricky thing to begin with. Even in the article, they mention they're not entirely sure why the relays fail.
• #### Re: (Score:3)

Just to add to this, the term "circuit breaker" refers to a switch that can open (interrupt) and close a circuit under load without arcing or damage. They are typically designed with springs or gas charges to very rapidly open the contacts without arcing or have channels around the contacts called arc chutes that extinguish arcs (older DC breakers had "blow out" coils, when opened the arc would be magnetically pushed out until it extinguished) . "Over current protection" refers to a device that guards again

• #### The TL;DR (Score:3, Informative)

by Anonymous Coward on Saturday February 09, 2013 @04:53PM (#42845601)

Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).

• #### Re:The TL;DR (Score:5, Interesting)

on Saturday February 09, 2013 @05:31PM (#42845821)
Circuit breaker was not "miscalibrated". The protective relay (which is separate from a breaker) possibly had a setting in it that was too low. Protective relay settings are based on time curves (which are plotted on logarithmic paper). For, say, 300 amps, it trips after 10s or 100s of seconds of continuous operation past the setting. For 10,000 amps, it may after .03 seconds (or you may have an instantaneous setting, or a definite time delay based on cycles). That kind of curve. If the load was drawing so much current, for so long of time, then yes, it will send a command for the circuit breaker to trip. Anyhow, it's easy to screw up a protective relay setting - and yes, I've done it. That's why relay settings are always checked by a second engineer as well, just to make sure you didn't miss something. IAAPE (I am a protection engineer, and a P.E.), though we don't use S&C relays (Schweitzer here).
• #### Re: (Score:2)

Q for you: is the circuit breaker actually set for $Fixed_value = Current \times Time$ or for a thermal setting which just happens to almost be equivalent to (current draw)x(time drawn) ? Of course I do realize that since the current draw won't be continuously constant, my question simplified what should be the integral of current draw over a running window of time being over a threshold into just the product of time and current draw, but you get the idea...
.
I know that a fuse is thermally activated with
• #### Re:The TL;DR (Score:4, Informative)

on Sunday February 10, 2013 @12:21AM (#42848005)

In my experience, most relays have a "Instantaneous" setting that goes off as fast as possible if you have like 20-30 times as much current as should be there, a "Short Time" setting that goes off in few seconds (a fixed time, exactly how long is settable) if the current is several times times what it should be (exactly how much current is settable) and the "Long Time" setting which follows $Fixed_value = [Current]^2 * time ("I squared T"). The "Long Time" setting integrates current squared when ever the current is above the "Pick-up" value which is typically around 20% over normal rated current. Exactly how much the integrated value has to reach to trip on "Long Time" is very complex and has to be coordinated all the other relays and systems. Generally, the lowest level of breakers are given time to trip first, in hopes that the problem is solved while only interrupting a single circuit. The upstream breakers are set with a higher value so they will trip after the downstream breakers had their chance. • #### CYA (Score:4, Insightful) on Saturday February 09, 2013 @04:57PM (#42845621) Yet, the manufacturer of the trip relay says "Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power..." [cnn.com] Based on Entergy's incorrect initial claims that "it wasn't us," I tend to think they're not being honest. • #### Re: (Score:2) "What the hell's your problem? Just shove a penny in it so we can watch the game!" • #### Re: (Score:2) Manufacturer of a over current trip relay says if you raise the high current trip point to above the amount of current you were drawing it wouldn't trip? Say it ain't so! • #### Re: (Score:2) I smell BS. Scenario: this part 'failed', the relay was opened, and the power went out. Subsequently the power was restored. Must be true: Either the relay was reset or the relay was bypassed. Known fact: The HID lamps they have take 20 minutes for a full reset cycle, and power was out for 35 minutes. Lights could have come back on as soon as 20 minutes into the outage. Conclusion: the faulty part was identified and handled just under 35 minutes into the power outage. This all seems reasonable. But... Now, a w • #### Seems like system failures (Score:3) on Saturday February 09, 2013 @05:06PM (#42845675) Are frequently caused by the devices installed to prevent them. Quite ironical. • #### Re: (Score:3, Interesting) That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system. I think the turning point was when journaling file systems came to Linux. • #### Re: (Score:2) That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system. I think the turning point was when journaling file systems came to Linux. I think it was a turning point when journaling file systems came to be -PERIOD-. • #### Re: (Score:2) I've never experienced any problems with my UPSs and they have definitely saved me from some unnecessary shutdowns and reboots. On a side note, I never install the included software. • #### Re: (Score:2) So what happens when you have a power outage? :P • #### Re: (Score:2) The computer turns off. The only difference is that it's less graceful that if you have a UPS. • #### Re: (Score:2) So you care not for data losses you were working on then. :( Are you sure all UPSes do that? I know one of mine did because it was very old. • #### Re: (Score:2) I very rarely have data losses while working on a computer. Back when I first started out I had the experience of using some quite dodgy software so I got into the habit of saving things quite frequently. It's served me well over the years. • #### Re: (Score:2) How about online stuff? That part sucks. • #### Re: (Score:2) Likewise, I find that the cheap-ass RAID controllers built into motherboards tend to cause more data loss than just running JBOD and taking your chances. Though nowadays there is software RAID on Linux, which is good enough for a lot of things. • #### Re: (Score:2) I'm very sparing with RAID as well. Maybe the balance would tilt in favor of UPS use if I used RAID more often. • #### Re: (Score:2) Wow, what kind of UPS's were you using? I don't recall ever having a failure caused by a UPS. • #### Re: (Score:3) I've had plenty with APC and plenty more with a cheap and nasty Chinese brand that has probably given up in shame. Have enough of them and the high chance of failure stacks up so you start seeing a few. Have them for long enough and the batteries will die and won't always die gracefully. A frequent battery replacement schedule instead of waiting for a warning could get you around the last one. The largest problem I had was when there were minor power fluctuations that would make lights flicker and make th • #### Re: (Score:2) If you're not trying to play big badass games you can build a low-power PC and power it with something like a PicoPSU [mini-box.com], that one will do 160W sustained. Then you can build your own UPS trivially using a car battery and charger. Put the computer against an external wall and run power through it to deal with venting issues. It just sucks to have the power go out when you're in the middle of something. Or, of course, use a cheap inverter. I would have done this already but both my battery chargers are manual, t • #### Re: (Score:2) I suppose it would be fairly easy to build a circuit to do monitoring and to switch the charger using an Arduino or MSP I'm sure you could do something better than what APC uses with such a thing since there is a lot more to it than their monitoring hardware. • #### Re: (Score:3) It is possible to buy a UPS for around$40, and all of these items are crap. CyberPower, DirectUPS, Tripp-Lite, Opti-UPS, Minuteman, Powercom...any of these can end up becoming the least reliable component to a Linux system.

Even with APC, who sells reliable units if you spend enough, you're looking at a few hundred dollars for one that is usefully reliable. And even there you have to be careful, do your research, and test to be sure. The biggest issue right now is how power is generated, which determines

• #### Re:Seems like system failures (Score:4, Informative)

on Saturday February 09, 2013 @05:25PM (#42845795)

True, but there is a failure and then there is a FAILURE. Lights going out... that's an oops. Trunk line overheating and starting a fire during the Superbowl... that's worse. Transformer exploding during Superbowl... that's worse, too. So, yeah, the system failed - and maybe putting the circuit breaker in-line makes a problem more likely. But it almost certainly makes the failure less severe.

• #### Re: (Score:2)

I've had that argument more times than I can count. Getting one resilient device is more reliable than two devices. Because the cluster/HSRP fails more often than the hardware does.
• #### Re: (Score:2)

I guess if your replacing one SPOF with another, it better be more reliable. ;)

• #### Re: (Score:2)

You can never eliminate a single point of failure. You can move it, but there's almost always a single point of falure left, usually the protocol/application that combines a box that's a single point of failure into a cluster that's a single point of failure. But as long as the lines are meshed, people are happy. Actual uptime is irrelevant.
• #### Superbowl 47? (Score:2, Funny)

In combat sports where people get hurt for the amusement of spectators, we use Roman numerals.
• #### Bill Maher (Score:2)

"Just when you thought the NFL couldn't get any blacker"

"God punishes Beyonce for acting like a stripper."

-- Bill Maher

• #### Re: (Score:2)

Bull shit! Bill Maher doesn't believe in God.

• #### Is the maintenance engineer related to... (Score:2)

There was this engineer who was in charge of the production studio that did the live broadcast of the presidential debate between Jimmy Carter and Gerald Ford. I wonder if this stadium maintenance engineer is that guy's son or something...
• #### Re: (Score:2)

Could you elaborate? Was there a lighting incident in 1976 that only you remember?
• #### OOO WEE OOO (Score:2)

It was not any piece of technology that caused that outage, it was vengeful spirits! [uncoveror.com]
• #### Alphas (Score:2)

Reminds me of an episode of the Syfy-channel show Alphas

• #### Re:Alphas (Score:4, Funny)

<mitreya.gmail@com> on Saturday February 09, 2013 @05:52PM (#42845961)

Reminds me of an episode of the Syfy-channel show Alphas

Yeah, Superbowl reminds me of SyFy shows too.

• #### Not a Football fan or Watcher (Score:2)

But I'll believe it when I see a well written, technical-level RCA.

• #### Working just fine (Score:2)

From TFA

The relay device wasn't put online until December 21. Between then and the Super Bowl, the device functioned properly during three major events -- the New Orleans Bowl, a Saints-Panthers NFL game, and the Sugar Bowl -- Entergy said.

If the device tripped out because the load it saw exceeded its settings, then I'd say that the device was functioning perfectly fine at the Superbowl.

Now were those setting suitable for Beyonce's half time show? That would be the question to ask.

• #### Slow electrons? (Score:2)

Now were those setting suitable for Beyonce's half time show? That would be the question to ask.

Apparently not, I was able to see and hear it 100%.

You do know that the outage occurred after the second half kicked off, right?

• #### Raise your hand if you wanted it to be "Windows" (Score:2)

I'll start by raising mine. Seriously. It's just funny to me now.

• #### Next Year... (Score:4, Funny)

by Anonymous Coward on Saturday February 09, 2013 @06:45PM (#42846265)

The NFL just announced that next year, the Superbowl will be played at a Motel 6, because they'll leave the lights on for you.

• #### TFA (Score:5, Informative)

by Anonymous Coward on Saturday February 09, 2013 @07:11PM (#42846471)

You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.

Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.

Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.

The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.

• #### Re: (Score:2)

That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

When its an event with that kind of money involved, you bet they get called in to explain it. Power outages are just not common in the U.S. and are normally only due to human error, or acts of nature. Devices that can fail should be introduced into systems in ways that provide redundancy. This is done for the sake of continued up-time, and safety. Not having sufficient redundancy can easily be referred to as human error. Saying that the task is "hard work" is an unacceptable excuse. If the system design was

• #### Sensationalist Journalism? (Score:2)

"Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages."

Isn't the point of the protective relay to CAUSE power outages when the load is too high in order to prevent damage to equipment and fires from the line carrying more load than it should.

• #### Dear (Score:2)

Mr. Leonard, Its me, jim. you might not know me very well, im not exactly C-level so i never really met you. I just wanted to ask, hows that Relay sub-subcontractor thing working out for you? Me? oh ive been pretty successful since the termination with my own engineering consulting firm. We work on switchgears, relays, you name it!

Hope your team won the superbowl, Jim Ex master relay engineer, Entergy INC Local Union affiliated.
• #### Adding complexity does not improve reliability (Score:2)

... in general. It is just the same as with security: If you do not design it in from the beginning, it does not really help and ha s a good change of making things worse.

As it looks like the circuit-breaker type device was configured wrongly, that holds even more: The more components that need a setting, the higher the probability somebody messes up or has his/her capability for understanding how the system works exceeded.

• #### Re: (Score:2)

But if you are competent and get the settings right, those relays can prevent huge amounts of damage. Well worth it. I know, I work with them, and if I did not have such devices the one site I worked at would have burned down when the capacitor banks faulted.

• #### Re: (Score:2)

I don't dispute that. I am merely saying that they need to be part of the initial design to be most effective and that later additions need to be done very, very carefully.

• #### No good info yet (Score:2)

Even EL&P's coverage [elp.com] doesn't say anything substantive. When we start seeing articles with diagrams of the feeders, maybe we'll know something.

An enclosed stadium in Louisiana in winter shouldn't be anywhere near its electrical load peak. No air conditioning load.

#### Related LinksTop of the: day, week, month.

The person who's taking you to lunch has no intention of paying.

Working...