How an Electrical Fire Shut Down Heathrow and Upended Global Air Travel (msn.com) 54
London's Heathrow Airport resumed operations late Friday after an electrical fire at a nearby substation forced a full-day closure, causing global travel chaos with hundreds of canceled flights and thousands of stranded passengers. The explosion at a Hayes substation 1.5 miles from the airport knocked out power early Thursday, requiring 70 firefighters to battle a blaze in a transformer containing 25,000 liters of cooling oil.
Despite backup generators, Europe's busiest airport couldn't maintain normal operations, forcing flights to divert to airports across Europe and as far as Bangor, Maine. "Contingencies of certain sizes we cannot guard ourselves against 100%," Heathrow CEO Thomas Woldbye told the BBC. "This is as big as it gets for our airport." British Airways, which planned to carry 100,000 passengers Friday, prioritized long-haul flights to Australia, Brazil and South Africa when operations resumed after 4 p.m.
Despite backup generators, Europe's busiest airport couldn't maintain normal operations, forcing flights to divert to airports across Europe and as far as Bangor, Maine. "Contingencies of certain sizes we cannot guard ourselves against 100%," Heathrow CEO Thomas Woldbye told the BBC. "This is as big as it gets for our airport." British Airways, which planned to carry 100,000 passengers Friday, prioritized long-haul flights to Australia, Brazil and South Africa when operations resumed after 4 p.m.
Collective action problem/low probability event. (Score:4, Insightful)
Is it in anyone's interests to have Heathrow go down and flights delayed all over the place? No. (more specifically 'no' among all airlines and national transport types; there's obviously a free-floating group of random extremists and foreign policy opponents of the UK who are in favor; but nobody expected them to pay for infrastructure anyway)
Is it in anyone's interests to toss their own cash into the pot when the power supply to Heathrow hasn't failed in years and, anyway, the backup generators are probably fine, no need to check what those whiners in facilities are saying? Also no.
It's perennially difficult to find someone to do preventative spending for fairly low risk situations; sure, sell me on paying two or 3 times more for electrical substations when the one we have has worked just fine for years; and it's even harder when the benefits of reliability are distributed among a fairly large number of entities whose day-to-day incentives are to lower costs to the degree possible: Having to cut Heathrow out of their plans suddenly and without notice likely cost the various airlines a small to midsize fortune; but, on any average day, every cent of 'cost recovery' that goes to Heathrow rather than to them is margin stripped away; and it's not like BA wants to get stuck paying more for the 'B' in their name when their competitors are paying less; so there's no obvious party on the hook.
Re:Collective action problem/low probability event (Score:5, Informative)
That's not how power systems or outages work. Just because your lights are off and your neighbours are on doesn't mean there was any extra money spent on them. It just means it wasn't their feeder which was affected.
Re: (Score:2)
Rather than beefing up and hardening existing electrical infrastructure which still has a single point of failure, I think it would be wise to look at redundant distributed electrical infrastructure. This would be more resilient. (Think of the internet)
And, no, backup emergency generators are not the solution. They do have those and they were totally inadequate. They only were capable of powering emergency lights, etc. and not the whole airport.
Re: Collective action problem/low probability even (Score:2)
The evidence of management failure is backup generators not able to handle the load. Observe the peak load, make sure you have backups that can do that. Barring willful sabotage or similar somebody should be on the hook somewhere.
Re: (Score:2)
The fire knocked out the backups and the grid supply from one of the redundant substations. This wasn't the problem.
Re: (Score:3)
Re: Collective action problem/low probability eve (Score:2)
Yeah, I was just reading deeper into it and about what philip2 was saying. I guess it really is a higher level grid design thing, but when you see the consequences are so severe, it is clear there really should be a distributed backup in place for things that can just happen, like fires. Totally possible we are just as vulnerable in US with crumbling infrastructure but we should not be.
Re: Collective action problem/low probability eve (Score:5, Insightful)
And airports get closed down due to weather all the time anyway, and diversions to alternative airports is a normal procedure. So the question is, do you want to introduce additional risks and spend a lot of money to prevent a day of downtime maybe once every couple of decades?
Re: Collective action problem/low probability eve (Score:2)
The question is how much did this day of downtime cost the airlines? Easily in the tens of millions of dollars, with used 7 megawatt generators going for a few million, I bet an entire solution would cost 80 million. It really is not trivial when infrastructure like this goes down. Really, if they were able to redirect power from a residential area so 40,000 homes lost power it would just be local news.
Re: (Score:2)
The point really is to think about what the solution is. A substation went down. So, have redundant power substations? Well that had that. And a result of which, following the fire, they were able to restore power, running on the two substations that remain.
The problem is that when the power went down, their systems shutdown and rerouting took time. So, what would be needed is redundancy that can switch in milliseconds, or you will have to reboot. And even then, what happens if the backups get taken out. Th
Re: (Score:1)
In principle: yes, you want.
Russia shoots 100 hypersonic missiles, at London. Prime targets airports and power infrastructure, perhaps a random railway station.
Power gone ... takes a while to fix stuff. Not just "a single day".
With back up power, probably mobile, you get the airports back online at least to the level that radio and air traffic control - radar - works again. Remove the debris, if the runways are okay, you still can operate. At least resilient crafts like Hercules can land and take off ... he
Re: (Score:2)
The whole system is timed off how fast one can get the people and baggage on and off the airplane. Without luggage handling conveyors it starts looking like 1952 again. You remove the air conditioning from the people and they pass
Re: Collective action problem/low probability eve (Score:4, Informative)
but when you see the consequences are so severe, it is clear there really should be a distributed backup
Well two things here. Heathrow has redundant systems + backup systems, but distribution is never easy. At the HV side of power supply systems are already setup with N-1 redundancy or sometimes even N-2. They are often setup in rings allowing not just loss of a feeder but also loss of a link between equipment, and generators can only do so much.
But the bigger question is how "severe" are the consequences. The airport was unable to process passengers for 18 hours causing a couple of hundred flights to be cancelled. That's not severe. That's about the equivalent to a bad storm for which contingencies already need to be in place. It sucks for the people affected by a day but ultimately this was not that major of an event as people would make it out.
On the flip side an oil refinery I consulted at definitely does have dual feeders, geographically redundant. A power outage for them *is* severe, not only does it take multiple weeks to recover operations, but there is a very high potential to kill someone as the sudden loss of systems there is incredibly unsafe. The safety aspect at Heathrow is already covered. Air traffic control at no point went down so it was little more than a pain for flights in progress to be diverted.
Re: (Score:2)
Shouldn't that be N+1?
I worked on a call center / data center for a credit card company that required two feeds from the grid. Unfortunately, the electric company ran both feeds from the same substation, so that eliminated a lot of the redundancy they were after. It's not easy or cheap getting all that infrastructure in place.
Re: (Score:3)
Shouldn't that be N+1?
Believe it or not in power systems engineering it is often referred to backwards. N-1 means full load is operational should one circuit path drop out.
I worked on a call center / data center for a credit card company that required two feeds from the grid. Unfortunately, the electric company ran both feeds from the same substation, so that eliminated a lot of the redundancy they were after. It's not easy or cheap getting all that infrastructure in place.
For the most part that is standard even in critical applications. It is vanishingly rare for a customer to be connected to two geographically diverse substations. Even the aforementioned refinery I worked at is connected to two different energy companies to achieve that. You virtually never assume the loss of a building. You typically assume the loss of a swit
Re: (Score:2)
Stupidest problem I encountered with emergency generators was in a place with known electrical grid reliability problems - the electric company even offered a discounted rate if the financial services company would go with interruptible power so they could be disconnected in case of impending brownout conditions, but they decided against it. Still had 100% backup generators which they dutifully tested every week. They wouldn't test live because the sensitiv
Re: (Score:2)
A fire in a substation is a low probability event. Even a substation getting damaged only happens rarely - maybe once a year somewhere in the world (often in places with more sketchy power grids).
In the US, you had two substations get shot at by robbers trying to rob a jewellery store. That's quite a rare event, especially when it happened twice in a year. Hasn't happened since.
And those are intentional acts. Try an accidental act like a fire - power utilities are doing tons of things to protect these asset
Re: (Score:3)
Having worked on new emergency generators for an airport comparable in traffic to Heathrow, that is not how they were sized, since it would be economically infeasible. An estimate of life safety loads, minimum loads necessary to stay open, and good-to-have loads were made and the generators sized for economic considerations. Connected load (as opposed to estimated or measured load) was more than the generating capacity and automated trans
Re: (Score:1)
You don't have real redundancy if you can't play Chaos Monkey with the power. Randomly shut down parts of the system. If anyone notices, someone doesn't get their executive bonus. No problems.
Anything less than that and it's lip service.
Sounds right to me (Score:2)
Not sure why you got downvoted; NOT deserved.
Get the manager to sign off as to their belief that the system is resilient. Then find out if they were lying...
Re: (Score:1)
What is the point to have a manager sign "something off" when he has no clue about it?
So, you have one you easily can sue later? What is the point?
The engineers have to sign it off. And the parties installing it. And the manager has to sign: "yes, as far as I can tell, everyone involved "signed off" that this system is okay".
More a manager can not do.
Engineers can be managers, you know ;) (Score:2)
I was assuming that the 'manager' was a technically competent senior engineer who has done the planning and contingency analysis etc etc.
Re: (Score:2)
They got downvoted because randomly switching of single things would have shown that Heathrow does in fact have perfectly redundant power. It's precisely this kind of thinking that lead to this situation.
There are far more aspects to redundancy than simply turning things off one at a time. Fires rarely turn one thing off. But then fires are also incredibly rare events and it makes you wonder if it's worth building a geographically diverse distribution system for an airport all so a couple of hundred flights
Good response, thank you (Score:2)
Of course what's needed is something rather more thorough than 'randomly switching off things', but at least doing that would encourage some thought. In this case the electrical supply to the whole site was dependent on a single substation NOT blowing up. That's a single point of failure that should not be, and additional supply wires to different substations could address.
Re: (Score:2)
Not sure why you got downvoted; NOT deserved.
Not sure why you think they got downvoted.
Re: (Score:2)
What gets me is that this is critical national infrastructure - it cannot fail through one fault, or have the redundancy located such that a single failure can take that out at the same time, like backing up your data to another partition on the same hard drive.
The terminals that went dark AFAIK use power in the 20-30MW range which you can fit on a standby basis in a handful of shipping containers onsite at a cost that would be a rounding error compared with BAA turnover and profits. I can see a new 3MW one
Re: Collective action problem/low probability even (Score:1)
The “Busiest” Bragging Problem (Score:1)
A power station failure may have shut down a single airport on this planet, but what “upended global travel” was the stupidity and arrogance that insists all we need to support millions of flying travelers in one particular spot via a proverbial two-lane road we refuse to expand.
If we can build an international space station, I’m pretty sure we can figure out how to build another fucking airport. Build the fucking thing. Build 10 more of the fucking things if we need to. Truly incredible
Re: The “Busiest” Bragging Problem (Score:1)
Indeed - why was this downvoted? (Score:2)
Indeed London has FIVE airports. Unfortunately the persistence of the hub and spokes model of airline scheduling means that a single is usually critical. This is despite the rise in popularity of air travel meaning that direct links between slightly smaller centres makes at least as much sense.
Re: (Score:1)
I take it that you never heard of City, Stansted or Gatwick airports?
Sure I have. They’re all as old or older (pre-WWII) as Heathrow, which begs the obvious question to fix an obvious bottleneck; What have you done for me lately. Expanding the proverbial two-lane road into a four-lane road via construction corruption doesn’t help much when Demand needed an 8-lane superhighway two decades ago.
Build the fucking thing. Don’t band-aid the old thing and pretend you’re keeping up while polishing that Busiest trophy. Same goes for Atlanta, DFW, Dubai, T
Re: (Score:2)
Build the fucking thing.
That's what they've been doing since 1946. Heathrow isn't so much an airport as a construction site with runways and charabancs carrying people from one aircraft to another.
Re: (Score:2)
City isn't pre WW2, it's from the 1980s.
London is polluted and noisy enough already. We don't want to be in the hook for the rest of the UKs "growth" by paying for it with our well-being.
Re: (Score:1)
Could they not just make a kind of underground air port?
Some big hole on one side of the city, the planes fly in, park under a subway station - they call those "underground, too, right? - let the people in and out. And then just fly out on the other hole on the other side of the city.
A little bit like a mix of Harry Potter and Lords of the Rings. But with technology, not with magic ...
Of course, for the pilots it would be scary ... just that damn big black hole and you have to fly into the middle of it!
Re: (Score:2)
If they proposed it, I would scrap all my objections just on the grounds of general awesomeness.
And yes we do call our underground stations underground stations :)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
via a proverbial two-lane road we refuse to expand
What the fuck are you talking about? London has 3 other commercial airports, and there are 4 airports just marginally smaller than Heathrow in spitting distance which flights could divert to. Passengers lucky enough to land in Paris were in London before Heathrow's power was even on again.
I’m pretty sure we can figure out how to build another fucking airport.
Building an airport is easy. Building it where you need it is difficult. Connecting it to where people want to go is difficult. Building it in a city of 9 million people destroying an entire community in the process is dif
Re: The “Busiest” Bragging Problem (Score:1)
UK, call me if you need another problem solved, or just post something here. I will be happy to help. Cheers!
Yours faithfully,
Some random dude on the internet
Re: (Score:3)
It's a terrible place for an airport: you'll keep getting horses on the runway and that won't do anyone any good.
Think, man, think!
Re: The “Busiest” Bragging Problem (Score:3)
Now excuse me, I just read about cleaning your ears with hydrogen peroxide. I want to try that. I've had an itch in my left ear for days and maybe this will help.
No doubt the "vibrant" folks ... (Score:1)
DRP, GENERATORS, AND INCOMPETENCE, Oh My. (Score:2)
The substation explosion fire reasons are not given, but that would be a good lede to any news reporting. WHY DID THE SUBSTATION GO OFFLINE?
The "BACKUP" generators were inadequate to address the power needs ("demand") at the time. This means it was neither REDUNDANT nor CONTINGENT, and certainly not N or N+1 or even 2N. (Just google "backup generator redundancy" if you care about the details).
So for some period of years, these generators were a)acquired, b)connected -- likely through automatic transfer s
Re: (Score:2)
The "BACKUP" generators were inadequate to address the power needs ("demand") at the time. This means it was neither REDUNDANT nor CONTINGENT, and certainly not N or N+1 or even 2N. (Just google "backup generator redundancy" if you care about the details).
Which is not properly true; Heathrow did have backup generators sufficient to address their power needs -- unfortunately, 'did' in this case is used in the longer historical sense, not that had been available at the time of the fire. As part of a move to become more Net Zero compliant, Heathrow got rid of their diesel backup generators, replacing them with a biomass-powered generator that was intended to supplement -- not replace -- the mains power. The statement on GBnews by Richard Tice MP [x.com] reveals this fo
Shouldn't use wet transformers there (Score:1)
Dry transformers should be used anywhere infrastructure-critical so that this problem doesn't exist.
Yeah, they cost more up front, but the maintenance cost is lower, plus when something like this happens it's very very expensive.
Re: (Score:2)
The transformer that burned in this case appears to be in a substation operating at 132 kV or thereabouts. It doesn't appear to be a 275 yard, but I've just looked at pictures. Definitely not a 66 kV yard so dry-type wouldn't be an option in this application.
Re: (Score:2)
So they need to use two or three of 'em to avoid having an outage affecting how many people?
Of course the usual actors daily fail etc (Score:2)
net-zero (Score:1)
A MP tells why the backup failed:
https://x.com/GBNEWS/status/19... [x.com]