Ten Mistakes Marred Firewall Upgrade At Australian Telco, Contributing To Two Deaths (theregister.com) 30
An independent review found that at least ten technical and process failures during a routine firewall upgrade at Australia's Optus prevented emergency calls from reaching Triple Zero for 14 hours, during which 455 calls failed and two callers died. The Register reports: On Thursday, Optus published an independent report (PDF) on the matter written by Dr Kerry Schott, an Australian executive who has held senior management roles at many of the country's most significant businesses. The report found that Optus planned 18 firewall upgrades and had executed 15 without incident. But on the 16th upgrade, Optus issued incorrect instructions to its outsourced provider Nokia. [...] Schott summarized the incident as follows: "Three issues are clear during this incident. The first is the very poor management and performance within [Optus] Networks and their contractor, Nokia. Process was not followed, and incorrect procedures were selected. Checks were inadequate, controls avoided and alerts given insufficient attention. There appeared to be reticence in seeking more experienced advice within Networks and a focus on speed and getting the task done, rather than an emphasis on doing things properly."
The review also found that Optus' call center didn't appreciate it could be "the first alert channel for Triple Zero difficulties." The document also notes that Australian telcos try to route 000 calls during outages, but that doing so is not easy and is made harder by the fact that different smartphones behave in different ways. Optus does warn customers if their devices have not been tested for their ability to connect to 000, and maintains a list of known bad devices. But the report notes Optus's process "does not capture so-called 'grey' devices that have been bought online or overseas and may not be compliant." "To have a standard firewall upgrade go so badly is inexcusable," the document states. "Execution was poor and seemed more focussed on getting things done than on being right. Supervision of both network staff and Nokia must be more disciplined to get things right."
The review also found that Optus' call center didn't appreciate it could be "the first alert channel for Triple Zero difficulties." The document also notes that Australian telcos try to route 000 calls during outages, but that doing so is not easy and is made harder by the fact that different smartphones behave in different ways. Optus does warn customers if their devices have not been tested for their ability to connect to 000, and maintains a list of known bad devices. But the report notes Optus's process "does not capture so-called 'grey' devices that have been bought online or overseas and may not be compliant." "To have a standard firewall upgrade go so badly is inexcusable," the document states. "Execution was poor and seemed more focussed on getting things done than on being right. Supervision of both network staff and Nokia must be more disciplined to get things right."
what about stop useing so meny contractors and sub (Score:2)
what about stop useing so meny contractors and subcontractors?
Re: (Score:2)
Optus said they could do it cheaper and better than Telstra.
Follow the money.
Re: (Score:3)
Companies want to save money and decide to employ operators rather than engineers. Then when they need to contract "experts" for projects they go with the lowest bidder. What do they expect to happen? You get what you pay for.... sometimes less.
Liability (Score:2)
So is someone gonna get sued for those two deaths?
Re: (Score:1)
Re: (Score:2)
Contributing to death and proving liability are two very different things with very different legal bars. To prove liability you will need to show beyond reasonable doubt that the call going through would have lead to survival.
455 calls failed and two callers died (Score:1)
Two dead out of 455. That's just a bad afternoon at the Brontosaurus Steakhouse in Omaha, on Wednesday, when the fried ice cream cutlets go two for one.
An incidence of 0.0043956.
Or for you, not exactly mathletes out there, less than 1/2 of 1%.
Clearly, AI is the new cholesterol.
Re: (Score:2, Insightful)
It's certainly a testament to how many of those calls didn't need to happen, and I'm not seeing here that we're sure those people would have made it if the calls had gone through, or they'd even have had a good chance.
On the flip side, if anyone certainly died because of negligence around a critical emergency service, some heads should roll. Unfortunately, it won't be the ones which it needs to in order to prevent it from happening again.
Re: (Score:2)
Well, as long as some heads roll. right?
I mean, the whole scheme of things nowadays is to identify a scapegoat, shame him/her profoundly, and move along.
It gives one armor?
Re: (Score:2)
It helps to read to the end.
Re: (Score:3)
I mean, the whole scheme of things nowadays is to identify a scapegoat, shame him/her profoundly, and move along.
I don't know about "nowadays": seeing how the "scapegoat" tradition is mentioned in Leviticus, the scheme has been used for at least 2500 years (if you agree with current estimates of Leviticus being written around 500 BC), and probably much longer.
On the contrary, I'd say the scheme isn't as popular today, at least in America. The current administration doesn't bother loading a goat with the sins of the community and punishing or sacrificing the goat instead of the sinners anymore. They just deny the sins
Lots of lessons for us all in the official report. (Score:1)
Well worth skimming the official technical investigation report. Ten human mistakes are discussed in detail -- many are project management and organizational-culture issues that anyone in technology may run into. (I've certainly run into such issues in my career.) Official report:
https://regmedia.co.uk/2025/12... [regmedia.co.uk]
lock on a gateway what network hardware has LOTO? (Score:2)
lock on a gateway what network hardware has LOTO?
Re: (Score:3)
Thanks for the reference. I will use this as one more teaching example of exceptional incompetence when running critical IT systems.
And, gross negligence or not? (Score:1)
Because I think the ones responsible should be charged with manslaughter. Gross negligence could allow that.
Re: (Score:1)
Re: (Score:2)
Naa, you are just showing off your functional illiteracy. Again. You may want to look up what "could" means.
Re: (Score:1)
Re: (Score:2)
Singapore. What's your point? Singapore isn't China, it's not known for cost cutting or cheap shit.
What A Hachet Job (Score:2)
I do love when some clueless "auditor" or government agency does an after action report and just buries the workers with blame. As if they know or could possibly do better.
Kick 'em again! The filthy fucking network team. They deserve the ire of an entire nation of clueless cunts. Amirite?
Read it to the end (Score:3)
'Process was not followed, and incorrect procedures were selected. Checks were inadequate, controls avoided and alerts given insufficient attention. There appeared to be reticence in seeking more experienced advice within Networks and a focus on speed and getting the task done, rather than an emphasis on doing things properly.'
Those are deeply negligent mistakes which should result in painful consequences for those who made those mistakes.
Re: (Score:2)
Are you sure that there wasn't a manager directing the actions of the low-level people?
Manager makes no difference (Score:1)
Someone, be it the staffer in the trench or the manager failed to follow procedure / make sure the correct procedures were followed. That is a culpable negligence.
Re: (Score:2)
If you don't read what was written you may come to that conclusion. If you actually read and understood what was written you'll see the auditor laid blame at virtually every level.
Impressive screw up. (Score:2)
It takes skill to mess up a firewall so that it kills 2 people.
Not everyone can be that incompetent.