Statewide 911 Outage Was Caused By 911 Vendor's Malfunctioning Firewall (arstechnica.com) 35
An anonymous reader quotes a report from Ars Technica: A 911 vendor's malfunctioning firewall caused a statewide outage in the emergency calling system in Massachusetts on Tuesday afternoon, the state government said. A Massachusetts government press release issued yesterday said the state's 911 vendor, Comtech, "has advised State 911 that they have applied a technical solution to ensure that this does not happen again." "A preliminary investigation conducted by the State 911 Department and Comtech determined that the outage was the result of a firewall, a safety feature that provides protection against cyberattacks and hacking," the announcement said. "The firewall prevented calls from getting to the 911 dispatch centers, also known as Public Safety Answer Points (PSAPs)."
Comtech's initial review "confirmed that the interruption was not the result of a cyberattack or hack," but "the exact reason the firewall stopped calls from reaching dispatch centers remains under review," the state said. A full review is continuing. The 911 outage lasted two hours. Shortly after it began, the State 911 Department alerted local law enforcement and issued a statewide emergency alert to residents advising them to call their local public safety business line directly if they had an emergency. "Although some calls may not have gone through, the system allows dispatch centers to identify the phone number of callers and return those calls. The Department has not received any reports of emergencies impacted during the interruption," the Massachusetts announcement said. State 911 Department Executive Director Frank Pozniak promised that the department "will take all necessary steps to prevent a future occurrence." Massachusetts has 204 Public Safety Answering Points that received an average of 8,800 calls, combined, per day in 2023. In case of a 911 outage, an internet user recommends everyone save their local dispatch number in their contacts. You can also use these methods to reach emergency services:
- Call the non-emergency police line in your area.
- Use alternative numbers to reach first responders, such as the direct line to the local police or county sheriff's office.
- Use a landline, Wi-Fi calling or another cell carrier if a cellular service issue is responsible for the 911 outage.
- Send a text to 911, if the service is available in your area. The Federal Trade Commission (FCC) provides a list of areas currently supporting Text-to-911 on its website.
If calls from landlines to 911 and their non-emergency hotline are not working, police departments can still see the numbers of those who called from cell phones and call them back as soon as possible, notes WTOL.
Comtech's initial review "confirmed that the interruption was not the result of a cyberattack or hack," but "the exact reason the firewall stopped calls from reaching dispatch centers remains under review," the state said. A full review is continuing. The 911 outage lasted two hours. Shortly after it began, the State 911 Department alerted local law enforcement and issued a statewide emergency alert to residents advising them to call their local public safety business line directly if they had an emergency. "Although some calls may not have gone through, the system allows dispatch centers to identify the phone number of callers and return those calls. The Department has not received any reports of emergencies impacted during the interruption," the Massachusetts announcement said. State 911 Department Executive Director Frank Pozniak promised that the department "will take all necessary steps to prevent a future occurrence." Massachusetts has 204 Public Safety Answering Points that received an average of 8,800 calls, combined, per day in 2023. In case of a 911 outage, an internet user recommends everyone save their local dispatch number in their contacts. You can also use these methods to reach emergency services:
- Call the non-emergency police line in your area.
- Use alternative numbers to reach first responders, such as the direct line to the local police or county sheriff's office.
- Use a landline, Wi-Fi calling or another cell carrier if a cellular service issue is responsible for the 911 outage.
- Send a text to 911, if the service is available in your area. The Federal Trade Commission (FCC) provides a list of areas currently supporting Text-to-911 on its website.
If calls from landlines to 911 and their non-emergency hotline are not working, police departments can still see the numbers of those who called from cell phones and call them back as soon as possible, notes WTOL.
No more "vendors" (Score:2, Insightful)
Or worse. We have the tech to make it so nobody every has to go through emissions but we don't use it because it's somebody's business to run the emissions sites. So every year I've got to drag my car down to get tested (it's an old car because of course it is) so that somebody can make a small fortune running the test sites. mea
Re: (Score:2)
PRIVATE ALWAYS BETTERERERERER!
That skim off the top is the entire point. That skim off the top for essential services turns into a nice stack of cash in public servant pockets. Ergo, "private industry is always better equipped to do everything, always, because $profit." I'm amazed we don't have private industry / for profit police and fire departments at this point. I suppose it's only a matter of time if our government insists that Wall Street is always the most important entity in the country.
Re: (Score:2)
You should look at the water suppliers in England and Wales.
Back in the late 1980s the Margaret Thatcher government privatised water supply and sewage handling, this fluff piece by the current suppliers [water.org.uk] explains why they did it. As Wikipedia points out, [wikipedia.org] the problems extant then were of the Thatcher government's own making.
That
Re: (Score:2)
You should look at the water suppliers in England and Wales. Back in the late 1980s the Margaret Thatcher government privatised water supply and sewage handling, this fluff piece by the current suppliers [water.org.uk] explains why they did it. As Wikipedia points out, [wikipedia.org] the problems extant then were of the Thatcher government's own making. That
looks rather misleading nowadays, river pollution is at levels not seen for a very long time and a large proportion of the beaches are not safe for bathing. So what went wrong? - the water companies have paid massive dividends to their shareholders over the years - the way the "measuring metric" for performance is drawn up, it can be achieved quite easily by routing sewage straight into the rivers. The "University Boat Race" is a major sporting event in the UK and the winning team traditionally jumps into the river to cool down. This year they were warned not to do that because testing showed that the E Coli levels made the water unsafe. Boat clubs further up the Thames have had to abandon the water and campaign for changes in the situation instead. One water company wants to triple the fees everyone has to pay so that they can start fixing their previous neglect.
The main argument for privatisation in the first place was that then water would no longer be subject to the whims of the government of the day. That argument applies to other fields as well - the National Health Service (NHS) is seriously underfunded and can no longer afford to pay the GP doctors they need, a large number of those GPs are either unemployed or have found alternative employment outside the medical field.
I have worked in state-owned areas, and in private companies. Neither has a monopoly on good "structures". The absolute worst place I worked at was owned by two Unions, incompetence was pervasive but not universal there.
I imagine our government officials reading this account and fapping furiously over the profit potential.
Re: (Score:2)
I can see why you post AC, I'd be embarrassed to put my name to an opinion that dumb too.
Re: (Score:1)
Here in Washington state they privatized the car tests a few years ago as well. Previously it was a decent if boring job, they did an actual exhaust test, and the workers could give you a clue how to fix any issues and/or where to go to fix it. Now they're still using the state-owned and maintained sites (for no charge, of course), and the workers frequently don't even speak any English beyond asking you to get out of the car so that they can plug into the computer.
Re: (Score:2)
Nonsense! I usually don't call out the BS I see on this site but this isn't sitting right with me. Your entire post is BS. Here in Washington State we did away with car test a few years ago. It was Jan 1st, 2020 that we ended those tests. They were never privatized or outsourced.
"...and the worker frequently don't even speak any English beyond..." Based on the fact that the beginning of your comment is a complete and udder fabrication I am also going to call you out for this BS. The nonexistent exhaust tes
Re: (Score:2)
Got my truckette tested five years ago, hadn't realized the program was phased out after that. It certainly was privatized then.
Sounds like incompetence (Score:3)
Firewalls can a bit tricky to configure for thinks like SIP. 2 hours to fix sounds like a rule-set change or configuration change, not a hardware replacement. Hence this looks a lot like incompetent (cheap) personnel, inadequate testing and, likely, inadequate penalties for outages.
Re: (Score:2)
Those voters are old and frankly they're not going to be around for a lot longer. The ones that don't go tits up will end up in nursing homes where for reasons pollsters aren't sure of they stop voting.
The younger
Re: (Score:2)
SIP and NAT don't play well together. Use routable addressing (ie move to IPv6) and it's a LOT easier.
Another thing that could trigger a 2 hour outage would be active responses - see excessive traffic from one address and block that address, systems like this get broken by NAT when multiple devices/users come from a single address (ie lots of traffic from one address), or are often trivially triggered by sending spoofed packets.
Re: (Score:2)
In this day and age configuring a firewall just doesn't cut the mustard: https://sizeof.cat/notes/17168... [sizeof.cat]
Canton (Score:1)
Fantastic recommendation! (Score:5, Insightful)
In case of a 911 outage, an internet user recommends everyone save their local dispatch number in their contacts.
911 was supposed to be an end to needing to know your local $police, $fire, $rescue numbers individually. Now, thanks to technology, we're rolling that shit back. Time to start handing out those funky glossy cards with all the local emergency service numbers on them again!
God, I love this future. It's full of failure and sadness.
Re: (Score:3)
Wait until they realize 911 can outsourced to India or the Philippines.
Re: (Score:2)
Not so.
You are supposed to be aware of the non-emergency numbers for non-emergencies.
If you need a VIN verification or a fire inspection or something and you call 911 you could be cited in some jurisdictions.
Re: (Score:2)
Why? It's not an emergency, so there's no time pressure to remember what my local non-emergency number is. If I have a need for it, I can look it up in many places - it's not an emergency after all.
And if you do have an emergency, those lines all tell you to hang up and dial 911 because all requests on the non-emergency line may take some time to get through before yours. If 911 doesn't work, there's almost always a first responde
Re: (Score:2)
Not so.
You are supposed to be aware of the non-emergency numbers for non-emergencies.
If you need a VIN verification or a fire inspection or something and you call 911 you could be cited in some jurisdictions.
If it's not an emergency you can just go look up the number. Emergency means you need the number right now, and 911 was supposed to be the universal (in this country) "I NEED HELP NOW!" number. If you don't know how to look up a phone number in a non-emergency, that's sad, but a completely different discussion.
Re: (Score:1)
Here's a million dollar idea to someone who wants government money: Every cell tower could be broadcasting a special set of contacts for whatever local non-emergency numbers happen to be nearest that cell-tower. If your phone picks up more than one, that's no problem be
Re: (Score:2)
You used to be able to dial zero for the operator, who had a set of contacts. This long predates cell phones, back to the times when a human would physically connect wires to complete your call.
Re: (Score:2)
I can still remember one of the first times I called my grandparents on my own - "Hello, operator? Can you tell me how to dial for Los Angeles, the number I was given is OL 32456"
Think my mom gave me the number that way on purpose, but the operator certainly got a kick out of someone making a call like that in the early 80s....
Re: (Score:2)
You used to be able to dial zero for the operator, who had a set of contacts. This long predates cell phones, back to the times when a human would physically connect wires to complete your call.
I 'member. I also 'member group lines out in the country. So you'd pick up the phone and the neighbors would be on a call with their family and then you'd get to argue about who got to use the phone. Turning neighbors into siblings! It was a real riot when all the farmers in the area needed to start calling around for the same supplies. You'd get five or six on the line together calling the same suppliers. Poor dudes felt ganged up on. LOL.
Re: (Score:1)
In reality, if you need to talk to the police for a non-emergency, you call 911 and tell them that "This isn't an emergency, however, I need to talk to the local police". They will give you the number of the most appropriate station (different possibilities based on who is working the local municipality, the county, and state), and/or forward your call
Re: (Score:2)
In reality, if you need to talk to the police for a non-emergency, you call 911 and tell them that "This isn't an emergency, however, I need to talk to the local police". They will give you the number of the most appropriate station (different possibilities based on who is working the local municipality, the county, and state), and/or forward your call
Why not have a separate number for that?
In the UK, 999 is the emergency number... This is the "I really need help right now" number.
For the police, 101 is the number for less immediate help such as reporting a crime or traffic accident. Something that's a priority but not life threatening. 111 is the NHS number, again for serious medical incidents but nothing requiring emergency care.
In boating speak:
999 = Mayday.
101/111 = PAN.
Also the old joke, why did Porsche call it the 911? so Americans k
Failure Point (Score:4, Insightful)
It's interesting to me that the firewall was the failure point.
I work for a regional telco and we recently worked on moving a local E911 call center. The new building was incredibly overbuild (earthquake, security, etc.). Dual facilities entrances (for dual 72-strand fiber cables, each going to a different POP). Feeds from 2 power companies. Voice feed independently from 2 COs, with failover to each other. Huge UPSs, generators. The equipment that answers calls has redundant modules to take over for each other. Internet feeds from 2 unrelated providers. They thought that all through.
I have no idea about the redundancy/diversity of their data network hardware. It only takes one point in the network to fail.
Re: (Score:2)
I'm mostly wondering why they farmed out a public safety service to a third party. I rather suspect that someone on the state legislature was related to a salescritter in the vendor's company.
Re: Failure Point (Score:2)
Frelling Libertardians, it's a religious belief with you, isn't it?
I wonder what the math looks like for this kind of (Score:4, Interesting)
When a large enough population looses access to 911 for a long enough period of time, some of those would-be callers are alone and suffering from a heart attack. We have tons of good data on the efficacy of medical intervention for heart attacks. We're damn good at handling them. But when an outage is large enough it means some number of heart attack victims are going to die as a result.
A corporate vendor might have murdered a few people via cost-cutting.
Firewall protects against cyberattacks :o (Score:2)
but wait, there's more! (Score:2)
in their haste to notify people, Mass emcon folks sent out an emergency alert via either IPAWS/EAS to the entire region (including other states) that the 911 system was down - causing those states to also activate IPAWS/EAS for their regions telling constituents to "PLEASE DISREGARD THE EMERGENCY ALERT THAT WAS RECIEVED. THIS WAS SENT BY ANOTHER STATE. MAINE'S 911 SYSTEM IS FULLY OPERATIONAL".
Sitting in a routine state emergency services committee meeting and having everyones phones detonate/alert was amu