Forgot your password?
Close
typodupeerror
Software

New York City 911 System Missed 1,700 Calls Due To Software Update (medicaldaily.com) 56

Posted by BeauHD from the what-gives dept.
Longtime Slashdot reader belmolis writes: Part of New York City's 911 emergency system was offline from about 3:13 a.m. to 10 a.m. on August 18th due to a software update, according to Medical Daily. Of 5,659 calls to the Bronx call center during that period, about 1,700 calls connected but could not be heard by emergency operators. Some were rerouted to other call centers, but an estimated 400 human callers never got through. The potentially serious consequences of delayed response to emergencies are not yet known. The city did not notify the public and only released information when journalists inquired. How buggy software came to be installed on a running system is not yet known. The early-morning software update was pushed by Motorola, though exactly what it changed or why it caused the failure remains unknown. The city is still investigating whether the update was adequately tested and whether the same software release was deployed elsewhere.

New York City 911 System Missed 1,700 Calls Due To Software Update More | Reply

New York City 911 System Missed 1,700 Calls Due To Software Update

Comments Filter:

  • No longer 911 (Score:5, Funny)

    by PPH ( 736903 ) on Friday August 28, 2026 @07:04PM (#66312032)

    Now its 0118999881999119725
    -3

    • Re: (Score:1)

      by ne0n ( 884282 )
      It'll cut down on pocket dials and makes a better song too.

      In all seriousness, NYC is working hard to curb subway [nypost.com] stabbing [hoodline.com] calls [theguardian.com] after the Zarutska incident.
      Ignoring the problem is easier if nobody can report an incident.

      • Re: (Score:2)

        by PPH ( 736903 )

        That nypost article is funny:

        Subway crime down, but underground murders, robberies surge: NYPD data

        How do they differentiate between "underground" crime and subways? Is the former that which is commited in manholes? [abcnews.com]

        But seriously, the big liberal cities are struggling to whitewash (is that racist?) their crime statistics. Here, in the greater Seattle area, many of the local police departments are "going dark" by encrypting their radio traffic. Seattle Police would be too. If not for the recent unfortunate incident, which led to the police chief's firing and a lot of "Hur, dur

  • Like strangely more things lately... (Score:1)

    by Anonymous Coward

    Probably AI-generated code, with agentic testing and debugging, with some junior saying "whelp looks good I guess or whatever" while he pushes it to production. All the while the original seasoned "bioware" devs are collecting unemployment.

  • 1K crises per hour? (Score:4, Interesting)

    by angryargus ( 559948 ) on Friday August 28, 2026 @07:25PM (#66312052)

    So does NYC really have on the order of 1,000 justified crises every hour or is there just a lot of abuse and drama queens in that city? That averages out to around a call per year for each inhabitant.

    • Well, a quick search tells me there are 8,546,038 New York City residents. 24k calls a day would be 0.002808319 of the population has a problem per day. Doesn't seem unreasonable when you look at it that way.

      • 8,546,038 New York City residents. 24k calls a day would be 0.002808319

        Uh, what?

        Are you trying to determine a percentage?

        24,000 different callers out of 8,546,038 New Yorkers means that 0.280832% of the population calls 911 in a given day...

        • Re: 1K crises per hour? (Score:4, Informative)

          by taustin ( 171655 ) on Friday August 28, 2026 @10:18PM (#66312208) Homepage Journal

          Still doesn't seem out of line, especially given the things people call 911 over. [buzzfeed.com]

          • I've called 911 once in my life and it was over 40 years ago.

            • I'm going to point out that NYC is a dense population metropolis. So in an average town maybe 3 people see something alarming and call 911; however, in NYC there could be 10-15 people that witness something and call. Same activity, but more observers.

              • Oh I've seen alarming things, including alarming things in NYC. I just never called 911. Of course I didn't have a mobile phone when I saw those things in NYC but I doubt if I would have called anyway.

            • Re: (Score:2)

              by taustin ( 171655 )

              And clearly, there isn't a single human being in the entire human race that is different from you in any way?

            • Re: (Score:2)

              by Hasaf ( 3744357 )
              I am 59 and I recall calling 911 once. I was on hold for over five minutes. I never did reach a person, and I hung up.

              Quite frankly, it wasn't my problem, and I was carrying a pistol (legally, with a permit). My concern was that someone would see it and the situation would escalate further. So, I walked away and tried to call 911.

              I later asked the metro police (most metro systems have their own police force) how I should have handled it. They said that I should have jogged up to the driver's compartme
    • Some calls are repeats, either different people calling about the same event or the same person calling when he or she doesn't get through. 911 also gets automated calls from alarms, some of which are false alarms. That's why, of the 1700 calls that could not be heard, they estimate that about 400 represent human callers who did not get through.

    • I don't see 1,700 "missed calls," I see about 400:

      Of 5,659 calls to the Bronx call center during that period, about 1,700 calls connected but could not be heard by emergency operators. Some were rerouted to other call centers, but an estimated 400 human callers never got through.

      That quote indicates that only 400 calls were "missed" if we define "missed" as "never got through."

    • That number is actually quite good. Chicago for example has about 750 calls / hour, which is higher than New York on a per capita basis. San Francisco has about 114 per hour and it's 1/10th the population of New York so very even.

      Many 911 calls do not lead to dispatch automatically. Some are repeats. Some are false alarms. Some are from people who genuinely don't know any better. The timing probably helped a lot. You expect higher call volumes earlier in the night.

  • A and B systems (Score:5, Interesting)

    by davidwr ( 791652 ) on Friday August 28, 2026 @08:03PM (#66312076) Homepage Journal

    Back in the last quarter of the 20th century, many POTS telephone switches had an A side and a B side.

    During software upgrades - and yes those were a thing - they would route all new calls to the A side and wait until the last call on the B side ended before they would take the B side down for the upgrade.

    Then they would do some testing before bringing up the B side.

    Only after they were confident that the B side was working would they upgrade the A side.

    This worked in part because they could schedule upgrades when the load was low enough to be handled only by half of the switch's capacity.

    • Re: (Score:3)

      by taustin ( 171655 )

      I was thinking that, in a perfect world, there would be two parallel (and completely isolated from each other) systems running simultaneously, but capable of handling the full load of both. Upgrades would involve shifting the full load to one while the other is upgraded, then very careful testing before switching.

    • This is still a thing. The problem is with any modern complex network eventually the routing happens through a system that is brought together somewhere in the middle. Just like cloud providers with all their redundancy can be brought down by a broken DNS configuration so can communications system at a point where they connect to the same network. The old analogue days made things easier with analogue multiplexors but these days a mismanaged routing configuration is a single point of failure, even if hardwa

    • Kind of like blue/green deployments.

  • I used to work in the industry (Score:5, Interesting)

    by evslin ( 612024 ) on Friday August 28, 2026 @08:07PM (#66312080)

    Not at Motorola, but another company in the space.

    Software upgrades were extremely methodical in nature. The change control calls had dozens of people on them, and the upgrade document was hundreds of pages long. The ops team (which was not the dev team) rehearsed the upgrade in lower environments for weeks before doing it for real. I was on upgrade calls mostly just as a security blanket in case developer intervention was required.

    I feel bad for the folks who have to deal with the FCC paperwork. You never wanted to get invited to one of those calls, it was like having the Inquisition show up at your doorstep.

    > How buggy software came to be installed on a running system is not yet known.

    It may not even have been the software. Well, it might have been, but it might also not have been. I got paged out at 2am one time because the domain controller, which I do not own, rotated all the passwords for the managed service accounts, which I do not control, and the SQL database, which I do not manage, suddenly lost its shit and wouldn't let any apps log in. It fixed itself in about 15 minutes, but the state noticed, and we were doing paperwork for about a week after that.

    • Software upgrades were extremely methodical in nature.
      ...
      It was like having the Inquisition show up at your doorstep.

      So ... they *were* expected? Maybe I'm thinking of a different Inquisition.

  • Unknown? (Score:3, Interesting)

    by Innovation ( 10503276 ) on Friday August 28, 2026 @08:51PM (#66312126)
    "How buggy software can be installed on a running system is unknown..." LOL. Breaking things is easy. It's keeping them from breaking that's hard.

  • Tested ? (Score:4, Funny)

    by Petersko ( 564140 ) on Friday August 28, 2026 @09:27PM (#66312146)

    "The city is still investigating whether the update was adequately tested..."

    "No."

    Hopefully I just saved the city a bunch of money.

    • Define adequately. There's literally no testing scheme that covers every possible failure scenario. Even when you have like hardware and software to stage a trial upgrade first, you can still somehow find yourself in the shit for completely unpredictable reasons when you go live.

      This applies to phonelines, to networks, to computers, to wireless infrastructure, to literally everything with software. We're long past the days where code is so short and executed so slowly that all hardware and software interact

      • Re: (Score:2)

        by rta ( 559125 )

        well, you try architect, and pre-stage, and segregate and all those things to give yourself as many is as possible and as few critical sections and points of no return as possible.

        It's not always possible (or just too expensive ) to cover everything, and there's also that thing nobody thought about... but still this one sounds pretty bad. Reading between the lines I'm guessing this want some bigger things, but that they didn't realize that it was happening at first. otherwise, if it's just sw update (vs

      • I agree with you... in the abstract. But this is emergency services 911. The definition of "adequately" should not be the common baseline.

      • It is very possible to test deployments before they happen. And for a 911 call center, this should be mandatory.

        The whole point of lower environments, is to test before go-live. Not just the code, but the deployment process itself.

        Reliable deployments *are* possible. You have to put effort into them, but it can be done. If your deployment is a series of manual steps, it's going to fail, sometimes catastrophically, a significant percentage of attempts. If the deployment is fully automated, and uses build art

        • I would expect 911 and the like to have a backup system (complete working copy of the hardware) just in case something like this happens.
          Pretty sure I remember ATC has a spare of everything ready to be hot swapped in.

      • A failure this severe, doesn't point to one of those extreme edge case scenarios. It points to sloppy engineering and sloppy testing.

        There's a reason Google Search always "just works". It didn't just happen, Google spends a lot of effort to make sure it "just works."

  • I worked for Digital for a time. They supported 911 all over the west coast at the time. An outage like that should NOT have happened. Triple redundancy hot swap environments on 3 way clusters means there should have been a stable certified environment available at all times. I'd bet either the vendor or the client cheaped out on hardware config. Unless there is a major backbone failure clients should be able to get through.

  • > How buggy software came to be installed on a running system is not yet known.

    A fully integrated cloud solution that can't be switch off, tested or rolled back. Oh, and nobody thought to get people to phone in test 911 calls post software update.
  • Calling 911 can be lethal in America, I certainly would not do it for a âoewelfare checkâ unless I want them dead
  • Yep our emergency services fail too frequently and people die, and the outages only disclosed later. The workaround is to have a different carrier off somebody else. I also want to say I have never heard of anyone being fined for incorrect or inappropriate use. There are repeat offenders.

  • It must be him. Honest. After all if Trump is assumed to be responsible for everything by some people who apply no thinking before speaking, then surely Mandani deserves similar treatment.

    And no, to be clear, I'm not a fan of either of them...

  • Probably just a routine emergency ~Dr Beeper

Slashdot Top Deals

To err is human, to forgive, beyond the scope of the Operating System.

Close