New York City Has a Y2K-Like Problem, and It Doesn't Want You To Know About It (nytimes.com) 119
On April 6, something known as the GPS rollover, a cousin to the dreaded Y2K bug, mostly came and went, as businesses and government agencies around the world heeded warnings and made software or hardware updates in advance. But in New York, something went wrong -- and city officials seem to not want anyone to know. [Editor's note: the link may be paywalled; alternative source] New submitter RAYinNYC shares a report: At 7:59 p.m. E.D.T. on Saturday, the New York City Wireless Network, or NYCWiN, went dark, waylaying numerous city tasks and functions, including the collection and transmission of information from some Police Department license plate readers. The shutdown also interrupted the ability of the Department of Transportation to program traffic lights, and prevented agencies such as the sanitation and parks departments from staying connected with far-flung offices and work sites. The culprit was a long-anticipated calendar reset of the centralized Global Positioning System, which connects to devices and computer networks around the world. There has been no public disclosure that NYCWiN, a $500 million network built for the city by Northrop Grumman, was offline and remains so, even as workers are trying to restore it.
City officials tried to play down the shutdown when first asked about it on Monday, speaking of it as if it were a routine maintenance issue. "The city is in the process of upgrading some components of our private wireless network," Stephanie Raphael, a spokeswoman for the Department of Information Technology and Telecommunications, said in an email on Monday. She referred to the glitch as a "brief software installation period." By Tuesday, the agency acknowledged the network shutdown, but said in an emailed statement that "no critical public safety systems are affected." Ms. Raphael admitted that technicians have been unable to get the network back up and running, adding, "We're working overtime to update the network and bring all of it back online." The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover.
City officials tried to play down the shutdown when first asked about it on Monday, speaking of it as if it were a routine maintenance issue. "The city is in the process of upgrading some components of our private wireless network," Stephanie Raphael, a spokeswoman for the Department of Information Technology and Telecommunications, said in an email on Monday. She referred to the glitch as a "brief software installation period." By Tuesday, the agency acknowledged the network shutdown, but said in an emailed statement that "no critical public safety systems are affected." Ms. Raphael admitted that technicians have been unable to get the network back up and running, adding, "We're working overtime to update the network and bring all of it back online." The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover.
$0 (Score:2)
NYC Maintenance budget. You would assume they would switch out 5% of everything a year for a 20 year refresh cycle, but No.
Re: (Score:3)
For $500M you could have built and deployed the system ($40M?) and then put the rest ($460M) in a trust whose proceeds could have funded maintenance forever.
Re: (Score:3)
From what I had read earlier last week, the issue was affecting an emergency response network where they pay Northrop Grumman 40 million a year to maintain.
Maintaining it each year is as much as building it should have cost.
Re: (Score:2, Insightful)
How much of that goes to union bosses who get paid to stand around on the job site and look important? This is NYC we're talking about here.
Re: (Score:3)
Since when does a union boss gets payment when a company conducts a project?
Re: (Score:2, Troll)
For $500M you could have built and deployed the system ($40M?)
Making project and construction estimates while not only not knowing the scope, but none of the requirements? I take it you spend your days posting on Slashdot because you're an unemployed project manager who ran your company into the ground?
Re: (Score:3)
He probably has no idea how few money 40million actually are (his proposal).
Assuming a router or repeater costs $20 and a work hour $20 too, and making the optimistic assumption a worker sets up 4 routers per hour, then we have about $100 costs per hour (including the routers). So 40,000,000 / 100 is 40,000 work hours and 160,000 routers which covers a square of 400 x 400 routers. With about 100m distance from router to router that would be a 40km square, something like 25miles x 25miles.
If a worker is real
Re: (Score:2)
I would assume it costs several hundred, and that core infrastructure runs in the millions. I was part of a project that installed a wireless multipurpose voice/data system at a single plant. That ran into the $40m mark.
Re: (Score:3)
I really do enjoy it when I get modded Troll by people who have no idea. $40million? Ha we spent more than that on a wireless project at a large chemical plant. If I had to give a thumb in the air estimate I would have come at larger than $500million for city wide multipurpose wireless infrastructure.
Re: (Score:3)
I was involved with a similar project in a different city, and just the infrastructure upgrades to the shelters and towers ran about $20 million per site for about 20 sites. None of them had been designed to a sufficiently robust criteria originally. The networking bits were easily another $10 million per site from what I understood.
Re: (Score:1)
That would require planning and brains. The antithesis of Government / Politics.
Re: (Score:2)
I doubt it is a hardware issue. I am more inclined to think it is software that is too tightly integrated, and software that was integrated to the "Mass" without a restart ability.
If A depends on B and B depends on C and C depends on A, then, how to bring up the system on A with all the dependencies?
Systems need to be more loosely coupled.
Re: (Score:3)
I live in NYC, and have my whole life. It mostly sucks. Giuliani fixed a lot of it, but going downhill to the 70s again, fast
Re: (Score:1)
Giuliani fixed nothing, you know nothing about it, and I doubt you've ever lived there on that basis.
Re: (Score:2)
well, it's hard to compete with the bright lights and excitement of Sheboygan, but New Yorkers seem to make do.
Re:New York sucksI am (Score:2)
Re: (Score:1)
>while you can't even write complete sentences and spell simple words like 'secede'
OP Anon didn't write that, Pgmrdlm did. Any moron could see that.
Re: (Score:1)
Maybe he just wants the nation to succeed? I mean, an admirable goal I guess?
Re: (Score:1)
FYI, if you split a quoted post
Testing
into multiple quotes to respond to parent bullet points,
Test
this now trip's slashdot's lameness filter. /offtopic
Test
Can't reproduce?
Re:Lets face it, this is pretty routine maintenanc (Score:5, Insightful)
They pay 45 million a year in support for that network to Northrop Grumman. GPS being the root of that downtime should have been easily fixable. The GPS epoch that ended was the second one since it's origin in 1980. It was entirely predictable down to day dates minutes and they had 20 years to prepare for it. Hell they even have 20 or so or more satellites with atomic clocks whose sole purpose for being built is calculating the time.
Re: (Score:2)
There's a bit of confusion here I think... There are tons of IoT devices, which may or may not have GPS problems of their own, but isn't the main issue just the network being down? Once that's back up, what remains of individual firmware rollouts for various types of devices can be addressed.
I'm assuming the NYCWiN has some sort of mesh topology? Maybe with scrambled GPS data it's now trying to reconnect to devices far away from where it actually is and is this failing to converge any routes? I'm a bit curi
Re: (Score:2)
Heh. If its the 2.5Ghz spectrum cellular radio network I think it is, the base stations all use GPS-referenced 10MHz source oscillators to feed the radio stacks. If they did not choose the reference clock hardware wisely then I could see why its taking so long to get things back: they have to touch every base station.
Re: Lets face it, this is pretty routine maintenan (Score:2)
Wow, that's amazing - you solved the problem by simply reading msmash's summary - you are like a technical savant. /sarcasm
The issue isn't the GPS hardware, it's the signal the satellites are sending and the software that is processing those signals.
Re: (Score:2)
The issue isn't the GPS hardware, it's the signal the satellites are sending and the software that is processing those signals.
The satellites send a week number from 0 to 1023, which is a range of slightly less than 20 years. The GPS hardware needs to turn this into a date. You could build GPS hardware with firmware that assumes it will never be used _before_ the hardware was built, so the date must be between (starting date built into firmware) to (same + 1023 weeks).
This is fine if you assume your hardware breaks down within 20 years. If you assume it lasts longer, then the firmware must be updateable, so the start date can be
Re: (Score:3)
Actually the firmware doesn't need to upgradable, you just need to be able to set and save the current date occasionally to non volatile memory. The device could do this itself periodically after synchronising itself. Yes, there is a small risk of spoofed GPS signals causing an anomalous date to be saved but even there the risk can be minimised. Time more that x seconds since last recording, more that y seconds of continuously synchronised signals. For mobile GPS devices more that z km's of travel as wel
Re:Lets face it, this is pretty routine maintenanc (Score:5, Insightful)
I didn't say this was absurd to have downtime. This isn't a planned outage for maintenance either. To be clear this isn't 1% down time. This issue began on Sunday the 6th, and the network has been down for at least nine or ten days. At this point the network has been down for about 3.6 % of the year, and that percentage is increasing.
What I said quite clearly, they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise. They had they designed the network properly would have been aware of this absolute unavoidable reality and been able to pre-emptively planned for and fixed the underlying causes.
They clearly did not, so the question becomes what exactly do they do for the 40 million dollar contract, if not maintain what they built and marketed as a safe alternative and reliable and viable critical information network.
Re: (Score:3)
they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise.
It was no surprise. The people that were there way back when knew perfectly well about the problem, and they also knew perfectly well that they wouldn't be around to be blamed for it in 20 years.
It was easy for them at the time to make the decision to "acknowledge the problem" and to quietly pretend that someone else would fix it later.
All those people are long gone, and the people that came after them just kept kicking the can down the road until they ran out of road.
Re: (Score:3)
they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise.
It was no surprise. The people that were there way back when knew perfectly well about the problem, and they also knew perfectly well that they wouldn't be around to be blamed for it in 20 years.
It was easy for them at the time to make the decision to "acknowledge the problem" and to quietly pretend that someone else would fix it later.
All those people are long gone, and the people that came after them just kept kicking the can down the road until they ran out of road.
Some of us realized we would still be around when the clock ran out. I watched others deal with DEC's date-75 problem. I fixed the Town of Hudson's Y2K problems in 1998, and later reported a Y2K bug in DEC's software. The next problem will come when the Unix 32-bit seconds counter overflows in 2038. There are people who are concerned with such things, but we don't get much attention until the crisis is upon us and it is too late for any of the easy solutions.
Re: (Score:2)
The next problem will come when the Unix 32-bit seconds counter overflows in 2038.
I can't wait to see the general havoc this causes, assuming I'm still alive to enjoy it.
Re: (Score:2)
Technically, the 32-bit signed integer that holds the Unix seconds value overflows and goes negative! It does not overflow to zero. Which means some systems will jump back in time to 1970 - 68 = 1902.
Note that Y2038 failures will start to manifest themselves when attempts are made to set timers past the overflow date. Therefore, failures will start to appear before the actual overflow date.
Any system calls, protocols and file systems that use 32-bit signed absolute time fields will also be impacted. Therefo
Re: (Score:2)
Sorry my math was confused. But my larger point stands;\ that ten days of a critical response network is not a prepared response. If it was possible that this was going to occur they should have spent months prior to alerting all parties and this news cycle would have been easily avoidable.
Re: (Score:2)
It seems that if they had been at all pro-active they could have spent those months making the disruption not happen at all. It's not like every device has to wait until the rollover to be patched. They could have fixed this 5 years ago and not even skipped a beat when the rollover happened.
Re:Lets face it, this is pretty routine maintenanc (Score:5, Interesting)
A big problem with GPS is so many 3rd party hw vendors made it proprietary-dirt-simple and "upgrading" to a new signal means gutting the existing logic, sometimes entirely. It's not a trivial upgrade whatsoever.
And yet every cheapie android phone out of China managed it just fine. Since the event's timing has been known down to the second since the GPS system came in to existence, most devices didn't even need an update, they left the factory ready for the event. It's not like it's a whole new protocol, it's actually the same protocol just with a counter rolled past zero.
Re: (Score:2)
Everyone making GPS devices should have known about this but now the Tom-Tom in my car thinks its noon once it gets a GPS lock.
Re: (Score:2)
According to their site [tomtom.com], there's an update available.
Re: (Score:1)
Re: (Score:3)
And yet every cheapie android phone out of China managed it just fine.
Mobile phones do not critically rely on GPS time for any function. Now the infrastructure behind them on the other hand critically require precise time sources for data synchronisation.
Your argument is like saying I survived just fine in the last blackout without a generator, why would hospital need one.
Don't make those arguments. They are anti-intellectual. And shame on the people who modded you up.
Re: (Score:2)
Actually, why would the communications network even NEED GPS to function at all?
Re: (Score:2)
... should have been easily fixable...
Yes, should. I imagine the reason it isn't simple is because some of the platforms are out of support, others are locked down and the vendor has no interest in fixing them, some require a development tool that only runs on Windows 95, and they lost the source code for the rest. Then there's the ones they probably bricked while updating.
Re: (Score:3)
Northrop Grumman designed and specced out the hardware. If they designed a network with a known death date due to a proprietary GPS it was a poor design. Even if it was an accident that they messed up and it was a human error in design, they still had years to research a replacement part to keep it viable, when they went and looked at the hardware in advance of the epoch change (what they get paid to do). This looks to me like they were caught flat footed (northrop grumman) with the failure to design/build/
Re: (Score:2)
No doubt everything you say is true. But, you stated that this should be easily fixable. I simply theorized that it isn't quite so easy to fix (or they would have done it), because of those poor decisions.
A good question here is: Should vendors be held to a standard of responsibility or is it the RFP writer's responsibility to ask for these things? If we decide to hold Northrop Grumman to a standard of responsibility, what is that standard and how is it enforced? What happens if all of the potential vendors
Re: (Score:2)
Northrop Grumman has an $299 unit fix.
Re: (Score:2)
When someone is paying you half a BILLION dollars, you are responsible for vetting the selected hardware against that sort of problem. When you're buying millions in hardware, you get to put things like that in the contract.
Given NG's line of business, none of this should have been new to them.
Re: (Score:2)
When you're buying millions in hardware, you get to put things like that in the contract.
You get to, but you don't necessarily have to.
It will be interesting to find out exactly what was in the contract. That will come out in the lawsuit, if it's not somewhere publicly posted already.
Re: (Score:2)
Since NG is also getting $45 million/year to maintain the network, it would be on them to put that in their contracts with any hardware bendors they worked with.
Re: (Score:3)
It will be interesting to find out exactly what was in the contract. That will come out in the lawsuit,
You're assuming NG didn't throw in a binding arbitration clause together with their general Disclaimer of Warranty and Force Majeure covering unexpected situations such as a GPS Rollover as part of the client onboarding.
Re: (Score:2)
It will be interesting to find out exactly what was in the contract. That will come out in the lawsuit,
You're assuming NG didn't throw in a binding arbitration clause together with their general Disclaimer of Warranty and Force Majeure covering unexpected situations such as a GPS Rollover as part of the client onboarding.
Force Majeure is in no way applicable to a man created event that has an 100% probability and known timeframe down to the second - 20 years in advance.
Re: (Score:2)
Re: (Score:2)
NYC is paying $37M a year for a managed service from Northrop Grumman. (Remember when they used to build airplanes?) Under any reasonable standard, the city did "prepare the network" - they paid NG to make it NG's problem to solve.
The forthcoming lawsuit/clawback should be fun to watch, though.
Got a link for that contract? (Score:1)
You happen to have a link to that contract, since apparently you've read it?
It's not unusual for a government contract to pay $X / year for the thing, with any changes or modifications of any kind requiring approval by the city, which means approval by several departments. The government is responsible for some things, such as providing locations to install the equipment, one contractor installs the utility power for the devices and keeps that running, another does the networking, a third provides the devi
Y2K-like? Possibly. (Score:5, Informative)
Maybe it went something like this?
Re: (Score:1)
This is the same reason programmers often deliberately inject bugs into systems. If everything is working hunky dory then management will assume they don't need their techs and fire them all. Of course, the only reason everything is working so well is because the techs are diligently and proactively dealing with failures before they can become dumpster fires.
End result: Dumpster fire.
The problem now is that they just burnt all their staff so there's no-one to fix it. Hire swarms of interns and turn the dump
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Step 5 happened at least 10 years before the equipment was initially installed. Arguably, it happened before the very first GPS satellite was launched.
Do we need a conspiracy here? (Score:4, Insightful)
Re: (Score:1)
To be fair, the government and cops expect full honesty and compliance when they are asking questions of you but they want to hide behind "everything is ok, just go away while we fix it" when you ask them. This isn't a private corporation that has a reasonable expectation and rationale for dodging media inquires. This is a public body that should be honest about the situation or at least not actively LIE about the situation. This is not and never was "an upgrade of the wireless network" unless upgrade ha
Re: (Score:3, Insightful)
No city officials informed anyone on the NY City Council about the network failure and apparent lack of preparation for the rollover. Some council members only became aware of it when the Times called to ask for comment on the situation.
And then there's this:
Laura Anglin, the deputy mayor for operations who is responsible for the information technology agency, refused to answer questions about it on Wednesday afternoon as she entered City Hall.
Asked if the city had taken the necessary steps to prepare for the GPS rollover, she said, “Talk to the press office.”
Not exactly forthcoming from the city official who's most likely to have the most pertinent information about the subject.
Media Hit Job (Score:1)
It's the directed government answer to all media inquiries. And let's be honest: getting ambushed on the way into a building is a hit job, not journalism.
Re: (Score:2)
No city officials informed anyone on the NY City Council about the network failure and apparent lack of preparation for the rollover. Some council members only became aware of it when the Times called to ask for comment on the situation.
Let's put this in the context we use for other segments of our government and see if we're being reasonable here.
If this was the federal government - especially the Trump administration - we would say something like "it's too complex, and the administration was too stupid to understand it" and we'd move on. Yet because it is the government of the largest city in the US, we expect for some reason that they will have a deeper understanding of technical matters that we would place "beyond the pay grade" of
Questions raised (Score:4, Insightful)
Re:Questions raised (Score:4, Interesting)
They didn't have to prepare. The preparation was paying 40 million a year to northrop grumman to maintain and be prepared for them.
Re: (Score:2)
Government outsources to private enterprise and they mess up = government contract management is incompetent.
Somewhere in there it needs to be realised that people mess up, regardless of whether they work for a public or private organisation.
Get rid of the blame game and work out a non-biased system that determines whether the infrastructure or service should be government run or not, and set them up as required.
oh no, the surveillance state! (Score:3)
> waylaying [...] the collection and transmission of information from some Police Department license plate readers
good.
Re: (Score:2)
Dangers of AI (Score:2)
Re: (Score:2)
My first thought was DedSec.
Re: (Score:1)
The underlying network should be pretty agnostic. Devices connected to the networks probably need timing to a reasonable degree of precision for things to stay in sync, and when your GPS receiver decides to party like it's 1999, you lose that precision.
Having just reviewed several GPS receivers from different brands over the last month or so to ensure that my workplace wouldn't have such problems (most are fine and dandy, one particularly ancient one needs replacing in the coming months - interestingly, th
Second GPS rollover (Score:2)
Uh, this is the second GPS date rollover since its inception in 1980. The first was in 1999, after 1024 weeks of operation.
There is no excuse for any device released or updated after 1999 to not account for this GPS glitch.
The answer is "No" (Score:3)
"The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover."
I'm no rocket scientist but seeing as how they're having massive problems due to the rollover I'd have to say no, they didn't.
Faith in 2038 (Score:2)