How a Power Outage In Colorado Caused US Official Time To Be 4.8 Microseconds (npr.org) 63
Tony Isaac shares a report from NPR: The U.S. government calculates the country's official time using more than a dozen atomic clocks at a federal facility northwest of Denver. But when a destructive windstorm knocked out power to the National Institute of Standards and Technology (NIST) laboratory in Boulder on Wednesday and a backup generator subsequently failed, time ever so slightly slowed down. The lapse "resulted in NIST UTC [universal coordinated time] being 4.8 microseconds slower than it should have been," NIST spokesperson Rebecca Jacobson said in an email. [...]
Since 2007, the official time of the U.S. has been determined by the commerce secretary, who oversees NIST, along with the U.S. Navy. The national time standard is known as NIST UTC. (Somewhat confusingly, UTC itself is a separate, global time standard to which the U.S. and other countries contribute measurements.) NIST currently calculates the standard using a weighted average of the readings of 16 atomic clocks situated across the Boulder campus. Atomic clocks, including hydrogen masers and cesium beam clocks, rely on the natural resonant frequencies of atoms to tell time with extremely high accuracy.
All of the atomic clocks continued ticking through the power outage last week thanks to their battery backup systems, according to NIST supervisory research physicist Jeff Sherman. What failed was the connection between some of the clocks and NIST's measurement and distribution systems, he said. Some critical operations staff who were still on site following the severe weather were able to restore backup power by activating a diesel generator the team had kept in reserve, Sherman said.
Since 2007, the official time of the U.S. has been determined by the commerce secretary, who oversees NIST, along with the U.S. Navy. The national time standard is known as NIST UTC. (Somewhat confusingly, UTC itself is a separate, global time standard to which the U.S. and other countries contribute measurements.) NIST currently calculates the standard using a weighted average of the readings of 16 atomic clocks situated across the Boulder campus. Atomic clocks, including hydrogen masers and cesium beam clocks, rely on the natural resonant frequencies of atoms to tell time with extremely high accuracy.
All of the atomic clocks continued ticking through the power outage last week thanks to their battery backup systems, according to NIST supervisory research physicist Jeff Sherman. What failed was the connection between some of the clocks and NIST's measurement and distribution systems, he said. Some critical operations staff who were still on site following the severe weather were able to restore backup power by activating a diesel generator the team had kept in reserve, Sherman said.
Cause and Effect. (Score:3)
But when a destructive windstorm knocked out power to the National Institute of Standards and Technology (NIST) laboratory in Boulder on Wednesday and a backup generator subsequently failed, time ever so slightly slowed down.
Its not that I’m chastising NIST backup capability as much as I’m shocked that is the steps it takes to hobble one of the country’s most critical services.
Then again, what exactly was the impact? Good time to bolster recovery processes if there is a measurable one I guess. Doesn’t happen very often.
Re:Cause and Effect. (Score:4, Interesting)
I am actually shocked. As claimed by the article, it's not the clocks themselves that were compromised, it's actually the network outage that caused the problem.
That makes sense, because the time protocol requires estimating routing delays, so when the outage happened there may have been a change to the estimated delays in the network. And that seems frankly ridiculous, if true, since there's no reason that the network relaying the atomic clock readings should be tampered with on an ordinary operating day.
So the fact that the reference clock is off by this much either suggests a flaw in the protocol (trying to do something that doesn't make sense, like re-estimating the delays continuously through another route) or a flaw in the system design (does the computer which averages the 16 atomic clock ticks actually have an atomic clock itself? It would be stupid to use a cheap commercial time crystal as a fallback in case of a complete outage).
4 microseconds is an eternity in some applications. The idea that it is so easy to tamper with the reference time by messing with the network is unacceptable. NIST has some work to do here.
Re:Cause and Effect. (Score:5, Interesting)
Also do not confuse the reference clock being slow, and the time propagation across the global network (ie Internet) being slow.
It is fine for the NIST clock to be inaccessible to the wider network for small periods of time, which would inevitably introduce drift on the downstream systems. Once those systems reconnect to NIST, they could correct themselves.
What is not fine is for the NIST reference clock to itself fail to stay synchronized with the atomic clocks which create the standard. That's what I'm finding shocking (although I suspect that old school NIST engineers are embarrassed and already on the case).
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Indeed. NTP is nothing to write home about, neither is PTP for that matter. It does the job for the general public. I don't know why you latch on to this, it's a straw man.
The original argument goes like this: NIST uses 16 (according to the article) atomic clocks to define the reference time. These atomic clocks are connected in some way for reliability and variance reduction. Some AC (whether you or not, doesn't matter) claimed RTOS time slicing limits the accuracy of the clock readings to a few micros.
Re: Cause and Effect. (Score:2)
I'm not sure if this is related, but I know that during those windstorms Boulder intentionally cut power to many areas in order to reduce risk of wildfire if a power line went down. This is in response to a devastating wind-driven wildfire last year. I'm wondering if some of the NIST clocks were subject to this.
Re: Cause and Effect. (Score:2)
Re: (Score:2)
Then again, what exactly was the impact?
The NIST timekeeping makes it way into GPS. In 4.8 microseconds light can travel 1439 m - nearly 1 mile.
That is not meant to imply, at all, that GPS is suddenly going to be off by 1439 m (compared to ground truth, or other GNSSs). But there is some chance of an error propagating.
That said, GPS satellites have their own precision timekeepers, which kept ticking away unaffected. Mostly for GPS we care about timing differences between satellites, anyway. The satellites' clocks are only periodically
Re: (Score:2)
There are two parts to GPS data. There's the time signal from the satellite as well as the ephemeris data that lets you calculate where the satellite is. The former is from an on-board atomic clock but the latter is from ground observations that undoubtedly use one or another NIST time standards.
Re: (Score:2)
Now the US is officially behind the rest if the wo (Score:5, Funny)
Re: (Score:2)
It will probably take a a few years to catch up
1124 days from now, to be precise.
Don't leave me hanging! (Score:5, Funny)
How a Power Outage In Colorado Caused US Official Time To Be 4.8 Microseconds
4.8 microseconds what? Longer, shorter - wider, thinner? Unless... (*gasp*) Time is now 4.8 microseconds. /s :-)
Re: (Score:2)
Yeah, I know it's de rigueur to complain about Slashdot's editors, and sometimes that's not really fair. But these mid-sentence title clips are happening more and more often, and it's hard to come up with an explanation other than simple laziness or sloppiness.
long story short (Score:2)
Re: Don't leave me hanging! (Score:3)
That's what I understood from the headline!
The time is now: 0:00:00.000 004 8
The time is now: 0:00:00.000 004 8 (Score:3)
Well that solves the problem of Social Security...
Re: The time is now: 0:00:00.000 004 8 (Score:2)
Re: The time is now: 0:00:00.000 004 8 (Score:2)
Re: (Score:1)
How a Power Outage In Colorado Caused US Official Time To Be 4.8 Microseconds
4.8 microseconds what? Longer, shorter - wider, thinner? Unless... (*gasp*) Time is now 4.8 microseconds. /s :-)
Yes, I thought the same thing. To the author: Please form complete sentences with the title of the post.
Re: (Score:2)
Yes, I thought the same thing. To the author: Please form complete sentences with the title of the post.
Agreed, but I think this is more on the editor, who knows there's a title length limit and ignored the result in his preview ...
Re: (Score:2)
Something is wrong with the post's hadline (Score:2)
something seems to be missing
Re:Something is wrong with the post's hadline (Score:5, Funny)
meet the new slashdot editor: clippy
Re: Something is wrong with the post's hadline (Score:2)
Does AI do English grammar better than any human?
It's official (Score:2)
The USA really is going back in time compared to the rest of the world. And here I thought that was just a joke we made about Trump and his policies.
Re: It's official (Score:1)
Must be because we're going so much faster than y'all that our clocks seem slower to you ;-)
Just declare it "correct" (Score:5, Insightful)
"Freedom time"
Our professional timekeepers (Score:2)
I just love that a bunch of people immediately just went to work when the power went out.
"How's clock number five?"
"Not sure, but eleven and twelve are holding. We seem to be losing synchronization."
"We can't lose NIST UTC!! Don't we have that generator in the back?"
This is a _very_ big deal! (Score:5, Insightful)
NIST have always been the world leader in creating ever more accurate clocks, the current masers work in picoseconds and below, so allowing the reference to drift by 4.8 microseconds means that precision dropped by at least 6 orders of magnitude.
If allowed to propagate to the GPS control clocks, this would have been enough to totally destroy the navigation system since a clock that is off by 4.8 us corresponds to a position error of 1500 kilometers. (OTOH, USNO has its own large ensemble of atomic clocks, so they don't depend short term on NIST updates.)
Full disclosure: I worked with the NTP Hackers (network time protocol) team for 20+ years, so I'm probably a bit more interested in precise timekeeping than most. I have personally soldered together 4 or 5 GPS-based reference clocks that would deliver 25-35 ns RMS precision.
Re:This is a _very_ big deal! (Score:4, Interesting)
It's an interesting scenario for GNSS receivers. Most are multi constellation these days. They will have one of three or four constellations giving them a very different time and location.
I'm sure the better ones test for this scenario, but even there the proof is the real world application.
Re:This is a _very_ big deal! (Score:5, Informative)
It appears most of the high-precision users either detected or were informed that the NIST time was no longer healthy. NTP was affected, a bit more disturbing is that some other high precision users (but not GPS) were affected:
Re: (Score:1)
Why not spread them out? (Score:2)
Why doesn't NIST have backup NTP servers in other parts of the country, where the weather is likely to be different and the same issues won't necessarily strike. From what I understand, there is one in Hawaii, but how about a few others - in Florida, Pennsylvania, Washington state and Arizona, in addition to Boulder? That way, even if Boulder had lost its power, the other stations would still enable the time to be available globally
Re: (Score:2)
DOGE was something that just started this year, to determine actual waste of money. There is no way they'd have classified back-ups and redundant sites as waste. Their main focus has been detecting fraud, such as USAID.
NIST has existed over a century, so there is no reason they couldn't have had more working locations, instead of just one in Boulder, CO
Re: (Score:2)
a clock that is off by 4.8 us corresponds to a position error of 1500 kilometers
1439 meters I think, not kilometers. (299792458 m/s * 4.8e-6 s)
Still, a large amount for navigation.
Re: (Score:2)
How could this affect GPS? I was under the impression that GPS satellites have their own atomic clocks onboard and that any correction (relativistic or otherwise) is done on the receiving end. One can't just "update" a GPS clock from the ground can they?
Re: (Score:2)
You also need to know where the satellites are. That's sent out in the ephemeris, which is derived from ground observations.
You can also update GPS satellite clocks and this is done fairly regularly. The clocks in the satellites aren't nearly as good as ground based clocks and also encode the entire flight history of the satellite.
What failed was the connection between some of the (Score:2)
Do slash editors not even read their headlines ? (Score:1)
Sorry for the rhetorical question.
Re: (Score:2)
or possibly: Do slashdot editors even read
orrrrr.... is slashdot using LLMs to write the
that explains it... (Score:5, Funny)
i knew something felt off this past weekend.
UTC? (Score:2)
I assume the US would be greater again by using TUT, based in NY. OK, the orange man would not understand the implication of changing a world-wide standard but Trump Universal Time just sounds better.
More issues than power outages over there (Score:2)
Those servers are so overloaded that I remember having over 60 seconds of clock skew on my servers at some point (detected on my customer's machines due to 2fa codes failing).