The Thorny Problem of Keeping the Internet's Time (newyorker.com) 95
An obscure software system synchronizes the network's clocks. Who will keep it running? From a report: To solve the problem of time synchronization on the arpanet, computer scientist David Mills built what programmers call a protocol -- a collection of rules and procedures that creates a lingua franca for disparate devices. The arpanet was experimental and capricious: electronics failed regularly, and technological misbehavior was common. His protocol sought to detect and correct for those misdeeds, creating a consensus about the time through an ingenious system of suspicion. Mills prided himself on puckish nomenclature, and so his clock-synchronizing system distinguished reliable "truechimers" from misleading "falsetickers." An operating system named Fuzzball, which he designed, facilitated the early work. Mills called his creation the Network Time Protocol, and N.T.P. soon became a key component of the nascent Internet. Programmers followed its instructions when they wrote timekeeping code for their computers. By 1988, Mills had refined N.T.P. to the point where it could synchronize the clocks of connected computers that had been telling vastly differing times to within tens of milliseconds -- a fraction of a blink of an eye. "I always thought that was sort of black magic," Vint Cerf, a pioneer of Internet infrastructure, told me.
Today, we take global time synchronization for granted. It is critical to the Internet, and therefore to civilization. Vital systems -- power grids, financial markets, telecommunications networks -- rely on it to keep records and sort cause from effect. N.T.P. works in partnership with satellite systems, such as the Global Positioning System (G.P.S.), and other technologies to synchronize time on our many online devices. The time kept by precise and closely aligned atomic clocks, for instance, can be broadcast via G.P.S. to numerous receivers, including those in cell towers; those receivers can be attached to N.T.P. servers that then distribute the time across devices linked together by the Internet, almost all of which run N.T.P. (Atomic clocks can also directly feed the time to N.T.P. servers.) The protocol operates on billions of devices, coÃrdinating the time on every continent. Society has never been more synchronized.
Today, we take global time synchronization for granted. It is critical to the Internet, and therefore to civilization. Vital systems -- power grids, financial markets, telecommunications networks -- rely on it to keep records and sort cause from effect. N.T.P. works in partnership with satellite systems, such as the Global Positioning System (G.P.S.), and other technologies to synchronize time on our many online devices. The time kept by precise and closely aligned atomic clocks, for instance, can be broadcast via G.P.S. to numerous receivers, including those in cell towers; those receivers can be attached to N.T.P. servers that then distribute the time across devices linked together by the Internet, almost all of which run N.T.P. (Atomic clocks can also directly feed the time to N.T.P. servers.) The protocol operates on billions of devices, coÃrdinating the time on every continent. Society has never been more synchronized.
Obscure? (Score:5, Insightful)
I realize we're headlining here, but calling NTP obscure stretches things a bit doesn't it? It's usually one of the first services one configures, often while setting up the operating system.
As the rest of the blurb suggests, it's crucial.
Funny story; I worked at a location some years back that had 200+ remote sites, several servers per site. So they had "admins" who's sole job it was to log in to each system and manually update the time ( VMs, so the time drifted a bit ). ntpdate? Of course not, they issued a date command with the appropriate time.
Time was, of course, critical because too much drift and their credentials would fail. You can imagine how often this happened.
While they knew of NTP, they didn't really know much about it and figured it was some weird unmaintained and ultimately unreliable service. They were floored when I showed them how it worked and what it could do for them. Last I heard they were still "running tests" with only a handful of sites participating, to ensure it worked.
Re:Obscure? (Score:5, Insightful)
"I realize we're headlining here, but calling NTP obscure stretches things a bit doesn't it? It's usually one of the first services one configures, often while setting up the operating system."
For most people, NTP is one of the first services that the setup configures for them, without them being even aware it exists. I'd say maybe 5% of computer users have heard at all of NTP, and that's being generous. It's just not something they interact with, other than telling it what timezone they're in, and even then they're not aware that NTP setup is why the computer is asking them that. It's universal, but obscure. A lot of vital bits in the computer are like that.
Re: (Score:2)
If we're talking about users, usually they are ignorant of all of the services. They don't know or have to know how any of it works, all they know is that they type or click and stuff happens.
If we're talking about admins, everybody knows about time synchronization. It's on all the tests, even the weak ones like A+.
Re: Obscure? (Score:3)
Re: (Score:2)
Why are we talking about NTP?
Because once a quarter Slashdot seems to run a pointless story about NTP, I suspect to try and keep some form of geek credentials to the bots... I mean new users who seem to sign up only to talk shit about one or the other political party.
Re: (Score:2)
Is there a proposal to change the protocol or replace it?
Yes. In the article this is mentioned. May I congratulate you on proper Slashdot behaviour and I promise to behave better in future.
Should we also talk about LDAP? That obscure authentication and directory protocol? Maybe RADIUS?
I see there are some exciting recent articles from Ericsson about EAP over Radius. That definitely sounds like News for Nerds. You should post it.
Re: (Score:2)
Re: (Score:3)
Does networking qualify as "obscure" then? Because when people set up a new computer, TCP/IP is never mentioned...
Re: (Score:3)
Re: (Score:1)
Re: (Score:2)
You consider NIST & USNO random internet sources?
Is USNO accessible again? Last I checked neither tick nor tock was responding.
Re: (Score:2)
It's pretty much rock solid as far as time sources go and paired with a few stratum devices you can build your own solid source of time so you don't have to rely on random internet sources.
What do you mean; stratum devices?
All clocks are "stratum devices", they just have a different stratum level. If you do "ntpq -pn", you will see the stratum level of each clock on each device. It's under the "st" column:
ntpq -pn
remote refid st t when poll reach delay offset jitter
-69.60.224.10 205.206.70.40 3 u 227 256 377 78.385 28.908 2.648
-216.46.1.2 132.246.11.229 3 u 58 256 377 67.949 2.351 0.735
-206.186.255.228 12
Re: (Score:2)
There is another time protocol called PTP, or Precision Time Protocol. It requires hardware support from the NIC, so it can accurately timestamp when packets arrive and eliminate any delay waiting for busses to become free, DMA to complete or CPU interrupts to happen.
It only works locally though, on a LAN that isn't too congested. As such it can't be synchronized over the internet, it needs a local source of time like a GNSS receiver. GNSS is the generic term for satellite navigation systems like GPS, GLONA
Re: (Score:2)
It only works locally though, on a LAN that isn't too congested. As such it can't be synchronized over the internet, it needs a local source of time like a GNSS receiver.
Sure, or to have an atomic clock (surprisingly affordable) or to be synchronized to a remote source via... NTP.
Re: (Score:2)
NTP is only millisecond accuracy, PTP will get you down to microseconds, even nanoseconds.
Re: (Score:2)
Who care? All the clocks on your LAN using PTP will still have microseconds, even nanoseconds discrepancies between them and for clock outside your LAN (WAN) it won't matter as much because of packet delays which would make them diverge more anyway.
Re: (Score:2)
Lots of people care. For example, you get radio transmitters that synchronize so they can do time division multiplexing, or beam forming.
Re: (Score:2)
or to have an atomic clock (surprisingly affordable)
An actual atomic clock or a radio-synchronized clock?
Re: (Score:2)
Re: (Score:2)
Suddenly I'm feeling much better about myself and my value. Thank you.
Re: (Score:1)
Re: Yes, obscure. To some. (Score:1)
Re: (Score:2)
To some "admins" even DNS is obscure. To kids who tinker with VMs all day, NTP easily seems unreliable. Don't run it in a VM. Run it on the hardware, and have the VM take its time cue from the host.
Not possible in some VMs, in that case you simply need to have their ntpd sync with hardware based ntpd running on the LAN (or bare metal host) and sync every 16 secs. Note the high jitter value but offset is fine.
ntpq -pn
remote refid st t when poll reach delay offset jitter
*192.168.23.1 216.197.228.230 2 u 3 16 377 0.076 -0.003 824.389
Re:Obscure? (Score:5, Insightful)
This is a New Yorker article. They're not exactly writing to the tech crowd.
Re: (Score:2)
Thank goodness that VM clock-drift problem was finally solved.
The random number generator's entropy source? Not so much. [archlinux.org]
Re: (Score:1)
Oh, well, security wasn't really a concern as it was already critically compromised.
You see, some contractor at some point in the unknowable past was tasked with "making this process easier", so they came up with shared ssh keys. As in, everyone in that dept shared a single ssh key.
You can probably guess which user the keys belonged to on these servers ( yes, someone manually logged in and uploaded the very same authorized_keys file to all 200+ site servers ).
So entropy was, and is, low on the list of conc
RFC (Score:5, Informative)
Re:RFC (Score:4, Informative)
>> Here's the RFC for NTP if anyone is interested. https://www.rfc-editor.org/rfc... [rfc-editor.org]
That's for the current NTP 4; the original RFC for version 1 is https://www.rfc-editor.org/rfc... [rfc-editor.org] from back in 1985 which demonstrates how far back this goes.
If everything is synchronized (Score:5, Funny)
Re: (Score:2)
Why am I always late?
I checked the specs and it referred to this as a "PEBKAC error"
Re: (Score:2)
Unfortunately, your system has not yet been updated to the latest version of NTP.
The need for time synchronisation sure has come a long way since the need was found to have all towns connected by rail aware of railway time. Prior to that, disagreement about the time of day by an hour or two did not much matter. Having a watch was kind of overkill when glancing at the position of the sun was generally good enough.
faulty eletronics (Score:2)
I don't think it would be wise to assume things are any more reliable than they were.
The internet was built on it's resilience, and many, many, many things on it may be secretly relying on this resilience to function properly, so attempts are gaining performance by removing some of this may end up quite badly.
Re: (Score:2)
That's why NTP works well. It assumes that everything around it may be squirrely, routes will possibly be asymmetric and over-buffered, etc.
The various inferior substitutes offered by MS and systemd are just that.
"and therefore civilization" (Score:3)
Re:"and therefore civilization" (Score:4, Insightful)
You are ignorant. Clustered systems such as do financial transaction require time synchronization. The collapse of those systems would indeed end modern civilization.
Re: (Score:2)
I get your point, but it would end *modern* civilization if it massively failed. Imagine a sudden and complete breakdown for scheduling all types of mass transit from trains to airline flights, for starters. Add all the financial systems having issues and ability to sign into pretty much any networked computer starting to fail as authentication servers drifted out of sync with each other.
Re: (Score:2)
They are important if you want a functional power grid.
Things like financial transactions, the power grid, and other utilities seriously should have consulted a metrology expert. They have a brittle dependence on synchronized time that only works because of robust time synch from NTP.
Re:"and therefore civilization" (Score:5, Insightful)
Historically, good communications and good timekeeping have been the backbone of civilizations from some of the earliest. Indeed, our very concept of time exists because the Babylonians needed it for coordination and bookkeeping. So, yes, good timekeeping has been critical for civilization for thousands of years. Modern civilization does indeed need this to be to sub-millisecond levels because communication operates at that speed.
Re:"and therefore civilization" (Score:4, Informative)
Re: (Score:2)
No. It is not critical to civilization.
Time synchronisation is essential and underpins everything you do on a daily basis. Free-run a clock on a mobile phone tower. After a day or so you won't be able to use wireless communication anymore. At all. Financial transactions, network equipment, authentication and credential management, distributed services you rely on for your day to day life all fail after a while when systems are not time synchronised.
Civlisation is defined as the current stage of social and cultural development in a particular era
Re: (Score:2)
It is not critical to civilization.
Well, it is critical for CDMA and 4G LTE mobile networks to work.
Very informative summary (Score:5, Funny)
tens of milliseconds -- a fraction of a blink of an eye.
Let us define the fundamental units, in Slashdot System of Units:
Time = blink of an eye
Length = school buses to be lined up end to end
Area = the Rhode Island (replaced the State of Delaware in 2012)
Volume = Olympic size swimming pools
Quantity of information = Library of Congress
Power = number of homes
Energy = we dont make a distinction between Energy, Power, Force, Momentum. If Roget's thesaurus links energetic, powerful, momentous, forceful as related, we define them all to be the same.
Re:Very informative summary (Score:5, Funny)
Power = number of homes
Energy = we dont make a distinction between Energy, Power, Force, Momentum. If Roget's thesaurus links energetic, powerful, momentous, forceful as related, we define them all to be the same.
Interesting. I like it.
"The avalanche came down upon the snowmobile with the momentum of 500 homes."
"Adding the turbo-charger to the car resulted in a corresponding increase in power of 1.5 homes."
"May the master plan community be with you"
Re: (Score:2)
Another really annoying Slashdot unit you forgot about: Temperature = Fahrenheit
Re: (Score:2)
Mass = pounds
The computer knows when it is at all times. (Score:5, Funny)
Re: The computer knows when it is at all times. (Score:1)
Re: (Score:2)
Re: (Score:2)
And movies [imdb.com] where you can't follow the plot even with the diagrams.
Ah, this was fun back then... (Score:4, Interesting)
NetWare
NDS
Epochs
Deutsche Bank used NetWare and NDS way back when. Time synchronization was crucial for banking, and similarly for NDS. Problems required Synthetic Time, allowing out of sync nodes to 'catch up', and usually declaring a new Epoch. This was nontrivial, and was usually done on Sunday evenings a little before midnight UTC.
Add in the time issues with the old NetWare IDE driver, and well we had fun, but it could be fixed at least.
FF to NTAS, and time sync was an unholy mess if it got wrong. ADS could wrestle you to submission or blind rage. It could be fixed if you rebooted enough times, until the patches finally got it right.
NTP. Anyone else remember the D-Link router fiasco?
Re: (Score:2)
NTP. Anyone else remember the D-Link router fiasco?
Of course! Remember the Netgear DDoS of The University of Wisconsin [dslreports.com]?
NTP Client (Score:3)
Many years ago, I discovered Socket Watch (swatch.exe). It was purchase-ware but relatively inexpensive; I recall it was about $10. The company that developed it -- Locutus Codeware renamed as Robomagic) was based in Canada. In an exchange of E-mails, the company said they would accept Can$10; so I had my daughter (living in Canada) pay them. I reimbursed her about US$7.50, which was the exchange rate for Can$10. That was at least 20 years ago. Through various Windows versions, I still use it.
Robomagic is no longer in business. Before liquidating, they made the last version (3.5b ) of Socket Watch freeware. Fortunately I archived a copy of its installer.
Socket Watch works off a list of over 100 time sources. Some are atomic clocks, and others synchronize to atomic clocks. On initializing after installation, the entire list is queried and scored. The responses are scored according to how quickly the sources respond; low scores are better than high. The list is then sorted by scores, lowest to highest. Socket Watch then takes a subset of top sources and periodically queries them; the size of the subset and the frequency of how often they are queried are user-settable. The responses are scored again, and the response with the lowest new score is used to reset my PC's clock. If any source in the subset has a higher score than a source not in the subset, it is replaced from the total list.
I setup Socket Watch to have a subset of 5 and to query the subset every hour. Since Robomagic no longer exists to distribute updates of the list of time sources, I occasionally visit http://support.ntp.org/bin/vie... [ntp.org] to update my total list of servers. Following rules stated at that site, I only use servers that are not restricted to geographical locations that are not mine and for which I do not need prior permission to query. Fudging one rule, I do include Stratum 1 (atomic clock) servers.
Re: (Score:2)
And when your boss asks why you're 10minutes late to a meeting please tell me your response is "Perfecting the accuracy of personal time keeping".
Re: (Score:2)
I retired 19 years ago, I no longer have a boss.
My interest in accurate time-keeping arose during my career as a software tester, testing the software used by the U.S. military to fly its earth-orbiting space satellites. Time and the earth's rotation are strongly related. The missions of those satellites required that time on the ground and in space be accurate to 1 millisecond and that the position of a satellite relative to the rotating earth -- the suborbital point on the earth's surface -- be accurat
So many dots (Score:1)
Re: (Score:2)
That's the Oxford Dot.
Re: (Score:2)
I'm fine with the dieresis in "Universal Coõrdinated Time"; the small caps for Arpanet and other agencies is weird, but OK; however, "three hundred and seventy-two days of twenty-three and a half hours each" was annoying to read.
But TIL that David Mills is blind :(
Re: (Score:2)
the dieresis in "Universal CoÃrdinated Time"
That's a New Yorker editors' thing that isn't catching on anywhere else.
New definition of "obscure" (Score:2)
obscure (ub-skewr), adj. - Some bit of information a blogger didn't know existed until he or she heard it mentioned by some other blogger on This Week in Technology.
Interesting (Score:2)
I'm now interested in knowing how hard realtime systems work, as they're generally single-process and that means you can't run anything other than NTP.
Actually quite simply (Score:2)
For such systems you typically get 2 signals (which may be derived form NTP) PPS and 10 MHz.
PPS is just a pulse per second. It indicates when a new second starts.
10 MHz is just that, a precise 10 MHz signal that allows you, in conjunction with the PPS signal to determine the proper time. Some systems also use different frequencies.
Inside of such a hard real-time system you can multiply the 10 MHz via PLLs to higher frequencies. For example if you want to timestamp an electrical impulse you can do the follow
Re: (Score:2)
At least in computing, Hard Realtime generally means that something is guaranteed to produce an output within a fixed amount of time of an event. That event could be a periodic interrupt (every 100ms for example) or within a certain amount of time of receiving a piece of data. This allows things to be predictable in their behaviour, which is critical to things like mechanical control systems. Back in the days of single core processors that did not have out of order execution, this was fairly simple. It has
...and no mention of NTPSec (Score:2)
It is amusing that they mention ESR at the start of the article but fail to mention he is part of the NTPSec fork.
Re: (Score:2)
It is amusing that they mention ESR at the start of the article but fail to mention he is part of the NTPSec fork.
Did you read a different article?:
Fuzzball Article (Score:1)
Re: Fuzzball Article (Score:1)
( The fact that it is written for non-nerds is lost on our great "editors". )
One of the healthier parts of the Net (Score:3)
After all there are thousands of NTP servers providing the correct time, and even without NTPsec it doesn't matter if some of them are wrong. While highly accurate time distribution over networks is a hard problem, NTP will get you down to much less than 100ms of error, even on consumer Internet lines.
NTP is also incredibly easy to set up. It's a single process, reading in a simple text file configuration. If you want to have your own Stratum 1 server the NTP-part of it is by far not the hard part of it.
Of course the "hyperscalers" try to break it by misshandling leap seconds because they think them not having to fix their software means that we all have to change the time, but NTP will survive that. Virtually any place that has the correct time publishes it via NTP. Virtually any university has their own public NTP setup. Even many people at home set up their NTP-servers to that they can take part in the global project that maintains the time on the Internet.
Re: (Score:2)
I run networks on the far end of a satellite link. Used to be geosynchronous, and is now over StarLink. It blows my mind at how well NTP handled that situation. When we were on geosynchronous, the jitter was actually really low, but the lag was long (around 550ms). Now with StarLink, the latency is usually around 40ms, but has very high jitter (can be up to 120ms). But NTP has no issue with that, and just happily keeps on trucking.
Re: (Score:2)
Yes, jtter actually is the higher problem than distance. As long as your distance is equally large in both directions you will get a very decent time.
BTW there are now fairly cheap GNSS based NTP server appliances available. (~120 Euros) I haven't tried them myself, but it's likely they work just fine.
Re: (Score:3)
It's healthy mainly because it's never really been under attack. Hackers haven't yet found a great way to exploit time servers to accomplish their goals, or maybe they just haven't realized just how much chaos they could cause by interfering with time server signals.
I'm reminded of a Start Trek TNG move where Captain Pickard suggested putting the Borg vessel to "sleep." Because it was a "low-priority subsystem" it wasn't heavily guarded. When the crew of the Enterprise took advantage of this weakness, the e
Re: (Score:2)
Well yes, but then again, unlike Andy Borg, NTP isn't centralized. It's designed to withstand falsetickers and clocks that are way off. It's designed for everything to go wrong, because things can go wrong.
Re: (Score:2)
I don't think it's so resilient, based on my experience. Many times I've seen a server's clock be wrong by a large amount because it was configured to use NTP from a server that no longer existed, and there was no alternate configured. It's often impossible to even log on to a server whose clock is far enough out of sync.
NTP is centralized in a fractal-type pattern. Everything initially comes from central servers like NIST / UNSO and a very small group of other primary servers. Each network then typically
Re:One of the healthier parts of the Net (Score:5, Informative)
I don't think you understand NTP.
You don't configure a server to use "one" other server. You configure it to use at least a pool of server. In fact that's the default configuration on most distributions. NTP does not work properly on a single time server. You might be confusing NTP with SNTP which is a rather brain dead simplification of NTP which is good enough for getting a very rough idea of the time. It is a "one shot" method to get an estimate of the time which is only usefull for things like mobile phones.
Actual NTP will actually adjust the clock rate of your machine. This way you essentially get a "calibrated" clock which will drift far less than usual. So even if we suddenly would loose all Stratum 1 servers, individual systems would drift apart much more slowly than without it. In fact any decent NTP network would probably stabilize quite near the actual time.
Also you missunderstand how NTP networks work. Yes, NIST and other organisations operate time servers, but virtually every country operates their own "national time service". Then again for around $130 you can buy a satellite navigation based time receiver with stratum 1 NTP server, or if you can live with a less accurate time, you can get DCF77 correlation receivers.
Yes, this all goes back eventually to a set of atomic clocks distributed all over the world, but that's outside of NTP, that's just UTC the globally coordinated time.
Re: (Score:2)
In the Windows server world, specific computers (domain controllers) are treated as time servers.
https://learn.microsoft.com/en... [microsoft.com]
Re: (Score:2)
Ahh, I see the missunderstanding. With "Net" I was talking about the Internet, not some local LAN.
Re: (Score:2)
It's healthy mainly because it's never really been under attack.
And if it were attacked what would you attack? You'd need to simultaneously knock both Stratum 1 servers off the internet (no easy feat since there are quite a lot of them and they are geographically diverse, and also block GPS the world over or knock all them off the internet as well.
And even when you do that a crisis doesn't happen instantly. Clocks free running drift in a matter of days, not minutes. Systems will keep functioning long enough for alternate time sync to occur. You could literally unplug me
Re: (Score:2)
And if it were attacked what would you attack?
That's kind of like asking, "If you were to attack a server monitoring system, what would you attack?" Remember Solar Winds? https://www.techtarget.com/wha... [techtarget.com] The hackers didn't just mess with the monitoring itself. Once they hacked the system, they basically could execute whatever code they wanted to.
Hackers find very creative ways of getting into systems, that no one anticipates ahead of time. They would certainly not limit themselves to "messing with" the time synchronization itself.
Re: (Score:2)
NTP will get you down to much less than 100ms of error, even on consumer Internet lines.
NTP will get you an order of magnitude better than that over a consumer internet line... You really need a highly congested network or for the error to get anywhere near the high end of double digits.
Re: (Score:2)
Well it depends on the quality of your consumer line. Yes it can filter out jitter fairly well, but having different delays on up- and downstream will introduce a static error.
Also particularly wireless providers often do have roudtrip times in the order of many seconds.
No problem in the summary (Score:2)
The headline mentions a "thorny problem", but the summary drones on about internet time synchronization without saying what's wrong. Maybe it's in the article, but it's a serious Slashdot failure if I have to read more than the summary before mouthing off in the comments.
Re: (Score:2)
There is no problem. This is a stupid dross-for-the-masses article that tries to imply there's a problem where there isn't one.
Re: (Score:2)
So many dots (Score:2)
If we're going to refer to NTP as N.T.P. I guess we better starting doing the rest too:
I.P.
T.C.P.
U.D.P.
S.M.T.P
S.I.P.
H.T.T.P.
My god...can we please ban forever whoever wrote that, the company they work for, the slashdot editor that approved it, and the user that submitted it?
Re: (Score:2)
Add a few more... /. is so A.R. about CAPS
ICMP
IGMP
LDP
RIP