Windows Feature That Resets System Clock Based On Random Data Is Wreaking Havoc (arstechnica.com) 119
An anonymous reader quotes a report from Ars Technica: A few months ago, an engineer in a data center in Norway encountered some perplexing errors that caused a Windows server to suddenly reset its system clock to 55 days in the future. The engineer relied on the server to maintain a routing table that tracked cell phone numbers in real time as they were being moved from one carrier to the other. A jump of eight weeks had dire consequences because it caused numbers that had yet to be transferred to be listed as having already been moved and numbers that had already been transferred to be reported as pending. "With these updated routing tables, a lot of people were unable to make calls, as we didn't have a correct state!" the engineer, who asked to be identified only by his first name, Simen, wrote in an email. "We would route incoming and outgoing calls to the wrong operators! This meant, e.g., children could not reach their parents and vice versa."
Simen had experienced a similar error last August when a machine running Windows Server 2019 reset its clock to January 2023 and then changed it back a short time later. Troubleshooting the cause of that mysterious reset was hampered because the engineers didn't discover it until after event logs had been purged. The newer jump of 55 days, on a machine running Windows Server 2016, prompted him to once again search for a cause, and this time, he found it. The culprit was a little-known feature in Windows known as Secure Time Seeding. Microsoft introduced the time-keeping feature in 2016 as a way to ensure that system clocks were accurate. Windows systems with clocks set to the wrong time can cause disastrous errors when they can't properly parse time stamps in digital certificates or they execute jobs too early, too late, or out of the prescribed order. Secure Time Seeding, Microsoft said, was a hedge against failures in the battery-powered on-board devices designed to keep accurate time even when the machine is powered down.
"You may ask -- why doesn't the device ask the nearest time server for the current time over the network?" Microsoft engineers wrote. "Since the device is not in a state to communicate securely over the network, it cannot obtain time securely over the network as well, unless you choose to ignore network security or at least punch some holes into it by making exceptions." To avoid making security exceptions, Secure Time Seeding sets the time based on data inside an SSL handshake the machine makes with remote servers. These handshakes occur whenever two devices connect using the Secure Sockets Layer protocol, the mechanism that provides encrypted HTTPS sessions (it is also known as Transport Layer Security). Because Secure Time Seeding (abbreviated as STS for the rest of this article) used SSL certificates Windows already stored locally, it could ensure that the machine was securely connected to the remote server. The mechanism, Microsoft engineers wrote, "helped us to break the cyclical dependency between client system time and security keys, including SSL certificates."
Simen had experienced a similar error last August when a machine running Windows Server 2019 reset its clock to January 2023 and then changed it back a short time later. Troubleshooting the cause of that mysterious reset was hampered because the engineers didn't discover it until after event logs had been purged. The newer jump of 55 days, on a machine running Windows Server 2016, prompted him to once again search for a cause, and this time, he found it. The culprit was a little-known feature in Windows known as Secure Time Seeding. Microsoft introduced the time-keeping feature in 2016 as a way to ensure that system clocks were accurate. Windows systems with clocks set to the wrong time can cause disastrous errors when they can't properly parse time stamps in digital certificates or they execute jobs too early, too late, or out of the prescribed order. Secure Time Seeding, Microsoft said, was a hedge against failures in the battery-powered on-board devices designed to keep accurate time even when the machine is powered down.
"You may ask -- why doesn't the device ask the nearest time server for the current time over the network?" Microsoft engineers wrote. "Since the device is not in a state to communicate securely over the network, it cannot obtain time securely over the network as well, unless you choose to ignore network security or at least punch some holes into it by making exceptions." To avoid making security exceptions, Secure Time Seeding sets the time based on data inside an SSL handshake the machine makes with remote servers. These handshakes occur whenever two devices connect using the Secure Sockets Layer protocol, the mechanism that provides encrypted HTTPS sessions (it is also known as Transport Layer Security). Because Secure Time Seeding (abbreviated as STS for the rest of this article) used SSL certificates Windows already stored locally, it could ensure that the machine was securely connected to the remote server. The mechanism, Microsoft engineers wrote, "helped us to break the cyclical dependency between client system time and security keys, including SSL certificates."
don't use windows servers. (Score:5, Informative)
period
Re: (Score:2)
Linux isn't any better. When you actually true to use it for stuff, you find it has just as many random issues and bugs as Windows.
Let me give you a recent example. Had a Linux machine that was hammering its SSD constantly with writes, to the point where database writes were taking too long and the buffer was overflowing. iotop couldn't show what was writing to disk because the kernel didn't support some feature it needed, so all it could do is tell us that there was data being written but not from any of t
Re: (Score:2)
Why was a headless workstation running a window manager? Why did a headless workstation even have a window manager installed?
Re: (Score:2)
It isn't always headless. Depends on the configuration of the product.
Re: (Score:2)
News flash-- Just because a window manager is installed, doesn't mean it's enabled. You misconfigured a headless system, and then wondered why you had issues. It's literally one command. If you don't need a graphical environment, DON'T RUN A GRAPHICAL ENVIRONMENT.
So, did you report this one weird bug?
Re: (Score:2)
Haven't finished diagnosing it yet. The idea is to have one set of software for both systems, to avoid supporting two different ones. Render the display even if there is nothing connected. Apparently that causes some kind of problem that results in endlessly writing data to disk.
Re: (Score:2)
On another note, whatever your specific needs, most headless systems neither need nor want a graphics display/window manager installed. But in Windows, you're getting one whether you want it or not. And if you have to access it remotely, here's the entire GUI to be crammed down your network connection. I could easily manage a Linux (or UNIX) system over dialup if I needed to.
Re: (Score:2)
Re: (Score:2)
You can have the same software on both systems, just change the default to "graphical.target" or "multi-user.target", depending on whether you expect to need a GUI. Why waste resources on not just the graphical environment, but you're also causing the display manager / driver to hang looking for a monitor that doesn't exist?
The first thing I suggest for people learning Linux is that they get used to the idea that Linux is Not Windows. Don't treat them the same, and you'll be much happier.
Re: (Score:2)
It's very useful for debugging and diagnostics to be able to connect a monitor in the field. Were were contemplating adding an external HDMI port for that purpose.
It doesn't seem unreasonable to expect the driver to fail in a way that doesn't result in a loop where it keeps trying to restart and then writes a failure log.
Re: (Score:2)
Fine. Connect a monitor. Log into a shell from the (text) console. Launch the graphical interface:
https://www.redhat.com/sysadmi... [redhat.com]
You are launching a process, which is looking for a monitor, so it can auto-determine the capabilities, so you can get a working display. If it doesn't have a display, it's going to have a problem.
My Windows 11 box gets a bit persnickety if I boot it without a monitor connected, for the same reason (I use a KVM, and if the Windows port isn't active, then the monitor might co
Re: (Score:2)
That does sound bad and dumb. But why do you have a graphic login configured on a headless server?
Re: (Score:2)
It's not a server, it's an appliance. Some variants have a display, some don't.
Re: (Score:2)
If it works with the display operating, then changing it to have the display not operating is only going to stop it working. There's no point, it works and we can support 1 version of the software.
Or at least that's what should happen. Not having a physical display connected should make no difference. It should render and output the video signal.
Re: (Score:2)
Well, yeah.
Leave it to Microsoft (Score:5, Funny)
... to totally screw up the implementation of an idea or protocol while confidently explaining that they know what they're doing.
Re: (Score:2)
It's a tricky problem. Meaning there are already lots of papers and patents about this issue. Randomly changing time though wasn't in any of my patent searches that I recall.
Re: (Score:2)
There are also several free ntpd implementations and there is an ntp RFC that explains how to do it right. By now I am convinced that MS "coders" cannot actually read.
Re: (Score:2)
So, they do. In AD they use a Kerberos-secured implementation, which is where the glitch seems to be.
Re: (Score:3)
There are also several free ntpd implementations and there is an ntp RFC that explains how to do it right.
Window's implementation follows the ntp RFC perfectly. Unfortunately ntp doesn't account for situations where the clock significantly deviates from network time. The issue here is in an implementation of a correction for something the RFC doesn't cover.
By now I am convinced that MS "coders" cannot actually read.
You have convinced us all that you can't actually read. Or at least refuse to. Your entire string of posts on this topic show an astonishing lack of understanding of what is going on in Windows, but you chose to comment anyway. Forgive us all for not taking y
Re: (Score:2)
Dear Editors (Score:5, Insightful)
That isn't a summary. That's a excerpt, one that provides no meaningful information on the issue being discussed.
This is your daily reminder that Slashdot editors could be easily replaced with ChatGPT. ... Early versions of ChatGPT.
Re:Dear Editors (Score:5, Funny)
You must be new here.
Re: (Score:2)
Re:Dear Editors (Score:5, Insightful)
That isn't a summary. That's a excerpt, one that provides no meaningful information on the issue being discussed.
Right you are. And it is not a well written article that surfaces the state of knowledge about the problem in an effective manner either.
If you read this article (and this is an important one) you find that the reason that this Secure Time feature fails catastrophically is not publicly known yet, due to the secret proprietary nature of Windows and Microsoft.
I get a real feeling of having been here before thinking about the early days of network time protocols. It was imperative to avoid having systems make huge time jumps, and a lot of effort was placed in developing algorithms that could only shift time values gradually. It appears Microsoft's engineers were out reinventing the wheel without looking at how actual wheels currently in use work.
Re: (Score:2)
Tao of Wheel (Score:2)
It has three sides, one of which loops around and reconnects at the other end, the essence of wheel. With no corners the wheel goes along the ground smoothly. Adding a dowel at a right angle to the middle of the wheel, we can attach the wheel to other objects and then those objects can go smoothly along the ground in travelling. Adding in pairs along the dowel increases stability. This is the nature of the wheel. Wheel provides.
Next up: Tao of Axl: The Jungle Welcomes You
Not sure what they were thinking (Score:2)
Re:Not sure what they were thinking (Score:5, Funny)
This is just plain stupid.
Yes. Yet another poorly understood, poorly implemented, and poorly tested Windows feature from Microsoft.
Re: (Score:2)
It is. But I can tell you what they were thinking: Nothing. That is the usual mode of MS. I mean, this is the company that recently has a cloud master key stolen while thinking their future is the cloud. MS screws up and screws up and screws up some more.
Surely you recognize that the solution is bullshit (Score:5, Informative)
Because Secure Time Seeding (abbreviated as STS for the rest of this article) used SSL certificates Windows already stored locally, it could ensure that the machine was securely connected to the remote server.
The only reason time has any influence on the machine's ability to make secure connections is that the wrong time can be used to make expired certificates look valid, and obviously that applies to the Windows certificate store too. You didn't break the cyclic dependency, "you choose to ignore network security or at least punch some holes into it by making exceptions" which you claim you wanted to avoid.
Re: (Score:2)
Yeah, I had to read that twice just to make sure I wasn't being stupid. If a device is "not in a state to communicate securely over the network" then how could it communicate securely over the network?
Re: (Score:2)
Look, so the thing about that is...shut up!
The random time is NOT from Windows (Score:2, Interesting)
The random time is from an implementation of the SSL protocol external to Windows that fills the informational SSL field "ServerUnixTime" with a random value rather than the actual time.
The feature that takes this bad time and uses it in the OS has been a feature of Windows since 2015 and has been the default since 2016. So the whiner is definitely seeing an edge case. Trying to improve security by using protocols in unintended-by-desi
Re: The random time is NOT from Windows (Score:5, Insightful)
So let me get this straight.
For seven - possibly eight - years, Microsoft has been setting, as default, a function which relies on specific snippets of outside information from a network ecosystem. Despite the published standard invovled, it's also *well known* common practice that in some cases, those data snippets are deliberately randomized garbage data.
It is also now fairly well documented that in certain cases, using those randomized values in the now-default function can result in the OS autonomously setting the system time to wildly incorrect values with cascading problematic consequences.
Microsoft designed the feature, has a reasonable responsibility to be aware of the de facto forms of the data they're utilizing, decided to make this feature enabled by default, has been informed of the flaws in practice their feature has displayed, and has left it there for seven - possibly eight - years.
But it's not Microsoft's fault.
Uh huh.
But this is somehow not
Re: The random time is NOT from Windows (Score:5, Insightful)
If Microsoft was really worried about expired certs compromising TLS security they would:
1) Stop downloading trusted certs on first access from Microsoft's mothership. (They used to preseed the cert bundles like everyone else. Then they switched to this stupid method of using the untrusted network connection to download a "trusted" cert to validate the untrusted network connection.)
2) Actually keep a record of what certs have expired in the trust store. I.e. If the clock hasn't been manually reset within the last X days, and it increments past the expiry time for a cert, mark that cert as expired in the trust store. If a new connection attempts to validate using a cert that has been marked in this manner, refuse it outright. Don't even bother checking the RTC. This way a malfunctioning clock, accidental or otherwise, won't reduce system security when validating TLS connections against known expired certs. (Rationale: Time doesn't flow backwards in the normal use cases for TLS. The manual clock reset check will catch most other benign RTC changes. Anything else, is something the sysadmin would need to be involved in fixing anyway. Although there should also be a way for the sysadmin to remove this mark in the trust store just in case.)
Re: (Score:2)
The standard has this to say about the
Re: (Score:2)
I doubt the engineer that thought up this solution knew that some implementations were setting a field named "UnixTime" to a random number.
No, if you look at Microsoft's doc on this "new" feature, they specifically call out the fact that you can, in fact, get a total whack-job of a timestamp.
The ServerUnixTime is supposed to be the current system time on the server, but it can also be set to a random value by some SSL implementations. We have observed that most servers provide a fairly accurate value in this field and the rest provide random values. We use this data field assuming it is somewhat accurate but can also be incorrect.
Re: (Score:2)
Re: The random time is NOT from Windows (Score:2)
No, we're blaming Microsoft for using a data field from the network ecosystem that is plainly, demonstrably NOT reliable, but treating it as sufficiently trustworthy that their time-verification system is fragging system clock settings without warning because of it.
Also we have movie sign, try to keep up.
Re: (Score:2)
The random times are not entirely random. They are the current time +/- a certain amount.
The idea is that by setting the clock to within at least a few months of the correct time, the server will then be able to contact an NTP or PTP server and get an accurate time value. It's a way to overcome the problem of the clock resetting to 2000/1/1 or whatever the default is, without having to blindly trust the time server.
It's a reasonable way of handling the problem, and appears to have been working just fine for
Re: (Score:2)
Microsoft designed the feature, has a reasonable responsibility to be aware of the de facto forms of the data they're utilizing
Which they did. The system does not just blindly assume the time it is given from a single source. Here's the thing, before you criticise ask yourself why a system that has been the default for 7 years and existed for even longer than this, only now has been identified as a problem, and why the discussions around this system seem to be a kind of new revelation to everyone. The fact that this story came out in 2023 instead of 2015 shows how reliable it is and how well it has worked.
Re: (Score:2)
X509 pubic key certificates
Well, there's your problem right there.
Re:The random time is NOT from Windows (Score:5, Insightful)
I would argue that if, instead of looking at the town clock, you ask the town drunk what time it is, it is YOUR fault if you're late.
Same applies to MS.
Re: (Score:2)
Except they don't ask the town drunk. They ask everyone in the town they see and then determine the most likely correct answer. There's a reason we're discussing this new revelation 7 years after it became the default deployed to 100s of thousands of servers around the world. It works exceptionally well in nearly every situation.
Re: (Score:2)
Apparently, they do sometimes ask the town drunk rather than the defined standard reference.
Re: (Score:2)
Re: (Score:2)
However, there is no requirement for them to do this. They could fill the time with some random information if they felt like it, or intentional to a highly precise time _exactly_ ten years in the future, and nobody c
And people worried about the Y2K bug! (Score:5, Insightful)
This is pathetic. Microsoft has a bug in their code that can get the incorrect time/date, that's been in their server products since 2016. And production servers around the world have randomly suffered database corruption and other issues due to it. But as of now, they're not quite sure what's wrong.
Seems to me this is another classic case of trying to over-engineer things? If I configure a server to request the date/time from a standard nntp time service, it should simply perform that function. If it turns out it's getting bad data when doing that? Then I should be reasonably certain I've got a problem relying on that time server or pool and need to change it to grab it elsewhere, right? That's how Unix boxes have handled this for decades.
But no... MS had to go and do some weird thing where they poll time info from multiple sources and try to estimate the "most correct" time from some kind of average of the results that come back?
"To prevent STS from resetting system clocks based on data provided by a single out-of-sync remote server, STS makes randomly interspersed SSL connections to multiple servers to arrive at a reliable range for the current time. The mechanism then merges the ServerUnixTime with the OCSP validity period to produce the smallest possible time range and assigns it a confidence score. When the score reaches a sufficiently high threshold, Windows classifies the data as an STSHC, short for Secure Time Seed of High Confidence. The STSHC is then used to monitor system clocks for “gross errors” and correct them."
IMO, if you're going to even do all of this, you should at LEAST inform users that's what the service does when people are in the screen to configure it.
Re: (Score:3)
Seems to me this is another classic case of trying to over-engineer things? If I configure a server to request the date/time from a standard nntp time service, it should simply perform that function. If it turns out it's getting bad data when doing that? Then I should be reasonably certain I've got a problem relying on that time server or pool and need to change it to grab it elsewhere, right? That's how Unix boxes have handled this for decades.
Indeed there are various sorts of NTP services specifically designed for providing various degrees of accurate and secure time. People who care about this use one, and should be capable of fixing it if it breaks. This STS is not a "feature". By design it is intended to hide a broken NTP implementation.
Like that stuff you can put in your cooling system to quick seal a leak, or the heavy oil they use to hide mechanical engine noises. If a mechanic did that I would consider them sketchy, not over-engine
Re: (Score:2)
Try this from the home user perspective though. They close the lid on their laptop, and it goes to sleep. The clock is maintained by the RTC chip on the computer, which can vary in accuracy from sundial to guessing. There are computers whose clock is basically nonfunctional. Then they wake their computer up and is Windows supposed to trust that the clock is accurate?
Just because you haven't experienced it doesn't mean you know all those billions of PCs around the world won't experience it - and chances are,
re: home user perspective (Score:3)
I'm not sure I see how any of that applies here, though? The home user closing a laptop lid and sleeping the PC isn't anywhere near the same situation as Enterprise "always on" servers hosting databases using the Windows Server product.
The home user with a PC with a nonfunctional clock because the RTC battery died simply needs to replace their bad CR2032 lithium battery (or equivalent). As far as I'm concerned, it's not the OS's responsibility to try to compensate for physical hardware issues. And if they c
Re: (Score:2)
Been forever since I've had a cmos battery go bad.... do current/modern systems still break out of the POST to prompt for valid date input?
Re: (Score:2)
The home user closing a laptop lid and sleeping the PC isn't anywhere near the same situation as Enterprise "always on" servers hosting databases using the Windows Server product.
Servers aren't always on. Services are always available. The servers that provide those services spin up and down constantly fall over from one hardware to the other, or are spread across VMs which sometimes may be spun up and spun down for a large variety of reasons, in each case requiring some time synchronisation.
This isn't 1998. Five Nines isn't achieved by handing IBM a million dollars and having them deliver you a mainframe. Just because business services are always available doesn't mean servers them
Re: (Score:2)
Try this from the home user perspective though. They close the lid on their laptop, and it goes to sleep. The clock is maintained by the RTC chip on the computer, which can vary in accuracy from sundial to guessing. There are computers whose clock is basically nonfunctional. Then they wake their computer up and is Windows supposed to trust that the clock is accurate?
So maybe for some subset of people running Windows Server on their laptop I guess Microsoft has saved them from being sad.
After all, you have untrustworthy clock, no way of talking to a trusted NTP server (might be filtered by an overzealous firewall, it happens way more often than you might think), so you need a way to get some idea of the approximate time.
You need to fix your NTP. And your firewall. Maybe a new CMOS battery. You could set your laptop manually to your phone's NITZ time, which should be close enough to help you fix all the broken things.
SSL seems appropriate since you can usually get through a firewall that way.
They should team up with one of those remote access services that specializes in getting around firewalls.
Re:And people worried about the Y2K bug! (Score:4, Insightful)
But if you can't tell if a certificate is valid, why should you believe anything a remote server tells you? There's a chicken-and-egg problem here.
If the OS is unsure about what the time is, the only secure option is to refuse all TLS/SSL negotiations and to log a critical failure and complain loudly.
There are counter-measures you can take to mitigate the problem. For example, once a minute you could record the maximum-seen current time on non-volatile storage and then check to see that what the clock says is at least that large before accepting a certificate. This is not bulletproof, but at least it won't result in clock-resetting insanity. (Of course, you will need a mechanism to manually update the non-volatile value if you really do need to move the clock backward for some reason... but that should never be an automatic process.)
Re: (Score:2)
But if you can't tell if a certificate is valid, why should you believe anything a remote server tells you? There's a chicken-and-egg problem here.
Assume I need the exact time to verify certificates. So what I can do is contact a time server, and get a time, ignoring the fact that I cannot verify the certificate. The reply is either a correct time, or an incorrect time, or some rubbish possibly intended to crash my computer. I don't and cannot know, because I cannot verify the certificate.
But I can write code that ignores any rubbish and only reads time stamps. So I use that time stamp to try to connect to a different server. If the time stamp was
Re: (Score:2)
NTP's security is flawed and basically worthless. Anyone inside your network, or who can intercept your traffic to public NTP servers, can give you the wrong time. The only way around it is to validate the time server some other way, i.e. via a secure tunnel, and then piggyback NTP on top of that. Accuracy will be reduced, and it's not a standard.
As such, the recommended way of doing it is to validate the NTP data. If it is trying to adjust your clock more than seems likely, like say by an hour when it shou
Re: (Score:2)
Anyone inside your network, or who can intercept your traffic to public NTP servers, can give you the wrong time.
Anyone inside your network can do all sorts of malicious things.
If it is trying to adjust your clock more than seems likely, like say by an hour when it should only have drifted a fraction of a second since the last check, you know something is up.
I think NTPs default panic threshold is 1000 seconds, which seems exceedingly generous to me.
Re: (Score:3)
"To prevent STS from resetting VCR clocks based on data provided by a single out-of-sync tape, STS reads randomly interspersed timestamps from multiple tapes to arrive at a reliable range for the current time. The mechanism then merges the iframe offset with the estimated tape length to produce the smallest possible time range and assigns it a confidence score. When the score reaches a sufficiently high threshold, the VCR classifies the data as an STSHC, short for Secure Tape Seed of High Corruption. The STSHC is then used to monitor VCR clocks for “gross errors” and correct them by setting the clock to flash 12:00 AM."
Re: (Score:2)
And production servers around the world have randomly suffered database corruption and other issues due to it. But as of now, they're not quite sure what's wrong.
[Citation required] What we're actually seeing is a time system deployed in 100s of thousands of servers around the world being in place for over 7 years resulting in one person reporting some problem in an edge case.
If I configure a server to request the date/time from a standard nntp time service, it should simply perform that function.
Funny you should say that. You realise by default if ntpd gets a time that is more than 1000s off it will restart its own demon, and not make a correction right? This solution is Microsoft's attempt to validate the incorrect time it receives.
Authoritively trusting anyone random (which is what y
Really? (Score:2)
Re: (Score:2)
You thought you were just doing that for better certificate management, ability to slow roll a TLS patch on MSFT and resiliency? If you are counting just on windows system time to be correct in anything impo
Re: (Score:3)
I have worked multiple instances where the time is wrong on underlying ESXi hosts and enterprises are shocked when the overlying VMs take the wrong time from the underlying VM host.
"Why would it do that? Time synchronization is disabled?!!"
Time synchronization keeps the virtual clock in sync with the vm host clock on a running machine. When a machine is paused and resumed or cold booted it still uses the underlying VM host as the c
Re: (Score:2)
I have worked multiple instances where the time is wrong on underlying ESXi hosts and enterprises are shocked when the overlying VMs take the wrong time from the underlying VM host.
"Why would it do that? Time synchronization is disabled?!!"
Time synchronization keeps the virtual clock in sync with the vm host clock on a running machine. When a machine is paused and resumed or cold booted it still uses the underlying VM host as the clock source.
(When I explain this, I get one of two responses. It's either an argument that can't possibly be true or the sound of someone forehead slapping because I've answered an annoyingly intermittent and non-reproducible issue they've struggled with for years.) :)
Which is why your OS should be getting it's time source from a central source rather than the underlying hardware. What you've said is true (as another VMWare engineer, I'll happily back you up) but it's also true for bare metal Linux and Windows. If you're depending on local hardware clocks (or hypervisor clocks) to be in sync for time sensitive applications, you're an idiot.
So when a server boots up, reconnects to the network it should correct it's own time from the central source.
I have a similar i
Re: (Score:2)
So when a server boots up, reconnects to the network it should correct it's own time from the central source.
And what if the time is bigger than some threshold causing you to question whether that central source is accurate? If I told you it is now January 3rd 2024 would you accept that? What if I were your father or friend, would you trust me then? Of course not.
The people who understood time synchronisation understand that you can't trust a time source if it is wildly different to your own which is why ntpd doesn't attempt to correct time differences beyond 1000s by design, and why WTS doesn't correct time diffe
Re: (Score:2)
My experience is, with the caveat that I'm on the Windows side not the VmWare, irrespective of configuration, the host always seeds the virtual hardware clock when a VM is cold-booted or pause/resumed.
Re: (Score:2)
Depending on time keeping by any PC is a pretty bad idea. The things depend on an unmonitored watch battery. You can buy various network time servers for a few hundred bucks, and could probably build a really nice one for under $100.
Re: (Score:2)
Re: (Score:2)
Sure. If you really wanted to be sure, a linux machine or micro-controller on a monitored battery backup that compared a couple of NTP servers and GPS would do well. Have three if it's super important.
Re: (Score:2)
The way I do it is to use the stock Linux NTP software on my Linux servers and configure them to use a few trusted NTP servers from government organisations. It has served me well. The stock Linux NTP software is mature and well maintained. I would rather gamble my business on such software than something closed sourced from Microsoft that was designed with maximum profit as the motive, rather than maximum reliability.
Your stock Linux NPT software will fail if the time differs by more than 1000s and gives you no recourse other than manual intervention. Microsoft's attempts to determine the correct current time in situations where all traditional methods (including your Linux one) fails is done precisely to improve reliability, because especially for something like AD controllers reliability = profit.
If you're using NTP from a trusted server you're not interested in reliability. If you were interested in reliability you'd
Re: (Score:2)
"AD controllers reliability = profit" is true mainly in the sense it reduces the number of lawsuits. You are overlooking the the number one most fundamental difference between why code is written for Microsoft Windows vs Linux. Windows software is not written to make PCs useful, it is written to maximise profit to shareholders. Anything Windows does on a
Re: (Score:2)
I don't think so. They have merely shifted the focus of their abysmal incompetence and so some things have stabilized a bit. You may also have overlooked all the ways in which Win11 is _worse_ than Win10. For MS in general, just recently this "cloud" company had a cloud master key stolen from them. It really does not get much worse than that. They are also not explaining how that key (which should have only ever been inside a HSM) got stolen basically means they screwed up extremely. Oh, and that key was ex
don't VMWARE and other VM hosts set the clock in (Score:2)
don't VMWARE and other VM hosts set the clock in the VM?
also sending an RTC battery issue from an windows server in an VM may lead to it being set aside as some odd running in an NON MS VM issue.
Some feature (Score:2)
At least it is only 55 days (Score:2)
When Windoze does it to me, it puts me 30+ years ahead, that breaks everything. Usually after an update, then locks the ability to change the time.
Well that will teach them to not use local clocks (Score:2)
Use a time server, people (Score:2)
Just use Airplane mode (Score:2)
Why is there Airplane mode in a server? Good question.
Is this exploitable, I wonder? (Score:2)
It seems to me that if malicious servers gang up to provide misleading (but not random) data in the ServerUnixTime field, this could be exploitable... at the very least to DoS someone and maybe worse.
MS cannot even get basic things right (Score:3)
Crond has solved this a long, long time ago: Keep offset and drift measures, adjust as needed and do not accept large changes from the ntp server (30 minutes being the default, I think) unless the sysadmin specifically requests them, e.g. via ntptime.
Never had this problem on Linux or any other Unix-like system. But MS is just incapable of even getting simple things right and in this case with decades old, reliably working implementations around they could fully legally just have copied.
Re: (Score:2)
Sorry, wrong reference. That should be ntpd.
Re: (Score:2)
Never had this problem on Linux or any other Unix-like system.
Because Linux / Unix fail to set the time. What you're describing is actually how Windows time works as well. The difference is while Linux/Unix will result in time drifting if the clock is wrong requiring manual intervention, Microsoft developed a way to handle that edge case.
Never had this problem on Linux or any other Unix-like system.
Nearly everyone on the planet has not had this problem on a Windows system either. This bug has been in place as a default for all Windows servers for over 7 years and yet we weren't talking about it until now for a good reason.
But MS is just incapable of even getting simple things right and in this case with decades old, reliably working implementations around they could fully legally just have copied.
Except
On my iPhone (Score:2)
The next step would then obviously be to change the clock back to "automatic" and see what happens.
From the article, do I get this right th
Re: (Score:2)
So why couldn't they do that?
Because calling an administrator out to fix something is not the way to achieve high reliability.
Because the time server needed an accurate time to allow the connection, so if you don't know the time then you can't ask the time server for the time?
Yes, all time servers work this way. Ntp will not make a correction if it is more than 1000s out from the system clock. You'll need to force the correction manually. Windows defaults to 15 hours for this.
But you got the causality wrong. It wasn't that the computer got to 55 years and couldn't correct time. It's that the computer couldn't correct it's time, then fell over to STS which due to a bug / design flaw m
GPS as time source (Score:2)
Now GPS seconds wrap arround every 20 years. Fix: If you built the software on March 20th 2023, assume that the time is from March 19th 2023 to M
Re: (Score:2)
You could always use GPS.
Of course you could. But the reality is most networks don't have such an authoritative time server.
so your time is somewhere from the satellite's time stamp to time stamp + 20 milliseconds.
If that's how you think this works, then GPS itself wouldn't be functional. Every GPS receiver does what is known as common view time transfer. The accuracy is the timestamp is the satellite timestamp +/- 6nanoseconds. The clever math behind it results in a time source which is incredibly accurate.
Now GPS seconds wrap arround every 20 years. Fix: If you built the software on March 20th 2023, assume that the time is from March 19th 2023 to March 19th 2043. Make sure you update with a new build within 19 years.
Funny story I couldn't figure out why my old Nikon camera kept resetting the clock to sometime in the early 2000s.
Re: GPS as time source (Score:2)
But this super accurate time stamp doesnâ(TM)t tell you _your_ time. But a single satellite gives you some information: The fact alone that you reiceived a time stamp proves your distance is less than North Pole to equator, so the signal travelled only a few dozen milliseconds. So y
Data Point (Score:5, Informative)
Have you ever tried to solve a problem and created a different problem? That.
Domain joined machines need to have the time accurate to within 5 minutes of the domain controller they contact for Kerberos to work properly. To facilitate this, domain members use NT5DS to synchronize time with domain controllers. Occasionally the DC will develop some emotional problem and their time will go wonky. E.g. the CMOS battery fails or the DC syncs time with the underlying VM platform that has bad time, etc.
In this scenario potentially thousands of clients will suddenly have the wrong time. To prevent this, they added limits to how far out of bounds the time can be and still be considered valid. (MaxPosPhaseCorrection & MaxNegPhaseCorrection). These default to 24 hours. FWIW, these correlate to limitations suggested by several industry security baselines.
Skip ahead a bit and assume some absolute cod-flogging genius created a system that didn't have a proper real-time clock implementation and the time went completely knobby when you left it in sleep for too long. If the clock is out by more than 24 hours, now that system has no mechanism for correcting the time. Shit.
The solution is to get time from some other source, but how can I discover time if I don't trust NTP or SNTP? Every SSL/TLS connection sneaks out some time data, so lets use that! Ok, but I don't trust a single source. Fine, we'll aggregate a bunch of time data from several SSL/TLS connections and use that as a secure time seed.
Right, that's how you get here.
More Data:
https://learn.microsoft.com/en... [microsoft.com]
More more data (including how to turn it off):
https://learn.microsoft.com/en... [microsoft.com]
Re: (Score:3)
if I don't trust NTP or SNTP
I, most admittedly, am not a server admin. Why would such a person not trust NTP? That just seems silly, from a lay-person's perspective.
Re:Data Point (Score:5, Informative)
Normally I do trust it, but the Max[Pos|Neg]PhaseCorrection setting says "Don't trust network time if it's out by more than 24 hours."
Re: (Score:2)
Re: (Score:2)
Local GPS disciplined time source. If you can't trust yourself, who can you trust?
Re: (Score:2)
I, most admittedly, am not a server admin. Why would such a person not trust NTP? That just seems silly, from a lay-person's perspective.
NTP is a protocol that was created mostly for time stability. It was not actually intended to blindly set time of a system. And if you think about it you would react the same way. Go up to a person in the street and ask them what the date is. If they tell you it's 17th August 2023 you'd thank them and be on your merry way. If they tell you it's 18th August 2023 you'd probably accept it as well. If they tell you it's 3rd January 2024 you wouldn't put any faith in it because as far as you can remember you hav
Re: (Score:2)
Re: (Score:3)
Meanwhile, for the last 30 years or so, Unix admins have simply queried a known good time server and hard set the time on boot to avoid this problem.
Re: (Score:2)
Indeed. But MS is the crap-show that re-implements Unix as poorly as possible and with as little insight as possible.
Re: (Score:2)
"The ServerUnixTime is supposed to be the current system time on the server, but it can also be set to a random value by some SSL implementations. " https://learn.microsoft.com/en... [microsoft.com]
So, if the service it's trying to derive the time from is not behaving according to spec, the problem may arise.
Re: (Score:2)
Re: (Score:2)
I wasn't speaking of a manual procedure, it is typically in the init scripts.
This is MS ignoring the designated known good time server or a pool of the same in favor of less authoritative sources which, as it turns out don't even promise to deliver anything like the correct time.