Facebook Engineers Develop New Open Source Time Keeping Appliance (techcrunch.com) 99
Ron Miller, writing for TechCrunch: Most people probably don't realize just how much our devices are time driven, whether it's your phone, your laptop or a network server. For the most part, time keeping has been an esoteric chore, taken care of by a limited number of hardware manufacturers. While these devices served their purpose, a couple of Facebook engineers decided there had to be a better way. So they built a new more accurate time keeping device that fits on a PCI Express (PCIe) card, and contributed it to the Open Compute Project as an open source project. At a basic level, says Olag Obleukhov, a production engineer at Facebook, it's simply pinging this time-keeping server to make sure each device is reporting the same time.
"Almost every single electronic device today uses NTP -- Network Time Synchronization Protocol -- which you have on your phone, on your watch, on your laptop, everywhere, and they all connect to these NTP servers where they just go and say, 'what time is it's and the NTP server provides the time," he explained. Before Facebook developed a new way of doing this, there were basically two ways to check the time. If you were a developer, you probably used something like Facebook.com as a time checking mechanism, but a company like Facebook, working at massive scale, needed something that worked even when there wasn't an internet connection.
Companies running data centers have a hardware device called Stratum One, which is a big box that sits in the data center, and has no other job than acting as the time keeper. Because these time-keeping boxes were built by a handful of companies over years, they were solid and worked, but it was hard to get new features. What's more, companies like Facebook couldn't control the boxes because of their proprietary nature. Obleukhov and his colleague research scientist, Ahmad Byagowi began to attack the problem by looking for a way to create these devices by building a PCIe card with off-the-shelf parts that you could stick into any PC with an open slot.
"Almost every single electronic device today uses NTP -- Network Time Synchronization Protocol -- which you have on your phone, on your watch, on your laptop, everywhere, and they all connect to these NTP servers where they just go and say, 'what time is it's and the NTP server provides the time," he explained. Before Facebook developed a new way of doing this, there were basically two ways to check the time. If you were a developer, you probably used something like Facebook.com as a time checking mechanism, but a company like Facebook, working at massive scale, needed something that worked even when there wasn't an internet connection.
Companies running data centers have a hardware device called Stratum One, which is a big box that sits in the data center, and has no other job than acting as the time keeper. Because these time-keeping boxes were built by a handful of companies over years, they were solid and worked, but it was hard to get new features. What's more, companies like Facebook couldn't control the boxes because of their proprietary nature. Obleukhov and his colleague research scientist, Ahmad Byagowi began to attack the problem by looking for a way to create these devices by building a PCIe card with off-the-shelf parts that you could stick into any PC with an open slot.
GPS will do just fine for most. (Score:2)
Almost nobody needs a time source this accurate. Every GPS has an accurate time source, because it is part of how it functions. A raspi with a proper GPS hooked up to it correctly can do the same job for most organizations, and are cheap enough to have as many of them as you want so you can put one on each network if you like to make sure one will always be accessible.
Re: (Score:3)
They are claiming accuracy down to tens of nanoseconds. That's better than you can get using a box with GNSS and PTP server.
Not sure what they need that level of accuracy for but there it is. They have an atomic clock module (rubidium) on there too so even when GNSS is lost of a while it should maintain pretty good accuracy, although again I'm not sure what you would want that for if you have lost GNSS and network access.
The main issue I see, apart from cost, is that it needs a GNSS signal which generally m
Re: (Score:1)
The main issue I see, apart from cost, is that it needs a GNSS signal which generally means that the antenna must have a view of the sky or at least be near a window. Not much use in a datacentre.
If only we had something that could carry signals across a distance from one place to another. Maybe a length of some conductor, like copper, wrapped in an insulator.
Re: (Score:3)
If only we had something that could carry signals across a distance from one place to another. Maybe a length of some conductor, like copper, wrapped in an insulator.
Madness! Pure poppycock! No one could come up with such a device!
Re: (Score:2)
If only we had something that could carry signals across a distance from one place to another. Maybe a length of some conductor, like copper, wrapped in an insulator.
Madness! Pure poppycock! No one could come up with such a device!
And if anyone did it would be far too expensive to deploy enough of them to be useful!
Re: (Score:2)
Re: (Score:2)
That is actually the killer with this one, chip-scale atomic clocks are... actually I can't remember what we were quoted and it was a year or two back but the beancounters' response was ZMOGWTFHOWMUCH??!??!. The fact that 100% of the production at the time was earmarked for the US military didn't help...
Apart from that this is a completely ordinary GPS-disciplined clock that you can make from a USB GPS receiver and any computing device of your choice (ODroid, Pi, Arduino, whatever). I have a DIY one that
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Looking a bit further, you can get complete Symmetricon GPSDO units for ~USD200 (under the name ZYT GPSDO, so it's a complete device using the Symmetricom as the time source) and NTP sources (along with the usual 10Mhz, 1PPS, and IRIG-B output) for around the same price. That's a pretty sweet deal...
I recently tried to add GPS as an NTP time source. It's not working so far, I think because the serial port I'm using doesn't provide 5V. I haven't had time to take the multimeter down to the basement and do some probing to find out.
Re: (Score:2)
For GPS that means an amplified antenna, and one per server or a distribution unit. Lots of cable... Would need a good reason to justify it. Why is PTP not good enough?
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
... which would, however, need electronics (switches, etc.) that take time to operate.
Re: (Score:1)
Facebook, some other large platform sites, and finance guys probably deal with event rates high enough having more precision than GPS time is really need to be able to strictly order events. There are obviously a lot of data acquisition applications, laboratory, medical and manufacturing monitoring that probably can use these accuracy levels.
A small market certainly but one that also certainly exists.
Re: (Score:2)
We run conduits to the roofs of our data centers specifically for this purpose.
Re: (Score:2)
Re: (Score:2)
Thinking about it at the timeframes they are looking at the delay introduced by the cable will be an issue. At least with PTP that can be compensated for. I guess if you know the length of the cable you can subtract it out.
Re: (Score:2)
Re: (Score:2)
If you look at the article, it does use GPS (though they call it GNSS, which is basically a generic name for all the various GPS-like systems).
PCIe gives less latency than just using something like a USB GPS receiver.
Looks like it also includes its own atomic clock, so that would allow it to keep extra-accurate time between GPS updates.
Most applications don't require *this* sort of time accuracy, but ... there are some applications that do indeed need millisecond or better accuracy. The usual example I've
Re: (Score:2)
HFT issue can be fixed if we have a rolling average of about 3 to 5 seconds for all trading. The higher level of precision does NOT serve a real purpose other than manipulation. It does not solve liquidity in the market.
Re: (Score:2)
I'm definitely *not* saying that high-frequency traders are a good thing -- I'm only saying that they are one group (of many, probably) who would probably want to buy a product like this.
In fact, they probably already have products like this, though this may be an improvement over what they're currently using.
Sure, they're using products like this to assist their manipulation, but ... I'm just saying that the market for things like this definitely does exist.
But sure, for most ... GPS will do it. Hell, jus
Re: GPS will do just fine for most. (Score:1)
Really ?
And what is magic about 3 or 5 seconds ?
Why not 3 or 5 hours, days, years ?
I think there's a flaw somewhere in your thinking. It starts with your assumption that time can be quantized in a way that is independent of price.
Re: (Score:1)
The magic is the top speed of the universe: Speed of light/Speed of information (vs proximity to the market). 3 seconds is just under 24 times the fastest possible transmission of information (.13 seconds) around the planet. Theoretically.
Hours/Days/Years argument could have been made couple hundred years ago, before the telegraph.
Re: GPS will do just fine for most. (Score:1)
What's your point.
Your post is very close to gibberish
Re: (Score:2)
I've heard that some large installations have NTP synch-ed to a stratum 1 server (either d
Re: (Score:2)
Re: (Score:2)
Any application that needs a clock that is as accurate as possible will want *both*, as much as is possible.
With no special hardware at all, ntpd tries to determine the latency of the network and to work around it, but ... ultimately, it's at the mercy of the network.
A GPS connected via the serial port that is spewing NMEA strings has a good amount of latency, but it's largely predictable -- but not entirely.
A GPS connected via USB has a much higher potential data rate, but 1) the ASCII NMEA strings are not
Re: (Score:2)
It's not an "atomic clock" - that's something like a rubidium or cesium oscillator. It's just a temperature controlled crystal, so that the crystal frequency is more stable as environmental temperatures change than your typical PC motherboard crystal.
Re: (Score:2)
Scratch that, reading the blog I see that the MAC part is a miniaturized rubidium oscillator.
However, it's still not going to achieve the stated 1ns accuracy, because you can never be more accurate than your source, and GPS is only defined as 40ns accuracy. You can potentially be more stable, but not more accurate.
Re: (Score:2)
I only looked at the product photo, so I admit that was lazy.
For a minuscule amount of money you can get a GPS module with PPS output, which was what I was imagining, although probably by now there is some standard way to get even more accuracy.
Re:GPS will do just fine for most. (Score:5, Insightful)
a board to do this is available from Adafruit for about $50 along with the instructions to connect it to a Raspberry PI
BTW, the DEVICE isn't called a "stratum one" the quality of the data provided is stratum one... And the data provided by that adafruit board IS stratum one. Data collected via GPS is stratum one.
Jeezus! the reporting from "tech news" outlets is getting REALLY bad.
What kind of 'new features' ?!? (Score:3)
Re: (Score:1)
Re: (Score:2)
You probably aren't their target audience.
There definitely is a market for rock-solid, "no matter what" extremely accurate time synchronization, and some companies do sometimes spend big money achieving this. At this level, they'll use expensive hardware like this, often using commercial software rather than ntpd, etc.
Most users use nothing more than ntpd or chrony (or even less, such as just daily resetting the clock via some time server), and somebody who's serious might throw in a $15 USB GPS module, bu
Open source because it's useless? (Score:4, Interesting)
The use of the Microchip MAC is completely gratuitous and unnecessary. This entire project for that matter is a bit absurd and looks a bit slapped together. I cant really see much benefit over any of the other open source GPSDO+NTP/PTP projects out there, unless being wildly more expensive is some kind of benefit.
Re: (Score:2)
They needed something more accurate than GPS and built it.
Re: (Score:2)
You don't need anything better unless you're doing hard science.
I suppose it's possible that x86 interrupt latency/jitter is so bad now you can't get a good sync, but a PCIe card isn't going to solve that.
Re: (Score:2)
There have been some pathological cases where this type of hardware has performed surprisingly badly - for example, some raspberry pi revisions had such a poor local oscillator, far worse than the control loops in ntpd were tuned for, that even when connected to a GPS PPS source they could not achieve
Re: (Score:2)
I did run into the clock error issue with one of my devices, the Odroid C2 clock was way off and took a very long time to stabilize. The device I built (a linux based digital dash/logging solution for our race car) doesn't need a particularly accurate clock but the PPS output was there a
Re: (Score:2)
The software stack running on the server consists of standard linux PTP and NTP servers. The PTP server is configured so that the PTP hardware on the NIC is operate
Re: (Score:2)
You only looked at the pretty pictures.
The board is designed with flexibility, you can choose to build with either a GNSS receiver or a small atomic clock as the PPS source. If you want time in a Faraday cage, a truly stand-alone solution, or simply think some future conflict will cause GNSS disruption, you can chose the latter. It's not even on the BOM, it's just an option.
Re: (Score:2)
Only Interesting comment, eh? Thus spake moderators.
My "potential interest" in the project is related to the centrality of time in my thinking about economics and philosophy combined with Zuckerberg's status as a wannabe Bond villain. But so far I haven't found any threats of interest in the story, even with my vivid imagination. (But a REAL Bond villain would conceal the motivation of the plot until he's dangling someone over the pool of sharks. (Sharks with laser clocks!))
How is this a story? (Score:2)
Re: (Score:2)
The device they needed did not exist so they built it. It wasn't designed to be groundbreaking or impressive but rather fulfill their specific needs. They just happened to release the design and all the software people here seem confused. This is no different than any company open sourcing code snippets.
Re: (Score:2)
So what is wrong with NTP for most purposes? (Score:5, Insightful)
"If you were a developer, you probably used something like Facebook.com as a time checking mechanism"
Umm. I would not have even thought of using Facebook.com to get the time. I may use Google, just because they are known to offer some IT Services, like DNS.
But what I fail to get what is wrong with NTP. To get really accurate, you calculate in the lag time, and your calculation time, then you get really close.
But if I were needed the time for scientific, or high performance engineering. I don't think I would have considered oh lets use Facebook to get the time. But perhaps some large university with an atomic clock.
Being that this hardware, which will use up a PCI slot still needs an internet connection (which is the real bottleneck in accuracy) I don't see how you are better off with it, vs just dealing with a Real Time OS, that will ping the NTP server more often.
Re: (Score:2)
If you read the article they mention it links with GNSS, not Internet-based NTP. Of course a GNSS antenna with clear view of the sky would likely come into play. But think remote locations without reliable Internet access. That's a primary use case I can think of. Although there are lots of DIY projects that already incorporate either GPS or GNSS for accessing time and whatnot...
Re: (Score:3)
It does still seem like a lot of effort on Facebook part to solve a problem that isn't really a problem.
If you are in a place without reliable internet, how many use cases where you are using a normal PC, that you are going to need such accurate time. If you are living without internet access, (where time may play an important part of encryption) you are need such accurate time for nearly any job, where your PC Clock, or your $0.50 digital watch that you got out a child's vending machine will probably do t
Re: (Score:2)
Almost all stratum 1 NTP servers have these features. Every one I've looked at -- and that's a fair number -- has an option, and usually several choices, for a higher-stability oscillator: going from TCXO to OCXO to rubidium, rarely to cesium. For nanosecond-level accuracy, handling the clock offsets between GNSS constellations is important, but neither novel nor particularly hard. It is a little bit mystifying exactly what problem(s) they thought they were solving with this card.
Re: (Score:2)
It is a little bit mystifying exactly what problem(s) they thought they were solving with this card.
They solved the problem of no viable open source designs for such a device.
Re: (Score:2)
By itself, that's a dumb problem to solve. Is their solution cheaper, more reliable, more precise in some scenario they care about? What actual benefits do they see from their open source solution?
It's not even true: https://developers.redhat.com/... [redhat.com]
Stratum 1 NTP servers are not cheap, but the market is pretty competitive. You can get oodles of them in 1U form factors, or others that could mount to a DIN rail in a 0U configuration taking less than 30 cubic inches. COTS products are probably going to be
Re: (Score:2)
But think remote locations without reliable Internet access. That's a primary use case I can think of.
If Facebook's data center doesn't have Internet access, knowing the nanosecond time isn't the big problem they need to solve.
And there are lots of GPS-based clock devices like this one, available for cheap. However, the optional addition of your own atomic clock is a feature.
But again, in what scenario is Facebook's data center needing to function, when it cannot access the Internet, and also GPS has gone down? John Conner is going to be on shortwave, not trying futilely to post on Facebook. Hmmm, or MAYBE
Re: (Score:2)
I believe that Facebook has a lot of other ventures besides just social media. As much as I loathe that aspect of their existence, their development group has given us everything from GraphQL to React JS.
There is a project I'm looking forward to delving into called FreedomFi. Basically using open areas of the 5G spectrum for private usage. The GitHub project is called Magma and I believe Facebook is behind it as well.
Re: (Score:3)
They needed greater accuracy than NTP or GPS based solutions. I mean it's stated halfway into the first paragraph...
Re: (Score:2)
But why offer it as a product to the rest of us.
There are a very few use cases where NTP or GPS is not good enough. I expect Facebook need for greater accuracy is probably overblown as well, but probably had some kid writing code above his pay grade, then found out that the time stamp still created dupe unique ID's cross systems.
Re: (Score:2)
But why offer it as a product to the rest of us..
Because 640k is enough for anyone.
Re: (Score:2)
It's an open source project. It's not a product for anyone but themselves internally.
https://github.com/opencompute... [github.com]
Re: (Score:2)
Except their time source is still GPS, so it is impossible to have the stated 1ns accuracy. IIRC GPS accuracy is only rated at 40ns (although it is usually better, you can't state something that traces to GPS is any more accurate than GPS is specified to be).
Encryption/authentication (Score:2)
Re: (Score:2)
But having it picking up with some other time keeper service, Such as GNSS doesn't seem like something that will be super accurate anyways. With encryption, you need to be good enough to be in sync with your weakest link. So while you may have the top time, if you connect to someone a ns off then you are just in the same problem as before.
Re: (Score:2)
If you look at their open source project you'll see that it utilizes an NTP (ntpd or chronyd) and PTP server.
https://github.com/opencompute... [github.com]
So many things wrong (Score:3)
Companies running data centers have a hardware device called Stratum One, which is a big box that sits in the data center, and has no other job than acting as the time keeper. Because these time-keeping boxes were built by a handful of companies over years, they were solid and worked, but it was hard to get new features.
Who wrote this? "Stratum One" is not a proper noun, nor is a stratum 1 reference clock typically a "big box". And what "new features" are needed? It tells time - it doesn't need to do anything but tell time.
Anyone that cares and that has rudimentary soldering skills can assemble a GPS-based reference clock for under $100. There are Raspberry Pi-based setups that don't even require soldering (although prior to the model 4, the Pi didn't make a great network refclocks because of jitter on the USB-connected ethernet port, although they were "good enough" for basic use).
Thanks, Captain Obvious (Score:2)
Re: (Score:2)
It's not a product you need to buy. It's an open source project.
https://github.com/opencompute... [github.com]
Re: (Score:2)
How to get the time from Facebook... (Score:1)
Basic factual errors everywhere (Score:5, Insightful)
1) 'Stratum One' isn't a product, it's a reference for how far you are away from an authoritative time source. (i.e. a GPS, what most 'Stratum One' boxes use). You can, and always could, make one with even obsolete hardware.
2) They aren't in the least proprietary, any number of open source packages from multiple vendors can get you a 'Stratum One' server
3) Those servers are still NTP servers. The clients connecting to them use the NTP protocol.
4) virtually every piece of desktop or server computer hardware has one of these that 'doesn't rely on the internet'. It's called....wait for it...a clock. a 'Real-Time Clock' . While some embedded systems do not have these, virtually every other system does, and has had, since at least the IBM PC/AT.
5) If you are looking for INTERNAL timekeeping, which can have sub-nanosecond accuracy and implications, there are other systems already built in to manage that as well, such as the 'High Performance Event Timer' (basically, just a more accurate built-in clock)
Whoever wrote this article knows nothing of the source material. Garbage from the first word,
Re: (Score:3)
Whoever wrote this article knows nothing of the source material. Garbage from the first word,
Pretty much. Also, NTP _works_ and with a drift file and a reasonable stable internal clock, it even keeps working pretty precisely when the net is unreachable. The only reason for this hardware is if you need time so critically being right that you cannot trust any external time source. That basically is almost never the case.
Facebook.com as a time checking mechanism? (Score:5, Insightful)
If you were a developer, you probably used something like Facebook.com as a time checking mechanism
I can assure you that I have absolutely not used any *.facebook.com dns name as an NTP time source, literally ever in my life. They may participate in pool.ntp.org, so I've potentially used their time infra as a result of that, but I certainly have never explicitly seeked a facebook ntp source by name. Seems like techcrunch could use a few more technical editors to catch this sort of nonsense.
This quality editing keeps me coming back to /. (Score:2)
TFA Link is broken (Score:2)
Either the link to TFA is broken or TFA was removed.
Anyone has a proper link to a FA ?
Re: (Score:2)
Here's the official page from FB Engineering:
https://engineering.fb.com/202... [fb.com]
What additional features could you possibly want? (Score:2)
Any additional features that are added to a box that hands out time are going to slow it down, and likely make it less accurate.
It's a clock. It needs to hand out a number when asked.
Sure, you could use additional sources for it to derive the time, but then you fall into the "A man with two watches is never sure what time it is" problem.
And Facebook as the time standard for the world? Seriously? Who would pick that on purpose?
Whatâ(TM)s the use case for this? (Score:1)
âoeIf you were a developer, you probably used something like Facebook.com as a time checking mechanism,â¦â. No. We âoeprobablyâ donâ(TM)t. I just did an informal (non-scientific) poll of the developers that I am currently working with and no one said that they had even heard that Facebook has a time service. Our computers sync to the NIST time server to keep our machine clocks accurate. When we need the time, we just use our system clocks. If we need something more a
WTF? Is NTP not good enough anymore? (Score:2)
Seriously. They are solving a solved problem. I can only speculate that this is about some bogus press coverage...
Re: (Score:2)
Well, there this item from the intro:
If I were to engage in speculation, I would likely begin by wondering why controlling a time source is so important to Facebook that they would fund an internal project devoted to solving this ``problem''.
Re: (Score:2)
Well, there this item from the intro:
If I were to engage in speculation, I would likely begin by wondering why controlling a time source is so important to Facebook that they would fund an internal project devoted to solving this ``problem''.
It is suspicious. What they do is not needed. Like, at all. They could just put in their own NTP master. Maybe this is just some engineer looking for something to do. Or maybe some military folks are being paranoid. There are security implications if you clock is wayyy off, but that is more in the minutes range before it has any effect.
Where this is required (Score:2)
Huge data centers are getting decentralized into smaller and more distributed âoeedgeâ locations that are closer to users and can serve them with millisecond latency or even less.
What happens if the external GPS antenna or its cable is disrupted? The miniature atomic clock on this device can maintain microsecond accuracy without external reference for days until a technician can be dispatched to this edge location.
While most of us donâ(TM)t need to meet such requirements, the cost of doing th
Time-keeping appliance (Score:2)
Isn't that what they call a clock?
HP z3801 (Score:2)
Uhhh, every CDMA cell site included something like that z3801 for precise time and frequency reference. How is this not a solved problem?
Re: (Score:2)
The amateur radio world is full of such projects. This remains a solved problem.
Correct me if I'm wrong (Score:2)
But basically they built one of those Spencer's Gift's style "atomic clocks" on a PCI card?
Basically this?
https://smile.amazon.com/Sharp... [amazon.com]
From the summary you'd think they invented time itself.
Re: (Score:2)
Consider yourself corrected.
Those so-called atomic clocks are just cheap quartz clocks that set themselves once per day from a radio broadcast of a time signal. The thing Facebook make gets it time from GPS, which is more accurate than the WWVB radio time signal, and has a real Rubidium atomic clock on the board to boot. It's orders of magnitude more advanced and precise.
But it's not groundbreaking. Something like this is a common amateur radio hobbyist project many people have already done. The miniatu
real motivation explained (Score:2)
... Facebook couldn't control the boxes because of their proprietary nature.
High accuracy had nothing to do with why they wanted to do this.
GNSS (Score:2)
Seems it calibrates with GNSS, what happens if GNSS goes wonky due to a hack? How big are the max. adjustments it can make?
Re: GNSS (Score:2)
Such a "hack" will signal the beginning of World War 3.0. At that point you will have much bigger problems than precision timekeeping in your data center, such as collecting bottle caps and fending off mutant looters.
Re: (Score:2)
WW3 over a teenager in his mom's basement hacking into a satellite control station, assuming even that is necessary? Plus an unintentional bug may also cause it to go wonky. Like how all Microsoft Zune players bricked themselves December 31, 2008.
Re: (Score:2)
his mom's basement hacking into a satellite control station
That's not going to happen. Not with GPS. Those people aren't idiots, so they don't put that hardware on the public Internet.
More info here (Score:2)
There’s a lot more technical information here:
https://engineering.fb.com/202... [fb.com]
e.g. “why not just use gps?” This device does use gps, alongside a highly accurate oscillator. It will keep accurate time even in the event of a loss of satellite signal.