Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
IT Technology

CloudFlare Was Hit By Leap Second, Causing Its RRDNS Software To 'Panic' (silicon.co.uk) 119

Reader Mickeycaskill writes: The extra leap second added on to the end of 2016 may not have had an effect on most people, but it did catch out a few web companies who failed to factor it in. Web services and security firm CloudFlare was one such example. A small number of its servers went down at midnight UTC on New Year's Day due to an error in its RRDNS software, a domain name service (DNS) proxy that was written to help scale CloudFlare's DNS infrastructure, which limited web access for some of its customers. As CloudFlare explained, a number went negative in the software when it should have been zero, causing RRDNS to "panic" and affect the DNS resolutions to some websites. The issue was confirmed by the company's engineers at 00:34 UTC on New Year's Day and the fix -- which involved patching the clock source to ensure it normalises if time ever skips backwards -- was rolled out to the majority of the affected data centres by 02:50 UTC. Cloudflare said the outage only hit customers who use CNAME DNS records with its service. Google works around leap seconds with a so-called "smearing" technique -- running clocks slightly slower than usual on its Network Time Protocol servers.
This discussion has been archived. No new comments can be posted.

CloudFlare Was Hit By Leap Second, Causing Its RRDNS Software To 'Panic'

Comments Filter:
  • by Anonymous Coward

    The blog post about this incident [cloudflare.com] says:

    RRDNS is written in Go and uses Go’s time.Now() function to get the time. Unfortunately, this function does not guarantee monotonicity. Go currently doesn’t offer a monotonic time source (see issue 12914 for discussion).

    and then later it says:

    When RRDNS selects an upstream to resolve a CNAME it uses a weighted selection algorithm. The code takes the upstream time values and feeds them to Go’s rand.Int63n() function. rand.Int63n promptly panics if its

    • Re: (Score:3, Insightful)

      by Anonymous Coward

      I don't know if you can blame the language, the devs should have added their own checks if the language didn't have a guarantee.

      • I don't know if you can blame the language, the devs should have added their own checks if the language didn't have a guarantee.

        Noting math/rand is part of the standard go library and more rigorous compile time checking would have prevented this seems like a no-brainer to blame the language.

        • Blaming the Language is like blaming a toaster for shocking someone in the bath because they felt a wee bit on the hungry side.

          Tools, just like features in languages, should not be made idiot proof.
          A warning is fine. In this case, someone clearly never checked the spec or put in their own check just for leap seconds. Doesn't need to be in the code forever, just that event then comment-out and recompile.
          Putting these checks in by default would add extra overhead. One extra check adds up even after a day.

    • Re: (Score:3, Insightful)

      by Anonymous Coward

      Why would you even think of switching programing languages due to the simple and sadly common 'bug' of programmers not verifying parameters match a function's documented pre-conditions? My only guess is you're paid to promote Rust. Lazy programmers will write bugs in every language.

    • by fubarrr ( 884157 ) on Tuesday January 03, 2017 @03:49PM (#53600037)

      >RRDNS is written in Go

      Their bugs are in HR department.

      Who in the world hired people who are dumb enought to use an experimental language in production?

      • Could also be a financial consideration. The smart folks wanted too much money. Or, maybe an unpaid summer high school intern was the choice.
      • A conservative programming language in a version numbered as 1.7 hardly fits any sane person's definition of "experimental".
    • Part of the fault can go to the Go programming language for their API design.

      But most of the blame goes to the developers.
      I haven't coded in Go, but I googled this quickly.
      https://golang.org/pkg/math/ra... [golang.org]
      The Go documentation clearly says it panics if n = 0.

      They could have
      1. validated their inputs.
      2. Handle the panic and assign a default value (I am assuming this is possible in Go. I have never used it)

      In the end, it seems like this is just used to distribute requests. Worst case, it should log the error an

      • by Obfuscant ( 592200 ) on Tuesday January 03, 2017 @05:07PM (#53600631)

        The Go documentation clearly says it panics if n = 0.

        And it says it panics if n is less than 0.

        If you write a library function that requires positive input always, and returns positive output always, then use unsigned input and output variables. A good compiler will flag the attempt at sending such a function a signed input as a warning at least. Pedantic compilers will fail -- better than the production program failing.

        And while it seems stupid, the proper action when asked for a random number between 0 and 0 is to return 0, not panic. (I believe the [ on the range means "including", but I could be wrong. If it didn't mean "including", then the documentation should be '(1,n)'.)

        But then, the test cases for the DNS code should have included 0 and negative, so this should have been caught when the function was tested.

        • Yep.

          I meant to type n (less than or equal to) 0.
          Not sure if slashdot escaped it or something.

          In any case, yes, the Go API was not the best taking in a signed int when negatives are invalid.

      • by Anonymous Coward

        Part of the fault can go to the Go programming language for their API design.

        Indeed - a random number function that panics. Now that is useful.

        You want servers to keep running. The better approach would be to return 0. and in the interest of debugging, spam the syslog with "bad parameter -x to rand.Int63n()". Complaints puts some pressure on devs to fix things.

    • So to me it sounds like this incident was at least partially due to limitations with the Go programming language and its libraries.

      For now, you could use a platform-specific workaround. (Just like you would have to do if you were coding in C). For Linux:

      func Uptime() (int64, err) {
      var si syscall.Sysinfo_t
      err = syscall.Sysinfo(&si)
      return si.Uptime, err
      }

      I'm too lazy to look up whether Windows has a similar feature.

  • I still wonder how these simple edge cases make their way in
    • by unrtst ( 777550 ) on Tuesday January 03, 2017 @03:40PM (#53599985)

      Read the article then. It shows it pretty plainly: https://blog.cloudflare.com/ho... [cloudflare.com]
      I was going to try to guess what they were doing, but they have some actual code snippets.

      AFAICT, a unit test wouldn't have caught this either (unless they planned for this sort of error, in which case the code wouldn't have been broken either). From TFA:

      RRDNS doesn’t just keep a single measurement for each resolver, it takes many measurements and smoothes them. So, the single measurement wouldn’t cause RRDNS to think the resolver was working in negative time, but after a few measurements the smoothed value would eventually become negative.

      So, a unit test with one negative example (which may have been difficult to mimic anyway, due to the direct usage of Time.Now()) probably wouldn't have triggered the issue on its own.

      IMHO, blaming a misconception of time always going forward is just convenient here. The fix was changing this bit:
      if rttMax == 0 {
            rttMax = DefaultTimeout
      }

      They just changed "==" to "<=". There was no reason not to have it as "<=" to begin with, even if one ignores where rttMax comes from. Any time I check if something is == to something else, and I don't have else conditions covering the other cases, I ask myself what should happen in those other else cases and ensure I'm covered. That may still have caused it to break, but it could have done:
      if rttMax == 0 {
            rttMax = DefaultTimeout
      } else if rttMax < 0 {
            panic("What the fuck happened to rttMax to make it negative!?!")
      }
      ...though it probably would have been better to just log that somewhere and set it to the DefaultTimeout.

      Anyway, I think it's a great example of a one character bug that only triggers on very obscure events under significant load.

    • by skids ( 119237 ) on Tuesday January 03, 2017 @03:40PM (#53599987) Homepage

      I'm still left wondering whether the decision to put a leap second on the night tech support staff are most likely to be over halfway through a bottle of JD was A) some intentional attempt to catch edge cases where leap seconds happen during a year change or B) some tinfoil conspiracy where we'll find out billions of dollars were stolen from a system where that particular edge case could be exploited or C) just made by people so socially isolated that they don't realize just how hard it is to fix crashed boxen over a crappy 3G connection in a dive bar bathroom using a phone covered in some chick's vomit while trying to keep down that pretzel you just washed down with sparkling water.

      • I'm still left wondering whether the decision to put a leap second on the night tech support staff are most likely to be (conspiracy options elided) ...

        Or most likely, the people who put the leap second where it was knew there was no perfect time to do it, and assumed that anyone who was writing software that was so time-critical that it cared if there was a leap second would properly handle the issue in their code.

        It's not their fault that some developers using an off-beat language that has a library that panics if a parameter is invalid (and was written so that there could BE an invalid parameter, which they could have avoided) didn't bounds check thei

    • by Anonymous Coward

      Millennial idiot coders are too busy refactoring everything every three months to bother testing anything ever. Mature codebases are too old, dudebro, old is bad, mmkay.

  • by Chris Katko ( 2923353 ) on Tuesday January 03, 2017 @03:14PM (#53599821)

    ...at exactly midnight, while I was playing Chivalry. I kept getting laggier... and laggier... and then everyone "froze" and the client-side prediction took over. I was recording video and it was pretty funny. Everyone just kept walking forward, until they were in a wall, and kept trying to walk forwards.

    It was interesting what the client prediction would let you do. You could change weapons. You could swing your weapon. You could throw axes (of which you have two) and they flew through the air, stuck in people, and even knocked helmets off. BUT, your axe counter never actually decreased. So you could just keep throwing hundreds of axes. The animation timings / speeds were unaffected. You couldn't "chant" or grunt. You obviously couldn't damage anyone.

    Anyway, my internet was down until the next morning and even then, it still required a cable modem reset to fix the connection.

    • by Falos ( 2905315 )
      Specifics vary, but some games have interesting latency tolerances that they're willing to resolve without a Nope shrug (rejection). Mounted WoW players often glide past "no fly zone" triggers, sometimes remaining airborne long enough to reach the unreachable. Usually inconsequential places, by design.

      PoGo players (and Ingress, I'd guess) have been known to capture indoor critters by dashing at building walls in the meatspace, then huddling over their phone to block signal. The game extrapolates position
      • I've actually been researching network game architecture lately and I was actually planning on doing some video-recorded analysis of various commercial-game network models when latency, jitter, out-of-order, and other errors occur. Extreme latency is a great way to "reveal" what's going on under-the-hood.

        So this time, I got video and I didn't even have to set up artificial lag!

    • by antdude ( 79039 )

      Do you have still have that video recorded to upload to share with us? ;)

    • "Anyway, my internet was down until the next morning and even then, it still required a cable modem reset to fix the connection."

      Some network equipment vendors sent out field notices about 2 weeks in advance of the leap second, recommending operators to use leap-second smearing (as implemented in chronyd for example) if they had affected versions of network device firmware deployed that could crash as a result.

      (We didn't have affected versions deployed, and it would have been non-trivial - at this time of y

  • 2016 says "Hi, remember me, beeotches?!!"
  • by Anonymous Coward

    We lose or gain a second here or there, who cares? The difference has been so far 27 seconds over the past 44 years or, extrapolated out, 1 MINUTE over 97 YEARS.
    Are we really going to notice if the sun goes down a minute earlier every century? We already have to screw around with daylight savings & leap years why not just make February the 29th 24 hours and 1 minute long once a century and have done with it.

  • I always remember time changes as busy nights in support when I worked for a large bank. The spring forward was usually a breeze, just a matter of a lot of server verifications and log checks, but the fall back was usually a messy night. Much harder to deal with and resolve issues involving duplicate timed log entries and transaction logs. I don't really miss those days...

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      UUIDs are also fun to deal with. Especially with VM images that are copied by those who don't understand how you can have a duplicate UUID.

    • by caseih ( 160668 )

      So the bank's systems didn't store transaction times in UTC or some other timezone-neutral format? How did they deal with transactions originating from other time zones?

      • Re: (Score:2, Informative)

        by Anonymous Coward

        I've seen too many companies, even large multinational companies, who insist that their servers run on the HQ's local time to be surprised by this anymore.

        It ain't the IT folks doing it, I'll tell you that. The smart ones quietly say "fuck that", set the system clocks to UTC and then set a TZ environment variable everywhere.

  • by SuperKendall ( 25149 ) on Tuesday January 03, 2017 @03:48PM (#53600027)

    Don't use services who names are terribly ironic in times of failure.

    Flare, Flame, Burn, Drop, Etc. Et.

    The universe just loves to throw a wrench at such forms of un-intentional hubris just for the LOLs.

  • The issue with leap seconds is much bigger than just Cloudflare. I’ve found there are difference in at least 4 types of time: Google Time (their unique version of NTP time protocol), International Atomic Time(TAI), Coordinated Universal Time (UTC), and multiple NTP protocol servers. Currently there is a difference of 37 seconds between International Atomic Time (TAI) and Coordinated Universal Time (UTC). https://www.timeanddate.com/ti... [timeanddate.com] When I checked the time sync of time.windows.com to time.is I n
  • "If Engineers built buildings the way Programmers write programs, the first woodpecker that came along would destroy civilization!"

    And if you did it that way because your "pointy-haired boss" said to, then it is still your fault... ;-)

Been Transferred Lately?

Working...