Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
The Internet IT Technology

One Fastly Customer Triggered Internet Meltdown (bbc.com) 46

Posted by msmash from the closer-look dept.
Thelasko writes: The company operates servers at strategic points around the world to help customers move and store content close to their end users. But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing '85% of our network to return errors', it said in a blogpost.
This discussion has been archived. No new comments can be posted.

One Fastly Customer Triggered Internet Meltdown More

One Fastly Customer Triggered Internet Meltdown

Comments Filter:

  • Empowered customer. (Score:5, Funny)

    by Ostracus ( 1354233 ) on Wednesday June 09, 2021 @12:27PM (#61470102) Journal

    But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing '85% of our network to return errors', it said in a blogpost.

    No I say not the pen is mightier than the sword, but the customer is mightier than the network..

  • Correct headline (Score:5, Insightful)

    by fermion ( 181285 ) on Wednesday June 09, 2021 @12:27PM (#61470104) Homepage Journal
    Fastly distributed software that was not adequately tested, and one customer helped then find the bug. It is not the best case scenario, but at least it gets fixed.

  • Old Quote (Score:2, Funny)

    by randalware ( 720317 )

    If houses were built like software, the first woodpecker that came along would destroy civilization.

    • Re:Old Quote (Score:5, Funny)

      by fermion ( 181285 ) on Wednesday June 09, 2021 @12:40PM (#61470144) Homepage Journal
      If architects built houses, they would kill the occupants in the first week. If engineers built houses, the owners would kill themselves within the first week.

      • I had a good friend studying to be architect, my sis studied to be a civil engineer (the latter doesn't make projects herself, lost contact with the former). In one conversation, I got told that architects do drugs and produce good-looking plans that the customer likes, then the civil engineer has to re-do the plans mostly from scratch to make them physically realisable and to allow the building not collapse the moment scaffolding is removed.

        • Re: (Score:2)

          by ghoul ( 157158 )
          If houses were designed by engineers there would be no challenge. The engineering challenge is in making possible the crazy things architects dream up. Engineers dont hate architects, they just like to cuss them but secretly enjoy the challenge of making real the crazy things people dream up on mushrooms.

          • Re: (Score:2)

            by uncqual ( 836337 )

            Then the contractor comes along and has to convince the engineer that what they speced is very difficult or costly to build and hope the engineer will change the design to make it more buildable.

            Case in point (where perhaps the engineer should have made a change, but obviously not the change they did make!) is the situation that led to the collapse of the catwalk at the Hyatt Regency Kansas City almost exactly 40 years ago (on July 17, 1981) which killed 114 people and injured over 200 others.

    • Re: (Score:2)

      by Z00L00K ( 682162 )

      My sig since early time on /.

    • Re: (Score:1)

      by Tablizer ( 95088 )

      A stray cigarette can burn down an entire neighborhood.

      • "Only you can prevent forest fires"

        There's 7 billion people on the planet. Why is it my sole responsibility? I don't even go to the forest...

    • Re: (Score:3, Insightful)

      by blahblahblahblahblah ( 7113665 )
      If houses were built like software you'd be able to revert to the last stable version, civilization wouldn't have much more than a couple hours of downtime before being back up and running like nothing happened.

      • Re: (Score:2)

        by uncqual ( 836337 )

        Well, except for the houses that collapsed and killed the occupants before the problem could be traced to the bad revision (and assuming that the database under the house hadn't been modified by the new revision and the old revision would no longer work correctly as it hadn't been tested on the new database format/field contents).

      • Now there's a new meaning to the term "bricked"...

  • The internet didn't melt down, Fastly did (Score:5, Insightful)

    by WoodstockJeff ( 568111 ) on Wednesday June 09, 2021 @12:39PM (#61470140) Homepage

    The rest of internet worked just fine while the one Content Delivery Network went bonkers.

  • SV mantra playing out in real time.

    • Re: (Score:1)

      by Tablizer ( 95088 )

      1. Move Fast
      2. Break Things
      3. Bail out before you get caught
      4. Move to obscure island
      5. Profit!

  • A bug in the software caused the problem. Uhh, that's why the problem is called a bug. If it looks like a duck, walks like a duck, well its a dead duck.

    A gash in the side of the Titanic caused the ship to sink...

    Or more colorfully, like Scully in the X-Files (forget the specific episode) in an autopsy (when she had a shot of histamines) she is asked "So what killed him?"

    "Bee....eep!" and Scully smacks her hands. Yeah he got mowed down by a bus as cause of death. That was the bug in this case.

    JoshK.

    • Re: (Score:2)

      by reanjr ( 588767 )

      Synchrony maybe. It's a sort of goofy episode with time travel and a guy warned by a future person he was going to get hit by a bus.

      Season 4, Episode 19.

  • Hoping for details. (Score:5, Insightful)

    by jythie ( 914043 ) on Wednesday June 09, 2021 @12:52PM (#61470196)
    While I suspect any post mortem they could post would be beyond an outsider's ability to really make sense of, I would be really curious to read about how exactly a valid configuration change took down a whole system. I could understand if it was something silly like 'customer requested -1 CPUs and the parameter went through an unsigned field then tried to allocate billions of CPUs to them!', but what kind of valid change results in such a cascade failure?

    • Re:Hoping for details. (Score:4, Insightful)

      by Anonymous Coward on Wednesday June 09, 2021 @12:58PM (#61470222)

      but what kind of valid change results in such a cascade failure?

      That's the first question I thought of. How can one customer, making a valid configuration change on their network, have a global effect on Fastly's entire network?

      If this explanation is actually true, it doesn't speak very well for the quality of Fastly's network.

      • Re: (Score:1)

        by Anonymous Coward

        Because basic concepts such as segmented access, input validation, and TESTING don't quite go along with today's mantra of break things quickly.

        I will have to say, they did recover rather quickly. I wonder how many rubles, shekels, farthings, or florins were lost by businesses due to the approximate 1hr outage window?

        Fortune Cookie at bottom of page when I posted this: Experience varies directly with equipment ruined.

      • Re: (Score:2)

        by amorsen ( 7485 )

        Can you imagine an explanation that speaks well of Fastly's network? I certainly can't.

        These things happen.

      • Can you test every permutation of valid configuration? Particularly for a large complex system, with end user configurable rules.

        The real world is how you discover that your automated tests don't cover enough cases.

    • Re: (Score:2)

      by gweihir ( 88907 )

      Probably something very stupid and very embarrassing that should not have happened in development at all and should most definitely have been caught in testing. You know, cheaper than possible software "engineering" at work. Hence we will probably never know. The lies have already started by claiming a customer caused it. In reality, a customer triggered it, but it was caused by Fastly.

  • Did Fastly really think Iran was going to run their nuclear reactors on the cloud? Want good software security. Stop hiring people who worked at the NSA and just cant help themselves from putting in zero day exploits.

  • that they blame it on the customer; fastly, give your PR person a raise. "Oh it wasn't our fault, a customer broke our system"

  • Yes it was a "customer" that caused the issue.
    *Unplugs toaster and coffee maker. Plugs Fastly server back in*
    • The only button in our Sun 3/280 rack that was within reach of our janitor's toddler, was the disk power switch.
  • When you outsource, actual Quality and Service is sacrificed by design, because it would cost you and them the same to do it right. So how did they quote that really great price?

    • Re: (Score:3)

      by bws111 ( 1216812 )

      That is really dumb. Really, really dumb. Do you think you can deliver your own package across the country cheaper than UPS can do it? Build your own car for less than Ford can do it? Fly somewhere cheaper than an airline can do it? There is such a thing as shared costs, I suggest you learn about them.

    • Re: (Score:2)

      by uncqual ( 836337 )

      In some cases the outsourcing will improve quality because it results in code being tested better.

      Imagine if you develop your own locking primitives portable across several architectures vs. outsourcing that by buying a library that is consistently maintained (where "buying" may just be accepting a FOSS license). It's likely that the library you "buy" is much more likely to take into account an obscure error in stepping K of a three year old Intel chip and work around that shortcoming than your software wou

  • my bad...
  • Who is this customer of which you speak?

Slashdot Top Deals

"The vast majority of successful major crimes against property are perpetrated by individuals abusing positions of trust." -- Lawrence Dalzell

Close