Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
The Internet Social Networks

A Big Chunk of the Internet Goes Offline Because of a Faulty CDN Provider (techcrunch.com) 157

TechCrunch reports: Countless popular websites including Reddit, Spotify, Twitch, Stack Overflow, GitHub, gov.uk, Hulu, HBO Max, Quora, PayPal, Vimeo, Shopify, Stripe, and news outlets CNN, The Guardian, The New York Times, BBC and Financial Times are currently facing an outage. A glitch at Fastly, a popular CDN provider, is thought to be the reason, according to a product manager at Financial Times. Fastly has confirmed it's facing an outage on its status website. Update: the issue has been resolved.
This discussion has been archived. No new comments can be posted.

A Big Chunk of the Internet Goes Offline Because of a Faulty CDN Provider

Comments Filter:
  • by Anonymous Coward on Tuesday June 08, 2021 @05:50AM (#61465324)
    If Stack Overflow is down, then how will this be fixed.
    • Re: (Score:2, Redundant)

      Argh- and I just spent my mod points!
    • by Kisai ( 213879 ) on Tuesday June 08, 2021 @09:41AM (#61465906)

      Don't put all your internet eggs in one basket. Same goes for Cloudflare and other CDN's.

      Honestly, it's really starting to seem like CDN's are a bad idea, because all it takes is one country to want to block twitter, and suddenly every site behind the CDN is blocked.

      • by Somervillain ( 4719341 ) on Tuesday June 08, 2021 @10:59AM (#61466090)

        Don't put all your internet eggs in one basket. Same goes for Cloudflare and other CDN's.

        Honestly, it's really starting to seem like CDN's are a bad idea, because all it takes is one country to want to block twitter, and suddenly every site behind the CDN is blocked.

        None of that is true. The whole point of CDNs is reliability. If they have an issue with one site, the CDNs ensure everyone else can get through.

        • by PPH ( 736903 )

          If they have an issue with one site, the CDNs ensure everyone else can get through.

          Which is exactly the situation we had when everyone rented a slot in a data center rack and rolled their own web site.

      • Honestly, it's really starting to seem like CDN's are a bad idea, because all it takes is one country to want to block twitter, and suddenly every site behind the CDN is blocked.

        Firstly it doesn't work like that.
        Secondly you can always put up your own "reliable" server, and then watch it melt into a puddle of plastic goo when Slashdot posts a story about you. Remember that? The pre CDN days, when a bunch of nerds reading news for nerds would knock whole sites offline?

    • by UnknowingFool ( 672806 ) on Tuesday June 08, 2021 @09:47AM (#61465912)
      No need. I had to make toast and coffee and forgot to plug the Fastly server back in. Sorry about that.
    • by Darinbob ( 1142669 ) on Tuesday June 08, 2021 @01:13PM (#61466584)

      Seen on Stackoverflow:
      A: "Help, my company's web site is down and people keep phoning me to fix it, what do I do?"
      B: "This sounds like you just want help with your homework."
      C: "Have you tried Rust?"
      D: "You should try turning your company off and back on again."

  • by Viol8 ( 599362 ) on Tuesday June 08, 2021 @05:57AM (#61465340) Homepage

    ... which isn't.

    Its about time one of these companies got sued into the ground after outages like this so they provide proper multiple failover and backup systems instead of whatever poundshop versions they tend to use instead.

    • by DamnOregonian ( 963763 ) on Tuesday June 08, 2021 @06:08AM (#61465362)
      LOL. Good luck with that.
      I've had customers threaten to sue us as well.
      Always cracks me up.

      The SLA is right there in the contract you signed. It includes our reimbursement/credit policy for performance below our SLA.
      The second you threaten though, I let you know the corporate lawyer will be conducting all further communications with you.
    • ... which isn't.

      Its about time one of these companies got sued into the ground after outages like this so they provide proper multiple failover and backup systems instead of whatever poundshop versions they tend to use instead.

      Perhaps we are relying on CDNs as as whole a bit too much, IF this is to be believed:

      "Today’s issue isn’t limited to a data center in particular. Fastly calls it a “global CDN disruption” and it sounds like it is affecting the company’s network globally."

      CDNs were initially justified to provide considerable performance gains. They should not completely disable shit when they go offline, but it would appear they are now more critical than the web servers hosting content now.

    • So they run for about 7 years and then what? Is the 7 years up this morning, is that the problem?

    • by fermion ( 181285 )
      When I was young, and not in the US all time, 24/7/365 meant that malls where cooled by natural ventilation and all orders were taken on pen and paper. Now, although electricity is not so reliable there, the tech stores have electric air conditioning and most stores have caved into the unreliable computer. Sure it may only be 99% reliable, but that is better than paying some error person to enter data after the fact
    • by Z00L00K ( 682162 )

      Downtimes happens for everyone, just accept it.

      However these days web sites are often built up by cross-linking scripts from many sites and if one script serving site goes down you suddenly have many other sites that starts to behave strange. Even if the script serving site is just one of the hated and useless trackers.

      • However these days web sites are often built up by cross-linking scripts from many sites and if one script serving site goes down you suddenly have many other sites that starts to behave strange.

        Right vein of thinking, but it's even messier than that.
        I find that anyone who acts like perfect uptime is easy has never tried it.
        In particular, the more scalable (wide) your service needs to be, the easier it is to run into situations where failovers don't happen correctly.

        We do our best, and we offer SLAs for a reason. If we fail to meet them, you don't pay us.
        But threatening to sue will only put you in my email trashbin, because at that point, I don't give a fuck about your money, and frankly, any

    • Another moronic comment by Viol8. No provider offers 27/7/365. Stop drunk-posting on Slashdot and actually read an SLA for a change.

    • Hey, they're FIVE NINES. They just happened to use up all their nines for the next 300 years.

  • Don't put all your eggs in one basket.

    Variations of that statement date back to the 17th Century.

    How about a big golf clap for the cheap-ass humans who still haven't fucking learned.

    • by UPi ( 137083 ) on Tuesday June 08, 2021 @06:27AM (#61465388) Homepage
      I'm going to make a wild guess here and say you never in your life operated a real life website with an actual user base, or at least, not in the past decade. If you did, you would know that the number of hurdles you need to clear for a three-nines uptime. CDNs typically have a service level agreement much better than that, and tend to keep it too, so they would be your least concern.

      You could try to contract with multiple CDN's and set up automatic fallback, devise scenarios where you can actually test your fallback, etc. But that will be money and engineering efforts spent on an error scenario that might not be happening this year or next year, and pose an opportunity cost for you that you could spend on improving your site in better ways.

      • The user based CDNs like arc.io seem pretty interesting as alternatives.

      • by Junta ( 36770 )

        Note that the commentary could be considered leveled at an individual site operator (in which case, your response is valid, it's impractical for an individual site owner to cover all the contingencies) or at the industry at large, in which case I think it's a pretty valid concern.

        There's a handful of companies that each represent single points of failure for massive portions of the internet, because hundreds of companies all pick the exact same providers for key services. The emphasis on consolidation in th

    • Don't put all your eggs in one basket.

      Here speaks someone who doesn't understand how the internet functions. They're not in one basket which is why they use CDN servers. They use multiple CDNs in multiple countries in order to deliver content more locally so it gets to that area faster than say the BBC only being delivered direct from the UK. Therefore a CDN going down in one area is only going to mean that area is affected, not the rest of the world. Certainly here in the UK I noticed no issue with any of those sites.

      • Don't put all your eggs in one basket.

        Here speaks someone who doesn't understand how the internet functions. They're not in one basket which is why they use CDN servers. They use multiple CDNs in multiple countries in order to deliver content more locally so it gets to that area faster than say the BBC only being delivered direct from the UK. Therefore a CDN going down in one area is only going to mean that area is affected, not the rest of the world. Certainly here in the UK I noticed no issue with any of those sites.

        Also:

        "Today’s issue isn’t limited to a data center in particular. Fastly calls it a “global CDN disruption” and it sounds like it is affecting the company’s network globally."

        Perhaps the "egg" we're relying far too heavily on these days, is the CDN network itself. Removing the turbocharger, should not disable your whole damn engine.

    • by StormReaver ( 59959 ) on Tuesday June 08, 2021 @07:10AM (#61465476)

      Don't put all your eggs in one basket.

      These companies all do a basic cost/benefit analysis: do the CDN outages cost us less than other alternatives (including self-hosting)? Do the benefits of the CDN when it is operating outweigh the costs when the CDN goes down? Apparently they do, or the CDN's wouldn't still be used.

      • These companies all do a basic cost/benefit analysis: do the CDN outages cost us less than other alternatives (including self-hosting)? Do the benefits of the CDN when it is operating outweigh the costs when the CDN goes down? Apparently they do, or the CDN's wouldn't still be used.

        This is correct, but I think that some companies actually want to have another party to blame when crap breaks, which is part of why they outsource it. In the recent past I worked for a Fortune 500 company that was technically an "international" company, but in reality North America was the only market that they really cared about. I'm not going to name them because I don't want to give them free publicity because they are kind of dicks. Anyway, I did internal support on a SASS type product we sold to

    • by bws111 ( 1216812 ) on Tuesday June 08, 2021 @07:35AM (#61465536)

      The law of diminishing returns goes back almost as far. How about a big golf clap for people living in some fantasy world who still haven't fucking learned.

      Perfection can not be achieved. At some point you must say 'enough resource has been spent on this, I accept the remaining risk'.

      When you are driving your car you probably wear a seat belt and have an airbag. Do you also use s 5-point belt and a HANS device like racers do? Do you have a motorcade surrounding you like the president does? Why not? Are you just a cheap ass human who still hasn't fucking learned'?

    • How about a big golf clap for the cheap-ass humans who still haven't fucking learned.

      I'm more amused by the implication that we all AREN'T "cheap-ass" humans. Ah the lies we tell ourselves to feel superior. Now put those items back on the shelf and wait for the sales to be over so one can pay MSRP like a good little consumer.

  • by v1 ( 525388 ) on Tuesday June 08, 2021 @06:40AM (#61465410) Homepage Journal

    https://downdetector.com/statu... [downdetector.com] has a nice graph showing the spike in reported problems, it looks like it's resolved though?

  • by coofercat ( 719737 ) on Tuesday June 08, 2021 @06:57AM (#61465442) Homepage Journal

    I wonder if any of the affected companies will now look to be able to run with multiple CDNs and then be able to switch out Fastly to Akamai or Cloudflare or someone?

    When I last worked in this area, the CDNs made it tricky - first of all, they work very hard to sell you into their (proprietary) edge products. Secondly, their commercials mean that you pay X per month for Y amount of traffic (or less). You obviously set Y to be higher than your average because going over it costs you more. Thus, the amount you pay (X) is pretty high - and you couldn't afford to pay it twice. They won't let you pay a sensible amount per byte, so you always end up having to "over provision" by some amount. Even the likes of Amazon haven't been able to knock that craziness out of the market, although they do offer a pay-as-you-go CDN option, which I guess you could use as your backup. How on earth you pay the AWS bill though is another question entirely - I'll bet you couldn't sue it out of Fastly.

    CDN sharing is technically not too hard (so long as you didn't get into bed with one provider). Getting the commercials in place to actually do such a thing is much harder - but I suspect big sites like SO and Github can probably do it. Lesser mortals no so much.

    One wonders if the bigger sites would start to run their own CDNs - perhaps not as comprehensive as (say) Fastly, but "good enough" to run their site if their chosen CDN provider was having a bad day?

    • Comment removed based on user account deletion
    • [...] How on earth you pay the AWS bill though is another question entirely - I'll bet you couldn't sue it out of Fastly.

      Am I right to take the meaning to be that AWS' CDN (CloudFront) is expensive, at least their pay-as-you-go option? We use CloudFront, and find it to be price-competitive with the competition, but would be open to other options if their was a significant savings.

    • Anyone can use multiple CDNs and many CDNs have contracts with one another so if one service is overwhelmed, they can route it through a competitor. For a company like Amazon, Facebook, or Microsoft, you can monitor their traffic and see them route through many different CDNs.
    • But Amazon's shop website was down, because apparently they use Fastly as their CDN

      It has been linked already in the comments, but https://xkcd.com/908/ [xkcd.com]

  • Using CDNs to deliver high-bandwidth resources is reasonable.

    Using them to deliver critical parts of your site is stupid.

    If your scripts have grown so large that you have to transfer them by CDN in order to make your page load in acceptable time, then you are relying too heavily on scripting and your page is going to run like a turd on many people's computers. Learn how HTML and CSS work so that you don't need so much client side scripting.

    • Comment removed (Score:5, Interesting)

      by account_deleted ( 4530225 ) on Tuesday June 08, 2021 @09:32AM (#61465888)
      Comment removed based on user account deletion
      • Your reasoning doesn't really make sense anyway. The megabytes of scripting you're talking about are generally the bits delivered by CDNs regardless of whether the main page is regenerated per page view or not. And "Learning how CSS and HTML work" isn't really the #1 issue generating a page, it's usually the code on the server side that's looking up articles in databases and trying to stitch everything together.

        I just don't understand it... you go to any random website do view source or push f12 and its one massive cesspool of libraries and frameworks and piecemeal loading of elements via separate requests. There are orders of magnitude more crap to support systems within systems than actual content with majority of time wasted on avoidable round trip delay.

        Indeed, half of the Javascript-heavy websites I see are JS heavy because they're trying to reduce load on their (non-cached or limited cached) servers.

        Isn't this what middleware is for?

        • I just don't understand it... you go to any random website do view source or push f12 and its one massive cesspool of libraries and frameworks and piecemeal loading of elements via separate requests. There are orders of magnitude more crap to support systems within systems than actual content with majority of time wasted on avoidable round trip delay.

          Yep, this shit-show is what the GP is defending. That's what CDNs help them with. "We need CDNs so our webpage can be shit!" That's all I can see in their comment.

    • by tlhIngan ( 30335 )

      Using CDNs to deliver high-bandwidth resources is reasonable.

      Using them to deliver critical parts of your site is stupid.

      Then you have no use for a CDN. They're not just for distributing big files because Bittorrent is a bad word. Most CDNs are used to host static files like images.

      A lot of websites are "not working" because the images are not being retrieved in a timely fashion. And images make up a good chunk of most websites and is a critical part of the website.

      The alternative is self hosting the big fi

  • Thankfully an incident like this is incredibly rare, but it does serve to illustrate something that perhaps we don't consider as much as we should would be the way that we are designing and adding fragility in to the internet as it evolves. Yes, OK, in this specific example, the irony is so strong we can taste it, given that the purpose of a CDN is to improve the quality and service for content creators and also to help to insulate and protect against contention and connectivity issues.

    Over the last few
  • by Martin S. ( 98249 ) on Tuesday June 08, 2021 @07:56AM (#61465584) Journal

    There was a time when this sort of thing was a common occurrence, Slashdot its self could bring it about by linking to site, it was so common not only did it have a name, but a Wikipedia page, the Slashdot effect [wikipedia.org].

    As time passed it became less common, lessons were learnt, bug were fixed, fail overs put in place and CDNs were part of that. This will continue, they sky is not falling in, this is an opportunity.

  • Looks like someone has tripped in the wire.

    https://xkcd.com/908/ [xkcd.com]

  • by gbr ( 31010 )

    The Internet was designed to get around single points of failure. Things have become far too centralized, and it needs to be fixed.

    • by msauve ( 701917 )
      >The Internet was designed to get around single points of failure. Things have become far too centralized, and it needs to be fixed.

      1) This wasn't the failure of a single device, it was a failure of a whole set of devices run by one company.

      2) Your understanding of the Internet is simply wrong. The "route around damage" applies to network traffic, not failures of endpoints.
  • They guarantee an uptime of 100%, but if they don't have much incentive to keep that guarantee.

    A downtime of 4.32min gives a 1% credit on the monthly bill. Up to 43.2minutes gives a 5% credit, up to 7.2 hours 10%, up to 14.4 hours 25% and a downtime over 14.4 hours only gives a 50% credit.

    https://docs.fastly.com/produc... [fastly.com]
    • by nadass ( 3963991 )
      Only "Gold and Enterprise Support" receive any sort of "invoice credits" at part of their SLA terms. Every other customer level receives nothing for downtime.

      The best part? Customers must request within 30 days of the affected billing period; Fastly will apply credits 2 months after the credit concerns have been settled.

      In other words, no big deal.
  • I'm not sure how many guys named "Fred" go by the name "Steve-O", live in Siberia, and are into moroccan cooking using only ingredients found in Peru. Might be zero, might not be. Therefore the number is "countless", correct? Either "countless" merely means "uncounted", or it effectively means a number so large that there appears to be no way to count it.

    The number of websites impacted could be determined. They aren't "countless" just because the author chose not to count them.

    Suggested alternatives include

  • by Somervillain ( 4719341 ) on Tuesday June 08, 2021 @11:11AM (#61466142)
    Fastly is a low-end CDN for porn or medium-sized blogs. They're also the cool-kid disruptor. However, the established players are much better at this.

    One of our architects wanted to be part of the cool club and found out Fastly was cheaper than Akamai. We rolled out a pilot and spent almost a year working with Fastly services to figure out why our services were performing inadequately in Asia and Latin America...like dial-up speeds. We put the service back on Akamai and everything was golden...reliable, no-drama, cost a bit more, but the downtime cost us WAAAY more than the difference in service costs...not to mention the cost of a good chunk of our staff trying to troubleshoot Fastly's many issues.

    It goes without saying, everytime we reported an incident, they'd blame us and we'd spend a week sending info proving that it was actually their fault...so to them, it was always our fault, until they actually looked at it and it wasn't....pretty unprofessional.

    I guess it's good enough if you only do business in the US or you just don't care about reliability or availability...but we do and that architect was almost fired for the Fastly fiasco.

Real programmers don't bring brown-bag lunches. If the vending machine doesn't sell it, they don't eat it. Vending machines don't sell quiche.

Working...