Gmail, Slack, Amazon, Spotify, Twitch, Hulu, Google Are Suffering Outages for Some Users 34
A wide-range of services including Gmail, Google, business collaboration service Slack, Amazon, Twitch, Hulu, and Spotify are suffering outages, several users and readers have reported. The reports started to come in half an hour ago, but the cause of the disruption is yet to be identified.
Update: Verizon says there is a fiber cut in Brooklyn. Further reading: Verizon Fios is experiencing outages on the East Coast.
Update: Verizon says there is a fiber cut in Brooklyn. Further reading: Verizon Fios is experiencing outages on the East Coast.
Verizon (Score:2)
Related to the Verizon fiber cut in Brooklyn?
Re: (Score:2)
Re:Verizon (Score:5, Funny)
I think you've cut to the root of the problem!
The cause of the disruption... (Score:5, Informative)
The cause of the disruption is a cut fiber in Brooklyn:
https://twitter.com/VerizonSup... [twitter.com]
Re: (Score:3)
Somebody did not do their due diligence when planning and verifying their redundant connections...
Re: (Score:2)
Over a decade ago there was fiber cut in the Midwest (farmer with a backhoe) that took out about 1/3 of the cross-country Internet capacity. Turns out that several providers ran their primary and redundant connections in separate conduits that ran through the same trench.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
A classic mistake. Same trench, same bridge, same tunnel, relay stations on same power circuits, etc.
Done right, you find these and work around them. Done wrong, you notice when it hits the fan, and sooner or later it will.
Re: (Score:3)
Re: (Score:1)
Re: (Score:2)
That costs money.
Which is really sad, that a major part of our infrastructure can get hit, because a company is too cheap to plan for redundancy, and rerouting. Because it is cheaper to take an outage and fix the problem when they happen than having a robust environment where outages don't happen.
Re: (Score:3)
Yeah, It came in pretty handy when the battery died in the primary vehicle a couple of months ago.
Re: (Score:2)
Why? I can rent a car on short notice. Good luck doing that with a fiber backbone.
Re: (Score:2)
Unfortunately, very true.
Re: (Score:1)
Apparently you can plan and verify, but when the datacenter where you have some servers are no longer accessible and you ask the company that owns the datacenter and they tell you the company that handled running the fiber in fact actually ran redundant lines along the same path instead of different paths unlike originally stated in their contract when the datacenter was being built, so all the fiber to the datacenter was cut at the same time.
Re: (Score:2)
Well, that is what I mean with "verify". Sure, it is expensive having somebody physically verify where the actually fibers are. I have done things like this on occasion, including also looking at power ingress. But there was always somebody in management that really wanted to know.
Chunk of the internet? (Score:2)
I do not think that this is a problem with hosting providers, but a problem with ISPs. There must be a trunk down somewhere, I can't connect reliably to a number of non-commercially hosted services.
Sure about the cause? (Score:2)
Re: (Score:2)
Google has a large office in Brooklyn (or downtown New York City).
Re: (Score:2)
Re: (Score:2)
cause BGP, which is the twin brother of DNS
the duality of evil in tech
Supposed to survive war (Score:3)
The old telco networks were supposed to be resilient enough to withstand severe damage and disruption from wartime. Old Bell built a tough network. But recent issues have shown that modern telecommunications infrastructure is incredibly fragile. Its a bit of a joke. Especially when the protocols the internet is built on are designed to be self healing. How do we have single points of failure anywhere in CONUS? We knew the ISPs were pissing away the tax payer dollars they've been given but I don't think we understood just how much.
Re: (Score:3)
probably due to CDN's and routing you based on location to the nearest source of data. if there is a fiber cut there might not be code to route you to another CDN server for the data
Re:Supposed to survive war (Score:5, Insightful)
To be "fair" the network did survive and is alive. They're experiencing latency, dropped packets, and as a result a loss of connectivity -- but the hosts _are_ reachable.
What this really shows is that their capacity planning is insufficient for a fault in the network. They're being hounded by Congress now that they have plenty of bandwidth, 'See, look, everyone's using it, and it works, even with the virus, even with usage spikes, you've got plenty of bandwidth, you don't need to restrict users.' The come-back is that they clearly don't have plenty of bandwidth - lose one fiber, and they've lost the ability to serve the needs. The compromise is to limit users when there's a detected fault. (Maybe that's not as easy -- but it is doable.)
Capacity planning is the fault. They need at least double the required capacity, at _least_ for major population centers (or datacenter-population centers) in order to be able to serve the needs in case of a fault. When we say "double" we mean coming in on two different cables, not two fibers in the same cable. All of that will increase costs further, and you can expect that those costs will be passed on (along with a "slight" percentage increase to profits). Did Verizon end up routing traffic through Level3, who over-advertised their capacity? Or did Verizon under-purchase the capacity to these data centers?
Re: (Score:2)
Billions of dollars (Score:2)
Everything is broken (Score:2)
An interesting analysis on how meritocracy and careerism produce incompetence:
https://www.tabletmag.com/sect... [tabletmag.com]
Ye Olde Fibre-Seeking Backhoe (Score:2)
Strikes again!
Or maybe it's finally The Storm!(TM)
Begs the question... (Score:2)
Time for those who got rich on internet to pay up. (Score:1)
Probably shouldn't have messed with basic rights (Score:2)
Not saying that it was the aliens but it was the aliens.