Amazon's Cloud Service Has Outage, Disrupting Sites (usatoday.com) 161
An anonymous reader shares a report on USA Today: Portions of Amazon Web Services, the nation's largest cloud computing company, went offline Tuesday afternoon, affected multiple companies across the United States but especially on the east coast. The outage appeared to have begun around 12:45 pm ET. It was centered in AWS' S3 storage system on the east coast. Many of the services that firms use AWS are for back-end processes, and therefore not immediately visible to consumers, though the outage could disrupt customer-facing activities like logins and payments. At least some websites that appear to be affected are: Airbnb, Down Detector, Freshdesk, Pinterest, SendGrid, Snapchat's Bitmoji, Time, Buffer, Business Insider, Chef, Citrix, CNBC, Codecademy, Coursera, Cracked, Docker, Expedia, Expensify, Giphy, Heroku, Home Chef, iFixit, IFTTT, isitdownrightnow.com, Lonely Planet, Mailchimp, Medium, Microsoft's HockeyApp, News Corp, Quora, Razer, Slack, Sprout Social, Travis CI, Trello, Twilio, Unbounce, the U.S. Securities and Exchange Commission (SEC), and Zendesk.
The dashboard of Amazon Web Services, which tracks the status of the service, is unable to change color, Amazon said. It is because the status dashboard also runs on the service that is down.
The dashboard of Amazon Web Services, which tracks the status of the service, is unable to change color, Amazon said. It is because the status dashboard also runs on the service that is down.
I guess that explains Strava (Score:3)
I couldn't upload or view my run this morning. I was a bit upset, but I guess it can wait.
Re: (Score:2)
You could go for a run while you wait...
Re: (Score:2)
Also affected... (Score:2)
Hosted connectwise - broken
Solarwinds/logic now MSP services - broken
Imgur is down
Amazon itself (music app will not connect, viewing past orders broken, probably more)
Re: (Score:2)
CNN is slow
No need for cheap shots.
Humor aside, https://www.minds.com/ [minds.com] was having issues also.
Re: (Score:2)
Too bad ... (Score:1, Funny)
Too bad it doesn't disrupt the ads on this site
Re:Too bad ... (Score:5, Insightful)
Meh. Finally gave in and installed Ad Blocker. Slashdot easily has the most invasive ad structure of any web site I go to.
Can confirm (Score:2)
Re: (Score:2)
They are specifically saying the issues with S3 are in the US-Standard and US-East-1 regions. If you have specifically put your stuff in a bucket in another region, you are probably fine.
Re: (Score:2)
They are specifically saying the issues with S3 are in the US-Standard and US-East-1 regions. If you have specifically put your stuff in a bucket in another region, you are probably fine.
Which is exactly why this isn't some catastrophic situation, since anyone who needed a highly available site would have it hosted in more than one availability zone.
Re: (Score:2)
They've also had cross-region S3 bucket syncing since 2015. If you really need it to be up, spend a few more bucks on automatically syncing between say US-East-1 and US-West-2.
If both of those go down at the same time, it's a very bad day for Amazon.
All good (Score:4, Insightful)
My website is on my own server network and is working fine.
Because I have not built up an infrastructure depending on computers I don't own, don't control, have no ability to see to the physical and network security of, where I don't have any control of reliability, redundancy, backup, availability of resources, longevity, OS level, OS and other software updates...
Oh yeah, and my costs are far less than the monthly dollop of blood extracted by cloud services.
"Live by the cloud... die by the cloud."
Carry on, suckers.
It's not "no dependencies" as much as "fewer" (Score:4, Insightful)
I have not built up an infrastructure depending on computers I don't own, don't control, have no ability to see to the physical and network security of, where I don't have any control of reliability, redundancy, backup, availability of resources, longevity, OS level, OS and other software updates
I'm interested in how you eliminated dependencies on your home ISP's DHCP server, backbone routers, the DNS, the OCSP server of the CA that provided your site's TLS certificate, etc. And without advertisement exchanges or subscription payment servers, how do you afford to keep your server powered on and connected to the Internet?
Re: (Score:3)
I have not built up an infrastructure depending on computers I don't own, don't control
I'm interested in how you eliminated dependencies on your home ISP's DHCP server
I have static IPs.
First, most home ISPs charge extra per month for that. Second, Comcast requires subscribers with a static IP to either forfeit the static IP or rent and use Comcast's modem, which reintroduces "depending on computers I don't own, don't control", and moving to an area whose cable company is not Comcast can prove cost prohibitive, particularly if your work is unrelated to your website.
I don't serve HTTPS. I'm not collecting information about you, or serving illegal or secret information.
Making even a completely public, completely static website available over cleartext HTTP and not HTTPS has three consequences.
Re: (Score:2)
Dyn is just as vulnerable as AWS [unionleader.com].
Re: (Score:1)
Serious comment here:
The only problem with running your own infrastructure is load estimation. When preparing a business case for running your own infrastructure, any deficit in load has to be paid for by the business. With using cloud services, the load deficit does not exist, and the company only pays for what it uses.
It is for this reason that cloud services are always cheaper than in-house infrastructure.
If your company decides that the many advantages (listed in the post above) of in-house infrastructu
Re: for many (Score:1)
even downdetector.com is down (Score:4, Funny)
Re: (Score:3, Funny)
Re: (Score:1)
Re: (Score:2)
The first time I looked at that URL (all lowercase, as in TFS), I read it as I Sit Down Right Now. I'm sitting down right now, too, so what's the big deal?
Re: (Score:1)
Re: (Score:2)
PEAC
Problem Exists At Chair
Re: (Score:2)
Domain names are not case sensitive (RFC 4343 [ietf.org]). The remainder of the URL may be (and frequently, not not always, is), depending on the host software. This is also true for SMTP addresses.
Re: (Score:2)
Domain names are not case sensitive (RFC 4343 [ietf.org]). The remainder of the URL may be (and frequently, not not always, is), depending on the host software. This is also true for SMTP addresses.
It's a sad commentary on the current state of /. that you felt compelled to explain what should be (and would have been just a few years ago) common knowledge to the site's readership. Here's the reason I mentioned the casing of the URL as presented in TFS: as you may have noticed, domain names are frequently presented in mass marketing with casing in order to clarify/emphasize what the site is about or to make it easier to remember what may be a long string of characters. "IsItDownRightNow.com" and "ISitDo
partly cloudy today (Score:2)
The solution (Score:5, Funny)
Re: (Score:2)
Yo dawg, I heard ah fuck it.
Re: (Score:2)
Ezekial saw the cloud
Ezekiel saw the cloud
Way up in the middle of the air.
Now Ezekiel saw the cloud was great.
Way in the middle of the air.
Chorus:
And the big cloud is run by Bezos, good Lordy
And the little cloud run by Sundar Pichai
In the cloud in the cloud in the cloud good Lord
Way in the middle of the air.
Who's that yonder backed up tonight?
Way in the middle of the air
. It must be the internet of the people that’s right,
Way in the middle of the air.
Ch
Yeah, this got me as well (Score:2)
Our mobile app hosts most of it's images in S3. We're basically displaying blank screens to our customers right now.
Re: (Score:2)
Does it also host your spell checker?
Re: (Score:2)
Re: (Score:1)
"Our mobile app hosts most of it is images in S3."
That's what you wrote above. I guess you're the one who doesn't understand when to use its vs it's.
Re:Yeah, this got me as well (Score:4, Informative)
Re: (Score:2)
Yore*
Re: (Score:3)
shut up, dickhead
Re: (Score:2)
Of course I do. They're different from a spell and grammar chess.
Re: (Score:1)
Re: (Score:2)
In most cases users don't care about outages either. It takes actual dataloss / monetary loss for users to care about something. Being inconvenienced will be brought back to "oh these silly computers are being silly again".
Re: (Score:2)
Wow AWS Goes down also? (Score:1)
Amazing AWS goes down randomly like Azure does? Huh?
Re: (Score:2)
More amazingly, Microsoft runs a system on the competitor's cloud instead of Azure. (I presume that HockeyApp is an acquisition that they haven't migrated, but still it's amusing.)
Re: (Score:3)
I caught that too and thought it was interesting. One other interesting thing about an application that really uses cloud services to their full potential (rather than just as an expensive VM/VPS) is that since the services are not commoditized/standardized there is a lot of cloud vendor lock-in. E.g. if you build a huge web app around AWS you're going to have a lot of rework to do (to some extent depending on how well you modularized your code) to migrate to another cloud provider.
Re: (Score:2)
Yeah, it's almost like it's a service run by humans, and humans are fallible no matter who their employer is.
Ahh, the cloud. (Score:5, Insightful)
AKA "just someone's else computer".
This outage is being going for over an hour now but, according to Amazon, their services are green all across the board [amazon.com] with "increased error rates". Almost feels like they're trying to cheat out their own SLAs.
Re: (Score:3)
Grrr, it's "someone else's". Thanks phone.
Re: (Score:2)
Nah, their service health dashboard was also affected by the issues. They've fixed it (the status page) and it's showing wider ranging issues.
Thingiverse (Score:2)
This explains why a lot of images won't load on thingiverse.com
losing nines (Score:1)
nine-nines ...
eight-nines
seven-nines
six-nines
In a few minutes, it'll be two hours down.
Why are they lying? (Score:2)
Their whole says GREEN and "No Recent Events". What a bunch of liars. [amazon.com]
"Increased Error Rates" my ass
This will lead to massive lost respect for AWS!
'Fess up, don't be lyin.
Re: (Score:2)
blew my href on the word 'dashboard', sorry
Re: (Score:2, Informative)
Ha!
Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.
Re: (Score:2)
Just saw that and came here to update the info. That's pretty funny!
Great time to handle failure cases... (Score:2)
Re:Great time to handle failure cases... (Score:4, Insightful)
internet images down... (Score:2)
Increased productivity! (Score:1)
Irony (Score:2)
isitdownrightnow.com
Yep.
what the fuck is with these ads? (Score:1, Troll)
whats with the shitty half page ads that wont go away /.?
Russia (Score:2, Funny)
Russia did it
Start time (Score:1)
GitHub (Score:1)
No GitHub icons either.
Shattered SHA1? (Score:2)
It's not all bad (Score:2)
At least there's something positive about this outage.
AMZN (Score:2)
Problems with the Cloud (Score:5, Insightful)
If it rains, you're in trouble.
Something else going on? (Score:1)
That's what you get (Score:2)
When you rely upon someone else to handle the shit you should be handling, this is your just reward.
The victim list is very long (Score:1)
Apple iCloud and iTunes services were intermittently going offline the whole morning yesterday. I had trouble viewing my TV episodes.
Re: (Score:1)
Cost and capability.
Re: (Score:2)
This. All jokes aside S3 is normally a great service at a very competitive price.
Re: (Score:2)
Amazon is not cost-competitive.
I get more value and redundancy out of spinning up dozens of 1-2 GB Digital Ocean droplets than I do with Amazon--especially now that they have their LBS service and an equivalent to EBS.
A 2GB RAM digital ocean droplet gives you 40GB of disk space for $20/month, or around 50 cents/GB. To get the same triple redundancy that S3 gives you, you're paying $1.50/GB, compared to $0.023/GB that S3 costs. Plus you've got to manage the redundancy yourself. Or, better availability, you can have S3 mirror your data between regions, and you're paying $0.046/GB.
I've done the latter -- mirrored my data between regions, so the us-east-1 outage took a single parameter change and service restart to point my
Re: (Score:2)
I've done the latter -- mirrored my data between regions, so the us-east-1 outage took a single parameter change and service restart to point my app to the us-west region (it could be done automatically, but full-region outages like this are so rare, I haven't bothered)
This is important: if you need HA, you should mirror across regions. Because regions go down, and this is not the first time. They will continue to go down in the future. Amazon.com is still up because they know that this is a problem, and they prepared for it.
Re: (Score:2)
That's easier said than done though. See for example the Amazon AWS status portal which stated things were just fine for far longer than they were because Amazon couldn't update it. Doing cloud based applications right is neither as inexpensive or as simple as a lot of people were led to believe/preached.
Re: (Score:2)
Doing cloud based applications right is neither as inexpensive or as simple as a lot of people were led to believe/preached.
I just think of it as a normal data center, except you don't have to drive down there to install a new box or figure out what's wrong.
So AWS is just more convenient, that's all. Still requires basically the same expertise as a datacenter, minus wiring.
Re: (Score:2)
In my own personal experience, good failover is not as easy as expected. Never.
Re: (Score:2)
You gotta test it.
Re: (Score:2)
Nobody expects the Spanish Inquisition.
Re: (Score:2)
Re: (Score:2)
Well, obviously, Amzaon's redundancy is either useless on non-existent, so I don't know what you think you're paying for.
There's a difference between durability and availability. I'm mostly interested in durability for objects I store in S3, but for when I care about availability, I mirror them across regions.
Re: (Score:2)
It is kinda cute when people try to think "cloud storage" in terms of HDDs alone.
Re: (Score:2)
Re:So why use these large cloud services? (Score:5, Informative)
Seriously...I would suspect this is due to an attack of some sort. Just a hunch.
The last time Microsoft's Azure platform had a huge, sustained failure, it was just an internal screw-up, not an attack. I've got no reason to think Amazon's east coast problems are any different. Not to say it couldn't be an attack, but no reason to think one way or the other, and lots of reasons to think "screwed up" - because that has happened at Amazon and elsewhere in Big Cloud many times.
And if so, isn't it the case that the larger the company, the bigger the target?
Yeah, but they've also got the enormous resources to help fend off problems that would crush a smaller provider. Works both ways.
Re: (Score:1, Insightful)
You're right. I'll have my devs spend a year learning and implementing some small cloud provider only for them to go under.
Also, the smaller ones go down too, it's just not noticeable or newsworthy.
Re: (Score:2)
Re: (Score:1)
Google is pretty much an also-ran at this point so the question is AWS vs. Azure (or of course self-hosting but we'll assume you really want to do cloud and I can't talk you out of it). In my experience the answer depends on your application. If you're building a new from the ground up web-based application AWS is probably the front-runner. If you're migrating an existing in-house system and want to do things more incrementally, do something hybrid with your in-house stuff, etc. Azure is far simpler to get
Re: (Score:1)
Your question is unclear. Are you asking why Google is an also-ran or why AWS/Azure split the way I suggested. I'm going to assume the former. It's got nothing to do with the infrastructure, it's about the business, offerings, service/support, and wherewithal. Simply put Google doesn't have 1) the cloud services customer base that either of the other two do, and 2) the cloud product variety/maturity that the others do. Google also has shown no serious interest in improving their cloud services and/or doing
Re: (Score:2)
Google's cheaper than AWS until you're dealing with serious scale.
Has Google started running their own websites on their cloud yet?
Re: (Score:2)
So why use these large cloud services?
Because this outage doesn't affect anyone who cared about ensuring their cloud systems had high availability. If they did they would have had servers hosted in multiple availability zones. No cloud provider, or home-brew solution, is going to be highly available if you only have servers in one server farm.
So it appears either us-east-1 or us-east-2 was down. If you wanted 99.9%+ up-time you should have servers in us-west-1 or us-west-2 also. This isn't rocket science.
Re: (Score:2)
Seriously ... these outages are usually caused by the organization itself, either by an immediate technical mistake, or when a minor glitch cascades into something major due to a design flaw.
I'm sure Amazon is a constant target of hackers, both pimply faced youths and of shady state-sponsored black-hats. But taking out Amazon isn't a very interesting goal.
Re: (Score:2)
...because today it was nobody's fault.
If your boss wanted a highly available site and you only hosted it in a single availability zone, it would still be your fault if it goes down (unless you have records of him denying funds for the second zone). Hosting it in the cloud doesn't mean you can start being stupid.
Re: (Score:2)
I was working with a customer today that was using opsworks to provision into us-west-2, and it was failing as well. Did the issue show up on the dashboard? Nope. Were they impacted in a DIFFERENT zone than what had the issue? Yep. As much as they would like to say, there were issues across the board as a result of the dependency on us-east-1.
Re: (Score:2)
Re: (Score:3)
This. Clearly a lot of people jumped into cloud "to save boatloads of money" (same reason so many jumped into outsourcing. Saving tons of money is not often a good reason to do something. Usually you can make incremental savings but it's never what the salespeople or service promises because those prices ignore things like redundancy, etc. In the end doing it right ends up costing about the same as you were paying before, maybe a little less or a little more and maybe you gain some more features, but it's a
Re:The irony of this... (Score:4, Informative)
Yeah, that guy is a douche, and you don't want to work for him.
I work for a company that hosts stuff in AWS, and we are doing cross-region backups, as well as a weekly dump to our on-prem servers. Yes, AWS has a good amount of redundancy between the multiple availability zones per region, etc. But why take the chance, when it's trivial to dump your stuff on a scheduled basis to another region, or even to something as cheap as a Synology box or one of those workstations being recycled into a FreeNAS box in the office?
Re: (Score:1)
I decided to go elsewhere.
You missed a great opportunity!
CxO : Dammit! The Cloud is Down - Do something!
You: "Right, I'm on it" //heads for exit
CxO: Yells "Where are you going at a time like this"???
You: "To buy a buggy whip For that Horse you're going to be needing".
Re:Coffee Doesn't Meet Bagel (Score:5, Funny)
CMB messaging has silently stopped.
The Cosmic Microwave Background has stopped?
Jeez. I didn't realize that Amazon was that important.
Re:ONLY EAST COAST (Score:5, Funny)
West coast best coast, east coast least coast.
Re: (Score:2)
Omaha: We Don't Coast [wedontcoast.com]
(Somebody actually got paid to think up that stinker.)