US Amazon.com Website Down For Over 1 Hour

CorporalKlinger writes "CNET News is reporting that Amazon's US website, Amazon.com, has been unreachable since 10:30 AM PDT today. As of posting, visiting www.amazon.com produces an 'Http/1.1 Service Unavailable' message. According to CNET, "Based on last quarter's revenue of $4.13 billion, a full-scale global outage would cost Amazon more than $31,000 per minute on average." Some of Amazon's international websites still appear to be working, and some pages on the US Amazon.com site load if accessed using HTTPS instead of HTTP."

Beer on the server?

I guess somebody spilled beer on the servers? I had no idea the guys from FARK also ran Amazon.

But...

It works just fine for me right now.

Also now you are Slashdotting it!

This will surely help

I'm sure the sysadmins appreciate Slashdot sending thousands of requests their way while they're site's already down. While we're at it, maybe we should find someone with a papercut and start squirting lemon juice all over them.

Re:This will surely help

It's really not all that difficult to survive a slashdot pounding for commercial web shops, even for dynamic content. Generally speaking, a popular link is going to generate perhaps 500k views a day for a day and some.

Only exceptions would be if there was a lot of heavy content being served on each page turn, saturation of one's uplink is a possibility - 10Gb links to the backbone aren't that common as yet, and CDNs like Akamai helps alleviate a good portion of that traffic.

My totally unsubstantiated guess is there was some DNS fooage that directed sites to a down cluster or possibly a screwed up CDN leg, but I'll be interested to see what's truly up.

sloth jr

Re:This will surely help

You think traffic from SlashDot would even be noticeable on Amazon's servers? You have some delusions of grandeur there.

Patents

Wait until a patent comes out for: "Taking a web presense offline, to generate discussion about the web presense, thereby increasing awareness about the site." Also, sucks to be the guy that stepped on the surge protector laying on the floor....

Re:Patents

The Twitter guys will have prior art.

So, it finally happened...

Believe me, if you've seen the code that runs that site, it's impressive it runs as well as it does. Try to imagine 900M static binaries that take almost an hour to link because of some tiny little code change, because they can't be fucked to make their deployment system deal with dynamic libraries reasonably.

Re:So, it finally happened...

Believe me, if you've seen the code that runs that site, it's impressive it runs as well as it does. Try to imagine 900M static binaries that take almost an hour to link because of some tiny little code change, because they can't be fucked to make their deployment system deal with dynamic libraries reasonably.

Fuck up a dynamic library and you fuck everything. Fuck up one of those 900M programs and you've fucked 1/900M'th of everything.

What does Amazon's back end compile for? If it's Linux, that's an issue right there. The GNU linker has pathological behavior when linking large numbers of static libraries. I work on a relatively small (~1 million line) codebase and it takes about ten minutes to link. Link it on another platform (e.g. Solaris) and it links in about five seconds.

The problem isn't the huge number of libraries. The problem is that the linker blows.

Re:So, it finally happened...

Um, I used to work there. Believe it or not, there are some people here with real jobs and stuff.

Re:So, it finally happened...

Okay mythical female Slashdot reader, with your unicorn and leprechauns friends chasing chupacabres out of the Amazon codebase. We believe you.

Re:So, it finally happened...

Yeah, it wasn't like that at all. There used to be a few chupacabras, but I think the shoggoths ate them all, and those leprechauns never did any work. They'd just wander around all day, picking people's pockets and teleporting away.

Re:So, it finally happened...

Well, if you worked there in the last 2 years you'd know that the giant monolithic app is dead and not mourned. I drew the short straw and had to sit in all day on the con call when they were taking it down area by area.

D&D did it.

One of the top sellers on Amazon is the D&D 4th Edition Core Rules giftset. It apparently is only shipping to some pre-orders. The geeks are freaking out and untintentionally DoS'ing Amazon.

Re:D&D did it.

I sure am glad you didn't take his statement seriously. Would have been pretty silly to argue a point against that.

They Think I'm a Robot

When I try I get to a page that says they think I'm a robot and I don't have access to see their website.

Well I think THEY are the robot. I don't know if I can win this argument...

The problem isn't Amazon . . .

It's all you people not typing in the write web address. Try it again. Make sure you put in the umlaut correctly. What do you mean there's no umlaut in Amazon.com? *Unplugs toaster and plugs back in Amazon's server* Wait 5 minutes and try again. --BOFH

$31,000 per minute!

would cost Amazon more than $31,000 per minute on average.

Because obviously if someone tries to buy something and Amazon is broken for an hour, they're just going to not-buy it or buy it from a competitor. Because you definitely can't wait an extra hour to place an order when it'll take 2-10 days for the product to get shipped to you anyway.

Re:How is this news?

Because this represents 31k USD every minute.

Re:How is this news?

Because this represents 31k USD every minute.

That assumes that everyone who would have bought something doesn't just try again when the site's back up. Nevermind that the number quoted is talking about a global outage -- this is just a partial outage.

Re:How is this news?

Exactly, except that not everyone that would have purchased their products in those 60 minutes will buy elsewhere. They hour they came back online they could make 1.9 x typical USD per minute. That and the fact that this is not really a holiday season of any sort, so sales are likely nowhere near the peak rates they reach around Christmas, New Years, etc...

Re:We're sorry...

Hmmmm... sounds like they are being DoS'ed by MediaDefender.

Re:do a whois. Looks like DNS got pwn3d.

That whois lookup says absolutely nothing... I could add amazon.com.myserver.net as a dns record too, and it would have nothing to do with the lookups for amazon.com. The trick is to use whois to see what IP address www.amazon.com currently points at.

However, as has been pointed out, HTTPS works, so it's defininitely not a DNS issue. More likely someone along the chain corrupted a pooling link to the main http server and it propogated. I've done the same thing on apache2 servers in the past and had the same result; https still works fine, but http returns an error on key pages.

Re:do a whois. Looks like DNS got pwn3d.

Of course, it's not a hack.

A fully-qualified DNS domain name ends with a dot, so you should type 'whois amazon.com.' instead.
Those "hacked" results you are getting are just bogus amazon.com.foo.bar. subdomains.

Re:OH NOES

That reasoning doesn't really work for me.

You'd have to factor in the ratio of income from the
US site v others (UK, etc.). IMHO, the US site is likely to be more profitable than others. You'd have to plow through an annual report to really know, and factor that in.

The larger flaw, though, is that you're subtracting one minute, when the title states > 1 hour. That implies going on A couple of million US$ in losses, which is significant, as investors don't know the reason, and caution would indicate that it could be recurring, such as the problems SalesForce has had. That hit their stock prices, etc.

The Amazon outage is more complex--TFA indicates that some of their services were unavailable for different amounts of time, etc. What are those service worth? All anyone has is a number--from CNET. Did they do anything like a real analysis, reading quarterly reports, etc? No, by long odds. Amazon does application hosting. What customers were affected, what percentage of the business is involved, and what do CxOs of large clients think?

The odds are actually quite good that many people give a crap. Investors (and CxOs) don't like uncertainty. It wouldn't surprise me to find some Wall Street analyst(s) making calls. Maybe it was an outage on a critical replication server, problem identified, fixed, and will provably never happen again. But maybe not. We'll see.