Akamai Having Problems?

A reader writes:"It appears that sometime during the night, Akamai had some problems causing some connectivitly issues with many hosts thoughout the night. Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs? " Not much details - but I can confirm having problems this morning. Thanks to alert readers for pointing that they were having "DoS related issues" and that service was restored as of 1400 GMT.

I thought they do file hosting also

[nev4]

Akamai also hosts files (images, binaries) for many major websites. Seems like they have some pretty insane bandwidth too...

Perhaps....

[Anonymous Coward]

Perhaps it is because of karmatic [slashdot.org] pretending to be posting mirrors of various stories on Slashdot over the last 24 hours, but instead using Akamai as an open proxy to mirror the sites for him:

• here [slashdot.org]
  
• here [slashdot.org]
  
• here [slashdot.org]
  
• here [slashdot.org]

Nice bit of bandwidth theft, there.

Redundancy

[some1somewhere]

Their system is supposed to be distributed in such a way that any major outage in a section of the internet would not affect their overall ability to deliver the content, so presuambly any outage an ISP would not hit their too hard.

BTW something interesting:
http://a1.g.akamaitech.net/6/6/6/6/w ww.peacefire.o rg/bypass/Proxy/akamai.html

several possibly related outages over the weekend?

[Servo]

At work we lost connectivity to a handful of remote sites located in the Northeast, Midwest, and Southeast. Other sites in the same region but different cities were not affected. I was told it was a fiber cut on AT&T's backbone.. wonder if it has anything to do with this.

Report from Akamai

[Anonymous Coward]

Our Akamai rep tells us that it was an issue with a software version rollout. They flushed all their image caches, and effectively caused a DOS on themselves.

Re:Single Domino Theory Revisited

[dalillama]

I know-- But if a significant portion of their load-balancing is knocked out of service, the effects are still substantial. Imagine Google running at 10% capacity...

WORM_AGOBOT.GN

[Anonymous Coward]

I was unable to get to the sites for the major AV Vendors this morning. I chalk it up to Agobot as it
DDOS's their sites. See the following link:

http://www.trendmicro.com/vinfo/virusencyclo/def au lt5.asp?VName=WORM_AGOBOT.GN&VSect=T

DNS flaky for the last 90 days

[sphealey]

As a small company we have a limited view of the Internet, but it seems to us that there have been DNS and connectivity problems thoughout the Internet for the last 90 days or so. I was guessing that there was a DDoS attack against the root DNS servers that wasn't being reported. This would seem to be along the same lines.

sPh

Re:It's simple.

[87C751]

Terrorism.

Funny you should say that. First thing this morning, I noticed my ThreatTray [threat-advisory.com] monitor was blank. I checked, and the Department of Homeland Security [dhs.gov] was not answering. And who do you suppose runs their webservers [netcraft.com]?

What about Bittorrent?

[KhalidBoussouara]

Bittorrent reduces the load on the central server by having everyone who downloads content upload content to other users. Couldn't a similar system be designed for HTTP connections? Obviously it would be designed with much smaller files in mind and with less overhead.

I realise no one give a shit about some large company's bandwith but for small community sites it could really make a difference. They wouldn't have to pay for a company to mirror their site and would save on bandwith costs.

This wouldn't work for server side scripts (as the HTML output would be different for every user) but for static HTML and images it would be perfect.

Re:I thought they do file hosting also

[Misch]

Like Livejournal [livejournal.com]. They use Akamai for hosting userpics. This morning my buddy list was having trouble loading the images. (as they note here [livejournal.com].)

I don't think it was just Akami.

[pedantic bore]

I was having all kinds of problems browsing the web last night: about half the sites I tried to visit (including slashdot and freebsd.org) simply failed to connect. The others were perfectly fine. I didn't see any pattern to it, but I wasn't looking very hard.

Since I've had problems like this with my ISP, I figured it was something local. I guess not.

OK, moderate me redundant because now I see a million other people saw the same thing...

I've noted a big problem...

[Eggplant62]

.. with akamai-hosted sites that has an odd effect in Mozilla Firefox 0.8 on Linux. A combination of Firefox doing an unnecessary reverse lookup on the IP that's being connected to (this is in addition to the regular forward lookup to get the IP, and waits until timeout, usually 30 seconds) and akamai's lack of any reverse zones configured for their boxes.

A buddy of mine worked through further diagnosis to reveal this problem and registered a bug report with the MozDev team, however, after he contacted Google to inform them of the problem, they put in a blank in-addr.arpa zone file for their IP's, which resulted in an immediate negative result on that reverse zone lookup. If the rest of akamai would get on the stick and do the same, the problem would be history.

Haha

[Short Circuit]

Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs?

Don't you see the irony? How much of the internet populace depends on Google for their searching needs?

I suspect the problem here, as there, is that there aren't many who can compete at a service level.

Here's what's happening

[Anonymous Coward]

Currently akamai is configured as an open proxy. This is obviously bad (for them), since anyone can steal the service they're selling for big bucks.

Try:
http://a40.g.akamaitech.net/7/40/1601/1d/i mages.sl ashdot.org/topics/topiclinux.gif

(it works with ANY URL)

Obviously, they noticed it, and tried to fix it. Their fix turned out to block valid customers (like Apple, as has been mentioned), so now they have rolled it back to the free-for-all setup.

They're probably working on a better fix right now.

I'd say it's near a natural monopoly...

[Kjella]

...for those that don't know, a market where it is unprofitable to be the 2nd company around (usually, you can sell cheap because the major company wants to reap profits). A small "Akamai" competitor is no competitor at all, really. You need to have a similar huge network in order to compete. They would undoubtably clash and one would come out as the winner.

So well, if it hadn't been Akamai it'd probably be someone else. Of course, one company can still build a helluva redundant network, if they want to... it's just usually not cost-efficient.

Kjella

latest advisory

[john_uy]

from contol.akamai.com

Akamai is aware of a service interuption earlier today affecting content delivery.

We have identified the root cause and have implemented the fix. Issues retrieving content should be decreasing or resolved. Updates will continue to be posted on the Akamai Edge Control Management Center.

so there is something wrong with their cdn. so much for 100% availability. my guess, all the edge servers were ok but there may be a problem with their noc or software.

:)

Re:24/7 Application Uptime

[djh101010]

I've been using Akamai for several years now at work, this is the first time we've had any interruption. The bandwith they serve for us for a couple grand a month offsets about 3 times as much cost if we had to bring it in ourselves, our customers get pages in half the time (better than that further away), and with the exception of this morning, _it just works_.

Wish I could point to one of my servers here that hasn't been down unexpectedly in 2 years. I don't think I can. It's cheaper, it's faster, and it's more reliable than trying to serve that content from here; even with this downtime, it's still the appropriate solution.

Now, if they go down _again_, without explaination, it could get messy.

Remember when Akamai DoS'd Microsoft?

[freelunch]

It took Microsoft down for DAYS.

All due to a router config bug introduced by Microsoft.. So it was really Microsoft DoSing themselves via Akamai.

And it would be unfair to blame the router config for more than a few hours of outage. The big problem was the complete and utter paralysis of management on the conference calls.

I don't think the details of that outage have been leaked much. It was quite a hoot talking to those involved during the outage. And it wasn't hard, given the duration.

It is my recollection that the problem related to Microsoft's filtering DNS requests from Akamai.

Re:I thought they do file hosting also

[afidel]

A lot of conent is also pre-pushed. For instance if Apple is going to have an ad campaign involving quicktime movies available from apple.com they will pre-push the content to Akami's servers several days ahead of time so that there is not a sudden rush of requests from Akami's cache engines crushing everything else.

Re:Ah, knee-jerk reactions.

[Anonymous Coward]

> I love how the first reaction when something goes wrong is to replace it, or introduce competiton, or whatever. Yes, there are plenty of times when a service needs competition to encourage it to suck less. But go find me another company that is even remotely prepared to do DNS load-balancing. Verisign? Oh, that's a great idea. Going to start one yourself? Let us know when you have the infrastructure.

*ahem* Speedera Networks [speedera.com]

Re:What about Bittorrent?

[ticktockticktock]

Well with sha1 hashes that wouldn't be too much of a problem since the site could dish those out and the clients can use that to verify that the content indeed is the same as what the site says coming from untrusted parties.

Imagine if a plugin for web browsers advertised their support for "protocol/torrent" when you visit www.site.com sort of like:

GET / HTTP/1.1
Host: www.site.com
Accept: protocol/torrent,text/html,text/plain,image/png,im age/jpeg,image/gif

Then the server, that wishes to have requests distributed would respond sort of like this:

HTTP/1.1 302 FOUND
Content-Type: protocol/torrent;text/html
Location: torrent://www.site.com:6969/03cfd743661f07975fa2f1 220c5194cbaff48451

Where the torrent:// URI would point the person's browser to the tracker to contact and what is after that is the sha1 hash of the content that the client can use to verify the integrity of what it downloaded from potentially untrusted third parties. As long as a user sits at a web page, the plugin could "seed" for others who wish to view the same page. When the user leaves the page, it would stop seeding that content immediately.