Akamai Having Problems? 216
A reader writes:"It appears that sometime during the night, Akamai had some problems causing some connectivitly issues with many hosts thoughout the night. Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs? " Not much details - but I can confirm having problems this morning. Thanks to alert readers for pointing that they were having "DoS related issues" and that service was restored as of 1400 GMT.
SBC? (Score:5, Insightful)
Re:SBC? (Score:5, Funny)
Re:SBC? (Score:5, Funny)
Word up!
Re:SBC? (Score:2)
Ha! I had scrolled past it when I got it
Redundancy (Score:5, Interesting)
BTW something interesting:
http://a1.g.akamaitech.net/6/6/6/6/
Re:Redundancy (Score:2)
Thank goodness you didn't you didn't embed that link, since less people will check it. I'd hate to see them get Slashdotted.
Re:Redundancy (Score:2)
"Not much details?" (Score:2, Offtopic)
Come on, Hemos. You're always making dumb grammar mistakes. This place looks so unprofessional as a result.
Re:"Not much details?" (Score:2, Funny)
Apple down, Microsoft up (Score:5, Informative)
Re:Apple down, Microsoft up (Score:5, Funny)
apple trailers (Score:2, Informative)
I thought they do file hosting also (Score:5, Interesting)
Re:I thought they do file hosting also (Score:5, Informative)
Re:I thought they do file hosting also (Score:3, Interesting)
Scalability and bandwidth (Score:3, Informative)
Re:I thought they do file hosting also (Score:5, Interesting)
A glass house moment... (Score:2)
You misspelled the word "misspelled".
Having problems? (Score:5, Funny)
Re:Having problems? (Score:3, Funny)
Comment removed (Score:5, Insightful)
Re:erm... (Score:5, Funny)
Re:erm... (Score:2)
Re:erm... (Score:3, Funny)
Re:erm... (Score:2)
Re: (Score:3, Insightful)
Not a problem with having one company (Score:5, Funny)
Internet Storm Centre has a little (Score:5, Informative)
It appears that websites that use Akamai's distribution system are currently not reachable. Security related web sites effected are symantec.com and trendmicro.com. Virus updates may fail as a result. Further details are currently not available and updates will be posted here as they become available. Thanks to Vidar Wilkens for alerting us of this problem.
According to a post to NANOG, the outage may be the result of a DDOS attack. At this point, Akamai has not ETA for a resolution.
Update 09:45 EST: Looks like some of the Akamai hosted sites start to come back."
You gotta love that "Quiet, well kinda quiet". ;)
Re:Internet Storm Centre has a little (Score:2)
Perhaps.... (Score:5, Interesting)
Re:Perhaps.... (Score:2)
Re:Perhaps.... (Score:2, Funny)
I just went to akamai's site to see if they have any terms of use or anything but all I see is a blank page, so it must be ok.
We do that already. (Score:5, Insightful)
We do that already. Remember when verisign introduced Sitefinder, thus effectively making various services (like spam filters etc) unusable because non-existing domains all of a sudden replied with a valid IP.
NOC Says: (Score:5, Informative)
Re:NOC Says: (Score:4, Insightful)
Comment removed (Score:4, Informative)
One Possibility.... (Score:2)
In this way a few thousand bytes of http requests could make the Akamai servers *EACH* attempt to fetch terrabytes, or more, of data...
Apple.com Slow down (Score:2, Informative)
Blueyonder UK
Re:Apple.com Slow down (Score:2)
What really happened... (Score:4, Funny)
The cleaning lady needed electricity to her vacuum cleaner.
Poor sysadmins.
Re:What really happened... (Score:2)
Re:What really happened... (Score:2)
In 1997 I worked for a local ISP, and sometimes in the evening around 7pm (when no one of us was in the offices), some of the servers were unreachable for a few minutes and rebooted. We investigated for months! Unfortunately it didn't happen, when we stayed there afterhours - so we had no clue what went on.
Finally we discovered it was the cleaning lady, that unplugged one of the servers to plug in his vacuum cleaner...
Now the servers are locked up in a room, where only authorised personnel has
Re:What really happened... (Score:2)
It seems to be a common problem. I have witnessed something like that myself, even though it wasn't any computer I would be responsible for.
I noticed one day that a server was down and inquired about it's status when I met one of the guys who administer it. What I heard was that someone, apparently the cleaning staff, had unplugged the power cable from the Linux server running Apache and some other smaller services. As it was a educational server there were no financial losses, but I don't think that fsck
Discussed on Nanog... (Score:5, Informative)
Rus
Answer (Score:5, Informative)
Of course it is a bad idea.
However, blame that on the other competing services who haven't become cheaper, faster or better at whatever it is that makes Akamai so popular.
I'd say it's near a natural monopoly... (Score:5, Interesting)
So well, if it hadn't been Akamai it'd probably be someone else. Of course, one company can still build a helluva redundant network, if they want to... it's just usually not cost-efficient.
Kjella
Ah, knee-jerk reactions. (Score:5, Insightful)
I love how the first reaction when something goes wrong is to replace it, or introduce competiton, or whatever. Yes, there are plenty of times when a service needs competition to encourage it to suck less. But go find me another company that is even remotely prepared to do DNS load-balancing. Verisign? Oh, that's a great idea. Going to start one yourself? Let us know when you have the infrastructure.
The fact is, we have NO idea what caused this. There's no link to any story anywhere - just one reader report. It could be Akamai's fault. It could be their upstream providers. It could be failures elsewhere in the Internet. Could be someone uploaded a bad zone file. Or maybe some over-zealous backhoe operator slashed some fiber somewhere.
It's probably best to reserve judgement until you have all the facts. (And if you're about to hit the reply button, yes, I'd say the exact same thing if MSFT lost their DNS service).
Re:Ah, knee-jerk reactions. (Score:2, Insightful)
Single Domino Theory Revisited (Score:5, Insightful)
Re: (Score:2)
Re:Single Domino Theory Revisited (Score:2, Interesting)
Re:Single Domino Theory Revisited (Score:3, Funny)
24/7 Application Uptime (Score:5, Insightful)
Re:24/7 Application Uptime (Score:4, Interesting)
Wish I could point to one of my servers here that hasn't been down unexpectedly in 2 years. I don't think I can. It's cheaper, it's faster, and it's more reliable than trying to serve that content from here; even with this downtime, it's still the appropriate solution.
Now, if they go down _again_, without explaination, it could get messy.
YIKES! (Score:3, Funny)
They just found the culprit (Score:2, Funny)
several possibly related outages over the weekend? (Score:4, Interesting)
Report from Akamai (Score:5, Interesting)
Silly Akamai (Score:3, Funny)
Re:Silly Akamai (Score:2)
Re:Silly Akamai (Score:2)
At a certain usage point, it made sense for us to switch from a g123.a.akamai.net/blah/blah type URL to one that has our company's name in it, which is cnamed over to an akamai server. So, the URL looks like a host in our domain to the customers (and their filtering software), but resolves through the cname to the nearest akamai edge server to them. It's more compl
Re:Silly Akamai (Score:2)
Closer to home (Score:2, Funny)
Is it a bad idea to rely so heavily upon one service for our major internet needs?
Yeah. Duh. But, where else can I get a /. fix?
eBay affected also (Score:2, Informative)
Re:eBay affected also (Score:2)
WORM_AGOBOT.GN (Score:2, Interesting)
DDOS's their sites. See the following link:
http://www.trendmicro.com/vinfo/virusencyclo/de
Notice on Akamai Control site (Score:5, Informative)
Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.
Re:Notice on Akamai Control site (Score:5, Informative)
Well, unless you have a *really* bad latency problem, I don't think that's going to be an issue with a problem on May 24th...
DNS flaky for the last 90 days (Score:5, Interesting)
sPh
Re: (Score:2)
Re:DNS flaky for the last 90 days (Score:2)
> having problems? Or do you send all requests
> directly to the root servers?
Our ISP is a pretty big one, and we use djbdns which I believe goes to the roots if the lookup is not cached.
Still, your questions are good ones and we have asked ourselves the same things. Usually when response seems slow in the office it is also slow on various cable and DSL services at employee's homes too though.
Maybe I am just getting old and cranky: "When I was youn
Re:DNS flaky for the last 90 days (Score:2)
Agent (newsreader) reports a winsock error (10060, connection to server timed out) and Outlook (email) reports that the connection was unexpectedly term
That's why (Score:2)
The new version works very well, and will make your network connection seem much faster.
What about Bittorrent? (Score:4, Interesting)
I realise no one give a shit about some large company's bandwith but for small community sites it could really make a difference. They wouldn't have to pay for a company to mirror their site and would save on bandwith costs.
This wouldn't work for server side scripts (as the HTML output would be different for every user) but for static HTML and images it would be perfect.
from their support website (Score:5, Informative)
Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.
Maybe the problem has recurred.
Re:from their support website (Score:3, Informative)
I don't think it was just Akami. (Score:2, Interesting)
Since I've had problems like this with my ISP, I figured it was something local. I guess not.
OK, moderate me redundant because now I see a million other people saw the same thing...
noticed it too. (Score:2)
maybe they autoupdated all their servers and made them reboot?
let's see what updates they have on their support site.
Official Report (Score:2, Insightful)
Re:Official Report (Score:2)
It's completely plausable that a 90 minute outage is a big deal for you, but I have to wonder if it was such a big d
I've noted a big problem... (Score:5, Interesting)
A buddy of mine worked through further diagnosis to reveal this problem and registered a bug report with the MozDev team, however, after he contacted Google to inform them of the problem, they put in a blank in-addr.arpa zone file for their IP's, which resulted in an immediate negative result on that reverse zone lookup. If the rest of akamai would get on the stick and do the same, the problem would be history.
akamai and spamcop.net (Score:2)
Re:akamai and spamcop.net (Score:2)
The Akamai issue did indeed affect www.spamcop.net, but I don't think your problems are related to that.
michaelmoore.com (Score:4, Funny)
Re:michaelmoore.com (Score:2)
That, my friend, takes a special kind of paranoia.
2.5 hours downtime (Score:2)
Akamai says it's a bug in the software, not DDoS (Score:5, Informative)
Haha (Score:2, Interesting)
Don't you see the irony? How much of the internet populace depends on Google for their searching needs?
I suspect the problem here, as there, is that there aren't many who can compete at a service level.
Here's what's happening (Score:2, Interesting)
Try:
http://a40.g.akamaitech.net/7/40/1601/1d/ i mages.sl ashdot.org/topics/topiclinux.gif
(it works with ANY URL)
Obviously, they noticed it, and tried to fix it. Their fix turned out to block valid customers (like Apple, as has been mentioned), so now they have rolled it back to the free-for-all setup.
They're probably working on a better fix right now.
Akamais distributed DNS & content solutions (Score:5, Informative)
This is nicely commented on in a recent story over at CFO [cfo.com] where it says "Broadly speaking, Akamai needs servers near the consumers of content..[] Akamai, on the other hand, has servers pretty much everywhere."
To trim the facts down a bit: Akamai has servers near by most users these days, and the distributed DNS gives you returning DNS to the closest contentserver. If I, who live in Norway, try to access fbi.gov from any computer from a ISP connected to the NIX (Norwegian Internet eXchange) I get a DNS response that leads me to Akamais servers in Oslo, Norway. I've tried this for some time, just to see what happens, with cnn.com, apple.com and fbi.gov. While on a trip to Sweden I tried this while connecting through a local DSL-provider and I got a response from a server located in Sweden, hence even the swedes have their own Akamai mirror these days.
The problems with a DDOS from someone in Norway would, if directed towards a domain or webpage and not an IP-address lead to downtime on that specific local mirror, not Akamais entire network. We can from this conclude that only such events as a major blackout in Akamais core network or like this time, DOS'ing their own network would take out their service.
Nothing to see here...move along. (Score:3, Informative)
Re:Nothing to see here...move along. (Score:4, Insightful)
Considering Akamai hosts the web sites for the like of the White House, FBI, and Dept of Homeland Security (among other gov't agencies), an Akamai outage is slightly more than "nothing to see here." Unless you are making a political statement, I guess.
Akamai was down from 8:00am to 9:15am (Score:3, Informative)
I spoke with Akamai support. They indicated that it was a far reaching problem, but I have not heard the reason yet.
The customer login to the admin portal was down as well. It was almost like someone dump the customer account database.
Akamai has a QOS commitment of 100% uptime based on the idea that not all of the 1,000's of servers could go down at the same time. But... There you go.
latest advisory (Score:4, Interesting)
Akamai is aware of a service interuption earlier today affecting content delivery.
We have identified the root cause and have implemented the fix. Issues retrieving content should be decreasing or resolved. Updates will continue to be posted on the Akamai Edge Control Management Center.
so there is something wrong with their cdn. so much for 100% availability. my guess, all the edge servers were ok but there may be a problem with their noc or software.
Remember when Akamai DoS'd Microsoft? (Score:2, Interesting)
All due to a router config bug introduced by Microsoft.. So it was really Microsoft DoSing themselves via Akamai.
And it would be unfair to blame the router config for more than a few hours of outage. The big problem was the complete and utter paralysis of management on the conference calls.
I don't think the details of that outage have been leaked much. It was quite a hoot talking to those involved during the outage. And it wasn't hard, given the duration.
It is my recoll
Explaination from Akamai (Score:5, Informative)
This degradation was the result of a bug within one of Akamai's backend content control management tools, which allows the expiration of content on the Akamai network. The degradation was not a result of any outside interference with Akamai's network (such as Denial of Service or hacking).
Upon identification of the bug, Akamai quickly took corrective action which returned customers to normal service levels. Akamai is currently putting measures in place to return the content management tool to its normal working order and is adding safeguards such that the issue will not occur in the future. In the meantime, Akamai customers are able to serve their content through the Akamai Network normally.
As part of Akamai's normal proactive customer communication policy, Akamai customers will be kept informed of the latest developments through the Akamai portal, the EdgeControl Management Center, https://control.akamai.com. Any further inquiries may be directed at Akamai Customer Care at 1-877-4-AKATEC.
It's probably a test (Score:2)
If you can bring down Akamai, you can bring down anyone.
Re:It's simple. (Score:4, Interesting)
Re:It's simple. (Score:2)
Re:i've always wondered... (Score:2, Informative)
Simple. Just like it looks.