Army DNS ROOT Server Down For 18+ Hours

An anonymous reader writes "The H-Root server, operated by the US Army Research Lab, spent 18 hours out of the last 48 being a void. Both the RIPE's DNSMON and the h.root-servers.org site show this. How, in this day and age of network engineering, can we even entertain one of the thirteen root servers being unavailable for so long? I mean, the US army doesn't even seem to make the effort to deploy more sites. Look at the other root operators who don't have the backing of the US government money machine. Many of them seem to be able to deploy redundant instances. Even the much-maligned ICANN seems to have managed deploying 11 sites. All these root operators that have only one site need a good swift kick, or maybe they should pass the responsibility to others who are more committed to ensuring the Internet's stability."

Was it the monitoring system?

[Antique Geekmeister]

I've seen numerous instances where the monitoring system, itself, was confused or detached. The results on a chart are then quite confusing, unless you know how to backfill the data in the chart.

Why, no, I've never been asked to do that for a 99.999% uptime SLA monitored site when some confused person in the offsite monitoring station put a bad IP address in /etc/hosts. No, no, no, couldn't happen.

Re:Lowest bidder

[Isao]

There are two main approaches to government contracting: Lowest Cost and Best Value. Contrary to popular belief, Lowest Cost is not always the one chosen, by a long shot. I also previously misunderstood "Close enough for government work." Turns out most "government work" has very specific requirements and specifications, or you don't get paid. If you see something different, please call Waste, Fraud & Abuse.

Non-story

[A beautiful mind]

You have to realise that the layout of the root dns server hierarchy is historical. It is composed of organizations that are vastly different now than they were 20 years ago. The H root server people don't seem to care about things very much and there are a couple of other root servers where the organizations operating them don't put too much effort into things.

Luckily, the internet doesn't really depend on them, as there are a couple of big organizations with heavy investment into making sure the root servers stay accessible all the time, like RIPE or Verisign. They operate thousands of physical machines at dozens of geographically distributed locations, all structured under one ip address, via anycast. This results in the situation where one logical root server outweights the other one in terms of physical boxes at least 100:1, if not more.

My last information about the Verisign operated root servers from a couple years ago for example is that they are ridiculously overprovisioned, operating well under 1% used capacity, even when subjected to a fairly large DDOS. As far as I know, the common dns servers all support rtt banding, so basically using a random list of dns servers for a given resource that fall below a threshold of latency, therefor they wouldn't really notice the H root being down.

It's just a drill: Cyber Storm III

[Xemu]

Could this simply be a part of the Cyber Storm III information warfare exercise?
http://www.military-technologies.net/2010/09/29/test-of-first-us-cyber-blitz-response-plan-begins/ [military-t...logies.net]

Re:Not the biggest problem out there.,,,

[gnieboer]

Agreed.

From the offending server's website: "BRL volunteered to host one of the original root servers ... to provide a root server for the MILNET in the event that MILNET had to be disconnected from the Internet."

The purpose of the G/H servers is not to support the greater good (that's a side benefit), but to ensure that the MILNET can function if the DoD cuts itself off from the rest of the internet.

And besides, If my math is correct, there are a total of 205 redundant root sites (http://www.root-servers.org/), so imagine going up asking for funding...
[IT Guy] "General, we need money to add another redundant root server site, if all the sites go down the internet collapses!"
[General] "That sounds bad! How many redundant sites are there now?"
[IT Guy] "Only 205"
[General]

Re:So the Internet worked as it should...

[Anonymous Coward]

Umm--having retired from the military, and having also been a networking professional for fifteen years in education and industry, I have less than a great respect for the products of the U.S. Army Signal School, who happens to operate that server. I was activated for service in Iraq, and watched a fellow captain, a graduate of that school, and someone with at least five years experience, insist that Ethernet Cat 5 had a maximum single link distance of 185 meters. And he designed his network around that premise. Which meant that we spent the next six months after they left cleaning up the mess and troubleshooting the bad links that SOMETIMES worked. Granted, this is not objective but it illustrates the point.

Re:Army Intelligence?

[sumdumass]

Actually, given the size and scope of the US military, you are right, 1 trillion dollars is about pocket change to most people.

I'm for increasing their budget more too. But I'm not sure that this outage wasn't planned. How better to test the ability to withstand a "cyber attack" then to lose your DNS servers and see if the your departments can fully function without them. This ability would greatly decrease the time needed to change to an alternative system if ever needed or more likely regroup resources and work around it. I'm not so sure that this wasn't just a readiness test of some sorts disguised as an accidental outage or something. It would make more sense to make it appear to be a problem server then actually gearing up to work around it that would create another potential target for any attacks.

Re:Why is it their problem?

[amorsen]

I know most are anycast. I still think DoD should give up their slot to someone else, especially since they have 2. There is no reason why any organisation should have two slots; the only reason for that is historical.

Re:There are 12 others - pick one.

[stephanruby]

Go ahead and rub your nose in it until you get over your "how DARE you claim incompetence within the Army" offense.

First, let me start by saying that the guy you replied to was rude, and I don't see why he needed to insult you to make his point. However...

What went wrong is that a server that's not supposed to ever go down went down.

Your argument seems circular. Your assumption is that this root server is never supposed to go down. In this physical world, that's a pretty huge assumption to make.

And no, saying that the server went down is no proof positive that it should never have happened. The fact is, there was redundancy and the redundancy kicked in as it was supposed to. Now we're saying the redundancy can be outside of root, or inside of root, it doesn't really matter. And you're saying that the redundancy has to be ***inside*** of root, there can be no other way.

Tell us, have you read something that gave you that idea? Me, I'm thinking that you probably read that recent Times (or Newsweek) article, if anything, I do agree that the article only seemed to romanticize and emphasize the importance of root servers, but I'd argue that the article was more a piece of flamboyant story-telling than an actual report on an actual technology. The truth about root servers is far less sexy than what the article did imply. The real truth is that if all the root dns servers went down at the same time, most of the internet and its dns would keep on working pretty nicely. We'd be running on old possibly slightly outdated cached dns information, but that wouldn't really matter -- it's only the end results that would matter anyway.