Storm Causes AT&T Outage Across Midwest

dstates writes "AT&T left users across several Midwestern states without cellular phone service yesterday. The outage apparently resulted from a power failure at a Michigan switching center and spread to affect level3 Internet communications. The powerful windstorm also left 400,000 users without electricity. Interestingly, except for a few reports in Chicago and Indianapolis papers, AT&T has managed to keep this out of the mainstream media. Widespread communication failures also followed Hurricane Ike in Texas earlier this year. With the increasing trend for users to drop landlines and rely only on cell phones, this is becoming an emergency preparedness issue." Yes this included me. Still does. At least my office still has power — maybe we'll just camp here tonight. :)

Roaming?

The problem isn't the cell phone network per-se, but rather the inability of these providers to peer with each other. AT&T may have been down, but what about T-Mobile, the other GSM provider in the United States? When a major failure like this occurs that locks out only some cell phone users in a given area, the problem is not technology but politics.

Why, given how critical cell phones are during an emergency, this is allowed to continue is beyond me. Congress seems to care more about protecting corporate profits and reputation than providing a robust cellular network for its citizens. Hey, homeland security, are you listening? Fix this.

Re:Roaming?

AT&T/Comcast/AT&T burned a LOT of bridges when they became Cingular. I used to be able to roam almost anywhere, and if a roaming tower was the best strength I connected to that. After cingular got involved they pissed off a lot of other carriers. Now my phone refuses to talk to any nearby roaming towers but tries to connect to that single AT&T tower about 12 miles away that gives me barely any signal. I have to hack my phones to disable this configuration to get decent cellphone service out of them.

Re:Roaming?

AT&T/Comcast/AT&T burned a LOT of bridges when they became Cingular. I used to be able to roam almost anywhere, and if a roaming tower was the best strength I connected to that. After cingular got involved they pissed off a lot of other carriers. Now my phone refuses to talk to any nearby roaming towers but tries to connect to that single AT&T tower about 12 miles away that gives me barely any signal

I don't think that really has to do with burning bridges. It has to do with AT&T not wanting to pay money for you to roam. It costs them money every single minute that you are using another carriers network.

I used to have a T-Mobile phone (had to ditch them for Verizon when I moved in with the GF -- no signal at her house) and they did the same thing. They would disable roaming on AT&T/Cingular in areas where they had coverage. Even if you were in a zone with no T-Mobile service you couldn't hop onto Cingular. To be able to roam on Cingular you had to drive out of the county where T-Mobile had native coverage -- then you'd be able to connect to and use the Cingular network. If you were within the county where they had native service but happened to be in a dead zone you were SOL -- roaming wasn't allowed.

I have to hack my phones to disable this configuration to get decent cellphone service out of them.

I'm surprised that worked. With GSM your home network decides whether or not you will be allowed to connect to that roaming partner based on the location area code [wikipedia.org]. If that LAC indicates an area where they have native service they probably won't let you connect to the roaming partner. If it indicates an area where they don't have native service then you stand a better chance of being allowed to use that roaming partner.

Re:Roaming?

Nokia phones can get set to have a preferred tower ID. I get into them and set the preferred tower id to a roaming tower that gives full strength at my house. I used to have a blackjack, but got rid of it because it cant do that.

the preferred tower setting overrides it somehow and it works. I've been doing this for the past 4 years and they still have not canceled my account.

Re:Roaming?

There are two things at play here: The ability to control network selection on the handset, and what the cellular networks allow you to do.

On GSM devices, there's a flag on the SIM card that tells your phone whether or not to show you the "manual network selection" menu. With AT&T, their SIM cards are configured to disable this menu when you're in the US, but enable it when outside the US.

However, on many devices, you can force them to ignore the SIM setting and have manual network selection enabled always. With Motorola phones, you can do it via SEEM editing, and there's obviously a way to do it on Symbian as well. This is what Lumpy is doing.

Shakrai, what you're talking about is the actual roaming agreements between providers. This also affects network selection. Providers can specify which phones are allowed on their network. Legally, all phones must be allowed to associate with all GSM towers to provide 911 capability, but they can be limited to just that. For example, when I only have T-Mobile coverage, my AT&T BlackBerry shows "SOS" where the signal bars usually are.

AT&T and T-Mobile have been doing this for a while. They had roaming agreements where specific cell sites would allow the other operator's phones in areas where the other operator's coverage was spotty.

So when Lumpy uses manual network selection on his phone, which he had to enable by modifying something, the rules that the operators set forth on their networks still apply. If he tries to associate with a T-Mobile tower in certain areas, he will most likely get locked out, but in other areas he may not.

Re:Roaming?

Does this kind of situation explain why we in North America are a little bit "behind" Europeans and Asians when it comes to the "cell phone" and exploiting its maximum potential?

No. All this situation relates to is which networks you are allowed to roam on. Europe works much the same way. If you have Deutsch Telekom phone you probably aren't going to be able to connect to and use Vodafone's network in an area where DT has native coverage. Why should they pay Vodafone a per minute rate for you to roam when they have a native network in the area where you happen to be?

I hear the "cell phone" elsewhere in the world is capable of so much more than simply making calls and texting, which are the bulk of what we use cell phones for over here.

*shrug*, we have a smartphone market. I haven't been in Europe for a few years but the last time I was over there I was actually disappointed with the state of 3G. It's probably better now but when I was in Europe their 3G offerings weren't any better than ours.

Japan and South Korea are different animals -- they are a full generation ahead of Europe and the US and people use their phones for all manner of things over there. That's as much cultural as anything -- how many Americans don't even see the point of SMS let alone internet browsing? Most people I know with phones regard them as phones and don't care about the extra features.

Re:Roaming?

They are trying to fix it [datacenterknowledge.com], much the same way that Sarbanes-Oxley fixes accounting problems. Communications providers are required to keep an 8 hour power backup on all sites and 24+ backup on 'important' sites like switching centers or something along those lines. The idea is that storms like those that hit New Orleans would not cause the problems that they did. This storm is exactly the thing this measure by the FCC is supposed to fix... HAHAHAHAHAHAHAHAHAHAHA

Legislators (Goodbye FCC) do NOT know how to run businesses. Some perhaps, but on the whole they are terrible business advisers and this legislation only proves it in the aftermath of this storm. I hold a harsh opinion of this situation because AT&T should have had backups in place to handle this situation. All Communications providers deal with such things and AT&T has enough history to know what to do... shame on them.

Re:Roaming?

BTW, besides those rescuers you see on television after a storm, remember that there are ordinary heroes sitting in a truck babysitting a generator, or climbing poles, or renting a snowcat to get to the tower site so that your phone will work when you want it to. They are suffering the elements, usually alone, and bear the responsibilities of ensuring your comfort with little or no thanks. They are up all night watching the weather, planning and preparing to go babysit that tower(s) so your phone or pager will work. They work almost tirelessly for the simple comfort of knowing that when the shit hit the fan, the equipment they are responsible for did not fail. They are little appreciated on a normal day, and unseen during the emergency, but it is their dedication that you count on and not the system or the company.

Re:Roaming?

I wish I had mod points for you. I used to work for a WISP and had the "privilege" of climbing towers in god-awful weather after storms to restore service to our customers. It's scary enough to climb the things in normal weather. Doing it Upstate New York in January after a storm has coated the tower with ice is absolutely horrible.

The worst part was that was my first job and I only got paid $10/hr to scale the friggen things. Looking back on it I wasn't making nearly enough money to be risking my life like that. At least our customers were appreciative though. That helped more than you can imagine.

Thats all a nice homespun story and all but..

.. if "the company" didn't pay them, they wouldn't get out of bed. Lets not pretend they're up those poles out of the kindness of their hearts.

Re:Roaming?

    Several years ago, I was driving across the country from Florida to California. I was only a couple days behind a hurricane that had made a mess. No, it was way before Katrina.

    Every hotel along I-10 for hundreds of miles had utility trucks from all over the place. I usually drive at night to avoid traffic, and I know these guys need to at least get a little bit of sleep. I know the utility guys get sent out from all over the place to fix problems induced by bad storms. It never really hit me that it wasn't just a few hundred sent, but an entire army. I would stop at hotels, and they would tell me that they were completely booked up because of the utility guys. Not just their hotel, but every hotel in the city. "Try the next town, 50 miles down the road." It's not like they were hogging the rooms, they were stuffed in. One guy per bed, and a guy or two on the floor. This is our civilian army, that keeps things working after a disaster.

    Unfortunately, all the general public knows is "they don't have my power on yet.", even though it is an army working hard to repair everything. People like instant gratification. They think it should be a light switch fix (hit the switch, it works).

    It's a job I wouldn't want. High voltages, dangerous weather, and long hours.

Re:Roaming?

We're staying with relatives on a farm south of Chicago. All the AT&T phones lost their ability to dial out, yet they could receive incoming calls. Our T-Mobile phones were fine.

You're right, this is a case of corporate power struggles trumping customer service, as any of these phones should automatically fail over to another GSM network. In an emergency, dialing out is essential.

emergency calls do whatever it takes

Put the keylock on and dial 112, 911 or 999 on any mobile, watch as it punches straight through the keylock and will dial. It will place the call over any mobile network it can reach, roaming or otherwise. Now try it with no credit on a pay as you go SIM. still works. Now take the SIM out. Still works.

Re:Roaming?

I would love to see the part of the Constitution that says "Congress shall provide a robust cellular network for its citizens."

Don't worry -- someone will come along sooner or later and twist the interstate commerce clause until they manage to come up with that interpretation. Remember, all parts of the Constitution should be interpreted as liberally as possible, except the 2nd amendment of course. That clearly provides for a collective right and doesn't apply to the states.....

If All You Have Is a Cell Phone...

then you're out of luck. Most landline phones have independent power and will work in an emergency. That's one reason I always have a landline.

Will AT&T repay me for the days my service was

First Question of any contract: Any contract in which any of the party is unable to fulfill the terms of the contract will be is liable to either NOT collect remuneration of the said contract for that quantity, OR reimburse the amount for that period.
Will AT&T send me a check for the days my service was out?
Because last i checked, weather is not a cause for NOT fulfilling a contract. For the same reason i cannot claim weather for not paying my mortgage.
Question is, will AT&T refund the amount for the period of outage.
If not a class-action suit can be filed in coordination with other users.

Re:Will AT&T repay me for the days my service

Most contracts include a "Force Majure" clause that absolves the service provider in the event of a natural disaster.

Re:Will AT&T repay me for the days my service

The inability for AT&T's datacenter in Michigan to have power backups that can last more than a day should hardly be considered a natural disaster.

I'd love to see something happen in terms of getting money back, but somehow I doubt most subscribers care enough to push for it.

Making AT&T pay...

A Credit is NOT the same as a check.
Reimbursement needs to be in the same terms of original contract.
You don't give them CREDIT for providing you service. They take your money: period.
Similarly, ask for a check or money order sent to your address. If they refuse, sue them in small claims court for violating a contract.
Why should you accept CREDIT when they don't accept yours?
A Credit is just an interest-free loan to these guys.
Refuse to accept unilateral changes to your contract and demand the repayment from them in check or cash.
Better yet, do the following:
1) Send a registered letter to their local office demanding financial compensation for the days not in service and quote the contract. Give them 7 calendar days to pay (not working days).
2) Wait for 7 days and when they don't respond, file a complaint with small claims court. Ask the clerk to send a notice to their local exchange in telephone directory with another 7 calendar days to respond.
3) They still wont respond for $5 or $10.
4) Enter a default judgment. Tell the judge they insulted him/her by deeming this court to be beneath their status (appeals to vanity and not logic). The judge will be furious on their own to consider MAXIMUM enforcement.
5) Get the judge to declare that AT&T is in default of a debt to you. This is very important. Because this disassociates the original contract from the debt. In other words, in the eyes of law, AT&T is indebted to you for the said amount as if they borrowed money from you (in a way true).
5) Now the judge will ask what do you want to do: Request that the judge allows you to seize their assets for payment of this debt. Get a bailiff order.
6) Call your close friends/relatives and state there (from a pay phone or get somebody else to call) to gather at the local AT&T office for a fire sale.
7) Get the Sheriff and a deputy. Go to the AT&T office and paste the order to their door stating their assets are being seized for discharging a debt. Throw every employee out. Ask any protesters to be considered as disobeying a judge's order.
8) Ask the sheriff to seize their equipment and conduct an auction at the doors with Sheriff and his officers as guards.
9) Your friends and relatives should have arrived by now. Make them bid 10 cents or 20 cents for iPhones, telephones, routers, computers, etc., and sell it to them at that price.
10) Finish the sale within 15 mins enough to collct your debt. The longer you delay, the exponential the probability that some retarded employee will call a AT&T lawyer who gets the judgment suspended.
11) Now you have a nice collection of worthy items for the $10 you were owed by AT&T. Plus AT&T credit is screwed because you had a lien and auction of their property.

If you had owed 30 cents on a mortgage that was overdue, ANY bank would do the same to you.

It is easy to keep that out of the news

They are major advertisers and can pretty much control what is reported about them most of the time. All they had to do was make a few calls... oh wait...

Not a conspiracy

AT&T has managed to keep this out of the mainstream media

I'm in Arizona, and I saw that AT&T service was down in the midwest from multiple sources, before I finished my first cup of coffee. If there's been any lack of information reported about this, my guess is that's because the press is more concerned about hundreds of thousands who are without power in below freezing conditions, rather than a few people who can't make phone calls.

A major reason not to cancel POTS service

You cell phone users just don't get it, do you?
One of the major reasons why they want you away from your landline service is that they don't have to comply with the state or fed rules, tariffs, and laws forcing them to maintain POTS service even through the worst of weather.

Lose a cell tower? Fine, two weeks to get it back up.
Lose a phone line, depending on your state, from 12 hours to a working week in compliance with PUC regulations (in Texas it's 5 days, then Austin gets froggy).
Business landline goes down, 3-5 hours, as fast as a tech can get on site after getting pulled from a lower priority job.
Business cell service breaks, ok, duh, you get the point now?

What is $15-$25/mo for basic POTS service that is there when you need it in comparison to $60+ cell service that is still not as dependable as we like it to be?
I'll keep my POTS service and call forward from it to my cell, keeps the telemarketers at bay.

Ice storms and winter weather are nothing new

And neither are technology problems.

In 1998 Maine suffered the worst ice storm in decades. Power, telephone, and cellular service were affected. Yes, cable TV also.

In Gray, Maine, I was without power for 11 days. My sister in Searsport was without for 17 days, 2 of which were unnecessary - her house was about a quarter-mile in the woods, and the crews missed her line. She had power restored a few hours after calling in and reporting she was still out, and could see lights on at neighbors' houses. Darn.

Among the events that would inform the Midwest utilities:

1. Bangor Hydro-Electric, serving North-Central and Downeast Maine reported virtually 100% loss of transmission lines and 100% of customers affected. Central Maine Power reported most customers affected north of Portland, and most transmission lines down. Both utilities reported to major customers that restoration would take weeks, and they pretty much beat thet estimate. Not bad for rebuilding either 70% or 100% of their transmission network. BHE in particular had to replace completely many miles of hi-v transmission line, with poles snapped off. Availability of basic equipment like insulators became the limiting factor. In light of this, customers such as Verizon and cell carriers were told they were genuinely SOL.

2. The Verizon maintenance supervisor for the state had just relocated from Cape Cod, where he survived a similar event a year earlier. He immediately commandeered all generators, battery packs for SLCs etc, and emergency equipment from Mass, lower NY state, and beyond. Upstate NY was also affected and could spare nothing. His actions permitted his team to keep swapping the batteries out of SLCs, recharging them, and swapping to keep basic phone service running. He also asked for and got fuel from the Maine National Guard to keep the trucks and generators running. Most gas stations were down for lack of power.

3. As is the nature of winter storms, power lines suffer the most because they are highest on the poles. Telephone is next, and cable TV is usually lowest and suffers the least. Cable companies didn't bother much for restoration, since TV is the luxury you give up when the generator needs more gas than you have. Thankfully, this also meant most telephone service survived, and all they had to do was keep their gensets running. 'That's All'... It was a massive effort.

4. Cell service then was TDMA and CDMA, and NAMPS. It was good, despite the problems of the carriers having to do their own bucket-brigade battery swapping. They did terrically.

5. From my observations, quick action by carriers to put plans into action, clever thinking, and looking beyond the usual boundaries of support saved the day.

6. And one saving grace - the NBC affiliate in Portand broadcasts on Channel 6. Audio was available on most FM radios, way down on the band. When HD kicks in, this will be lost. No replacement I see.

It appears that AT&T is caught here with a central switch/datacenter that is stranded. We'll dissect their planning, no doubt, but ultimately they needed to plan for a week of power failure. I know that sounds preposterous, but my hospital clients at the time were even parking water trucks in the lot in case power outages resulted in public water supplies failing. Diesel tankers also came in. One hospital had backup privileges with a sister facility in Pennsylvania, and we would have transferred back-office processing there and flown/driven key personnel for a week to keep paychecks, billing, and patient care data current. Fortunately, my old stomping grounds were no longer my business. That hospital was out for 5 days, and ended up with a National Guard generator on site. The Guardsmen went without power for their armory to do that, sleeping in trucks and tents. Fortunately, it was not that cold for January. If it was 10 degrees colder, a lot of people would have died, never ready for that sort of trouble.

I escaped to friends in New Hampshire. Yes, I'm a wuss.

AT&T should own up to bad planning, despite the unusual weather. Redundancy is crucial, expensive, and worth it.

Not conspiracy, but AT&T has not been forthcom

I would argue that the OP has a point. I am a doctor, was on call (I'm not kidding), and missed several important messages due to my cellphone going out (my blackberry just silently stopped receiving all work mail, all internet functions went dead, full 3G signal but "tunnel failed.") Granted, there is a lot of redundancy in communications, so my pager later started going off with a lot of people saying "where are you???", and I then called them on a landline.

I thought it was my phone, rebooted 3 times, and only today did I find out that it was a national outage (saw here, confirmed all over the net.) I think AT&T should just have sent a free txt saying "We are having problems" or made an large scale announcement via voicemail, which would have helped me (and others) plan. I was about to get a replacement phone from a friend and plug my SIM into it.

The point is we start to rely on these devices, and blackberries, for better or worse, are used for very important things in business, health care, and otherwise.

Re:Not conspiracy, but AT&T has not been forth

If AT&T's service was down, how would they send you a text or voicemail?