Failed Software Upgrade Halts Transit Service 125
linuxwrangler writes "San Francisco Bay Area commuters awoke this morning to the news that BART, the major regional transit system which carries hundreds of thousands of daily riders, was entirely shut down due to a computer failure. Commuters stood stranded at stations and traffic backed up as residents took to the roads. The system has returned to service and BART says the outage resulted from a botched software upgrade."
Re:I Guess (Score:3, Interesting)
San Fran will turn into Detroit?
While from Reddit posted a day ago, it's so on topic to your post I had to post it your reply
http://www.reddit.com/r/explainlikeimfive/comments/1r6f8w/eli5_americans_what_exactly_happened_to_detroit_i/ [reddit.com]
Very good read if you want to know about Detroit
BART (Score:5, Interesting)
BART is run by the dumbest people on Earth. First off, it's takes a special kind of stupid to create a rail system that goes almost, but not quite all the way to the airport. 30 years later they extended to one of them but you still have to transfer to a bus for the last mile on another. Then you have to wonder what kind of idiot puts light carpet and cloth seating on public transport. 35 years later they start testing non-porous flooring/seating and maybe in another five years all of the trains will be switched over. Then, some bean counter got a bonus when they closed all the station bathrooms when 9/11 happened, ostensibly for security. Now a fifth of the escalators are out of service at any one time because they are clogged with human shit.
I also heard there was some sort of labor dispute.
Re:BART has drivers. (Score:1, Interesting)
Because there is no means in the "cockpit" to actually make the train go. There are three buttons in a BART rail car:
Open Doors
Go to next stop
Emergency Stop
Not even a "close doors" button - that is handled by door sensors and the computer when "Go to the next stop" is pressed.
Everything is automated. A chimpanzee could operate a BART train.
Re:Snapshots? (Score:2, Interesting)
No. Just no.
Have you ever actually tried this on a production system? I haven't (I'm not stupid enough to do that), but I've seen many others try. In almost every case, the resulting mess from "rolling back" a VM was greater then the mess of a botched software update to begin with. In one particular case, I witnessed a certain VM running some very expensive enterprise software totally hose itself and then proceed to blow away the majority of a database hosted on another VM after it was restored following a broken update. Despite their attempts to restore both VMs and bring them back in sync, they eventually determined that the data couldn't be trusted on either and the entire system had to be restored from backup. The downtime this cost them was greater then the downtime would have been had they simply called the vendor and said "your update broke our stuff, fix it" (they had the support contracts and the fix would have taken 10 minutes instead of 8 hours).
Another time I saw someone restore a VM that was running a network daemon for a cluster of hardware locks attached to one of the nodes (of course, this VM was locked to that particular node since it required passthrough access to the USB dongles). That was a good one- not only did none of the licenses get checked back into the network daemon (so they basically lost all the capacity they had in use at the time of restore), but the licensing software freaked out and shat itself when the time stamps coming off the hardware were suddenly in the future (as the clock had not yet been synchronized back to local time). It took those guys several days of pleading with the software vendor to send them new keys and get the licensing system sorted out and working again (snapshots were permanently disabled on that VM thereon after).
Now, it's an awesome feature to have for testing and development stuff- but for production, you should have procedures in place to deal with this kind of thing rather then reaching for the Big Red Button and nuking everything from orbit. I keep hearing about this kind of thing- "oh just restore the VM from snapshot in prod", and it makes me cringe every time I hear it. You don't restore a server from tape unless you absolutely have to. I fail to see why anyone thinks that restoring a VM from snapshot is any different- the only difference is that it takes seconds to complete, instead of hours.
Re:BART has drivers. (Score:5, Interesting)
You've almost certainly never ridden BART, much less seen the driver's cab. Why do I say this? Because there's a section of the BART system (the Oakland Wye, bane of commuters who want to get anywhere during rush hour) where drivers are instructed to go to manual control, limited to 25 MPH. It's the result of your vaunted "automated" system designed in the '60s never having worked properly in the past 50 years, and one of the contributing factors to a crash in 2009 (thankfully no one was seriously injured). There are many well-documented incidents of entire train sets disappearing from the computer system, as well as "ghost" trains randomly appearing.
Here is what an actual BART cab looks like:
http://i.imgur.com/IbYtYTa.jpg [imgur.com]
Re:I Guess (Score:2, Interesting)
Interesting that you bring up Haiti. They occupy the same island as the Dominican Republic; while Haiti has been a disaster for a very long time, the DR has always been totally different (just look at a satellite photo showing the deforestation on the Haitian side, while the Dominican side is lush and green). Now, if you go look at the people there (which you obviously haven't, because you're a dumb troll who lives in a trailer), you'll see that they're all black! The main difference between them is that the Haitians speak French, while the Dominicans speak Spanish. Also, many of those places in Africa that are fucked up are former French colonies. So maybe that's your common denominator there.