Software Update Shuts Down Nuclear Power Plant 355
Garabito writes "Hatch Nuclear Power Plant near Baxley, Georgia was forced into a 48-hour emergency shutdown when a computer on the plant's business network was rebooted after an engineer installed a software update. The Washington Post reports, 'The computer in question was used to monitor chemical and diagnostic data from one of the facility's primary control systems, and the software update was designed to synchronize data on both systems. According to a report filed with the Nuclear Regulatory Commission, when the updated computer rebooted, it reset the data on the control system, causing safety systems to errantly interpret the lack of data as a drop in water reservoirs that cool the plant's radioactive nuclear fuel rods. As a result, automated safety systems at the plant triggered a shutdown.' Personally, I don't think letting devices on a critical control system accept data values from the business network is a good idea."
Hmmm, threw an exception (Score:5, Insightful)
Fail-Safe (Score:5, Insightful)
Re:Lesson learned: (Score:3, Insightful)
"... when the updated computer rebooted, it reset the data on the control system, causing safety systems to errantly interpret the lack of data as a drop in water reservoirs that cool the plant's radioactive nuclear fuel rods. As a result, automated safety systems at the plant triggered a shutdown."
From that snippet alone, it stands to reason that _any_ reboot of the computer would have caused this reset in at the control system. Nor is this at all surprising; go reset any data collection system connected to controller software for any sort of industrial process and see if the controller doesn't receive spurious data.
To me this is an example of the automated system doing it's job. "Hark! I am a coolant reservoir monitor and I have reason to believe there may be a loss of coolant inventory. Time to trip the system."
Re:More like bad system design (Score:5, Insightful)
a) high chance of accidentally shutting down a reactor harmlessly
b) small chance of fucking up a nuclear reactor
you'll always go for (a), if your sane.
Re:Install Complete... (Score:2, Insightful)
This was Good (Score:4, Insightful)
Re:Fail-Safe (Score:5, Insightful)
big increases in your power bill! (Score:4, Insightful)
They have a perfectly adequate safety system that did exactly what it's supposed to do. It read confusing data and decided to shut the reactor down until a human came along and explained things satisfactorily. What's wrong with that? Aside from having the reactor offline for 48 hours, there was no other cost.
Re:Fail-Safe (Score:4, Insightful)
On, the contrary, shutting down because the system is shit sounds like a much better option than continuing to run despite the shittiness of the computer monitoring everything.
Of course, the ideal situation would be to have good computers that only get updated in scheduled, planned ways so that you don't have the issue at all. But shutting everything down when something is amiss is the only sensible response.
Working as intended (Score:3, Insightful)
Re::O (Score:5, Insightful)
Re:Wow that is so funny (Score:4, Insightful)
Re:No! (Score:1, Insightful)
Only the biz machine was updated. Why trouble? (Score:5, Insightful)
I have no problem with a computer on the process control subnet reporting information to a computer on the business subnet.
I have a BIG problem with a computer on the business subnet being able to modify and corrupt data in a computer on the process control subnet.
"I can't dump data to the business side" is a reason to make a log entry and maybe sound a minor alarm. It's not a reason to shut down the reactor (unless the data is needed for regulatory compliance and the process control side isn't able to buffer it until the business side is working correctly.)
But if a business subnet computer can tamper with something as critical as a process control machine's idea of the level of coolant in a reservoir, it rings my "design flaw" alarms.
Is it ONLY able to reset it to "empty" as poorly-designed part of a communication restart sequence? Or could it also make the process control machine think the level was nominal when it WAS empty?
IMHO this should be examined more closely. It may have exposed a dangerous flaw in the software design.
Security flaws don't care if they're exercised by mischance or malice. If nothing else, this is a way to Dos a nuclear plant through a breakin on the business side of the net.
This was not a "fail-safe" incident (Score:5, Insightful)
In the article they mention that the system wasn't designed for security (since it was meant to be internal) - but this isn't a security issue at all! Any sort of system that relies upon other systems should be designed to assume failure can and will occur in other systems - that is not to say that it needs to verify/evaluate incoming data to make sure it is "good", but rather that it can tell the difference between receiving data (such as current water levels) and receiving no data at all (system failure). Once it has that it can ideally automatically switch to a backup system, or do what it did here and enter a fail-safe state (the difference being that it does so while pointing out the actual problem and not a incorrectly perceived problem in a different part of the system).
Re:Wow that is so funny (Score:5, Insightful)
This was NOT a failure! (Score:2, Insightful)
It says the software is supposed to sync data between the control system and the business network. Obviously it has to be connected to both sides somehow. I'm not a power plant designer, but there's probably a good reason why people might need access to that data from the control system, and thus some kind of system acting as a safe bridge between the two rather than allowing unrestricted access from the business network.
The update f'd up and the control network went "Holy crap where did the cooling water go? Abort!" Everything worked like it was supposed to. The failure was caused by not testing the update in a lab environment before applying it to a live system.
It kinda worked then... (Score:4, Insightful)
That is definitely a glass half full, as opposed to empty.
Re:Wow that is so funny (Score:3, Insightful)
Re:just to shortcircuit the nuclear hysteria (Score:5, Insightful)
India's accelerated thorium idea is also very promising.
The major problem I see with US nuclear power is the assumption that it is a solved problem and almost zero has been spent on R&D for decades. The "new generation" of reactors from Westinghouse and others is little more than 1960's white elephants painted green.
Re::O (Score:5, Insightful)
Re:Wow that is so funny (Score:5, Insightful)
An area where that loosely controlled type of team work gets into trouble unless all coders treat data passed to their code, and from their code in the same uniform functional ways.
It also makes me wonder how the code will react to certain malicious software, should it get loose in the facility. If I were writing code to destroy a nuclear facility, it is how data is passed from one process to another that I would definitely attack as well as other vulnerable places.
It is sort of reassuring to have seen a failure result in a controlled shutdown rather than some other, more undesirable action.
damned if you do... (Score:4, Insightful)
Reboot. The tweaked configurations happen to go away. No one remembers which ones they were. The system is b0rked for a while.
I would hope that isn't the case for that system, but I have seen it happen before.
Re:Wow that is so funny (Score:5, Insightful)
Re:Only the biz machine was updated. Why trouble? (Score:1, Insightful)
Re:One begs the question (Score:3, Insightful)
Then again, maybe intelligent and well-educated people will just ignore people who aren't intelligent enough or who can't be bothered to learn how to properly communicate. The medium is the message, and a badly-formed message says to the recipient either "I don't care enough about talking to you to take the time to say it properly" or "the content of the message can't be that great if I can't be educated enough to learn to express it well enough".
I don't get out of the way for subgeniuses.
Time for someone to review the shutdown system (Score:3, Insightful)
Yes, the safety system kicking in is "a good thing".
Pulling data from another computer system for a safety related control system is not a bright idea (the weakest link problem).
Historically a safety control system in an Oil & Gas environment, all the inputs to the safety system are either hardwired or pulled from another safety system controller which has the appropriate level of redundancy (CPU boards and communication paths with communication watchdog timers).
Even transmitters in some circumstances can not be trusted hence the 2 out of 3 voting systems (take three transmitters measuring the same value and pick the middle of the three, if one of the transmitters fails high or low your choice will be the safe option).
Someone needs a serious think about where this plant is getting data for its safety shutdown system.
ZombieEngineer
Re:Wow that is so funny (Score:5, Insightful)
I've set nagios up to monitor my network, and any los of signal is considered CRITICAL, not just a warning, but critical... and I need to know then.
Re:This was not a "fail-safe" incident (Score:3, Insightful)
Re::O (Score:1, Insightful)
Re:One begs the question (Score:3, Insightful)
One of the great things about English is that one can phrase something a million different ways and still get the same meaning; banning the use of one phrase because it happens to also be the name of a logical fallacy is silly and pointless.
Re::O (Score:4, Insightful)
Well, the auditors seem to expect it... as do the vendors when we call for support - "Oh, you say foobar isn't working... well it looks like you're 15 revisions behind; why don't you just fix that and call me when you're done. Oh, your policies state you need to test and certify them? Well I guess I won't be hearing from you for a while, then."
--
.nosig
Re:Fail-Safe (Score:3, Insightful)
Systems without flaws will never exist, so we need to design systems that do reasonable things when they encounter flaws.
In this case, the flaw wasn't even caused by the machines, but instead was directly caused by the "fleshy" parts of the system, and the machines still managed to handle the problem safely.
Major system dependent on a minor system bad logic (Score:2, Insightful)
Re:Wow that is so funny (Score:5, Insightful)
Re:Wow that is so funny (Score:5, Insightful)
Re:Wow that is so funny (Score:5, Insightful)
I think you're missing the real point, which is that the central safety systems are being fed data from a 'business network'. What would happen if that computer had an issue that caused it to send the same data continuously even when the coolant level had really dropped? WHY are any safety systems receiving data from an insecure network?
It's bad enough that most reactors use regular PC's to do the data collection and reporting, given the security risks posed by such systems (especially if networked), but I never realized they would be so stupid as to feed data in the other direction like this!
Re:Fail-Safe (Score:3, Insightful)
Re:More like bad system design (Score:3, Insightful)
Probably not. Web servers are complex, and likely targets for attack. And the business people will end up doing endless cut and paste.
A better solution would be to accumulate the data that the businesspeople need on a single system on the control LAN. That system rsync's CSV files onto a system on the business LAN. No connections are initiated from the business LAN into the control LAN, and the data are more useful to MIS people on the business LAN.
Re:More like bad system design (Score:4, Insightful)
*However*, one of the more powerful ideas in configuring highly secure LANS is that the more-secure LAN is simply never allowed to accept connections from the less-secure LAN. It's also something that's really easy to firewall, your network becomes easier to audit, etc. If you're a security practitioner, it makes your life easier. You still have to worry about the sneaker-net, physical security, etc., but now you're more able to focus your resources on those areas. Once again, simplicity is better than complexity if you're really after security.
I don't know where you got the idea that I thought it was, "sooo impossile to have a perl script or whatever fetch the webpage and cut out the data you care about." It's easy. But pretty much nothing is as easy to extract data from as a CSV file, which you could process with nothing more than awk. That doesn't get you far with automating report generation, populating a database, or whatever else you intend to *do* with the data, but there are endless tools for those jobs--Perl included.
Also, in my experience, people want to mess with Web pages. They're more visual, and people tend to want to 'improve' them, meaning your Perl screen-scraper likely has to change as well. I see a lot less clamor for changing the data format in CSV files.
In the end, use what you need--XML, for all I care. Just *don't allow your less-secure LAN to initiate connections into your more-secure LAN*. That was the root cause of the failure described in TFA. It's one of many reasons the rule is so basic, though obviously not yet widely-enough followed. Ideally, hosts on a secure LAN communicate with *nothing* outside that LAN. You justify and document every[1] step away from that ideal, if for no other reason than that it plays hell with formal trust models, which can be important inputs into designing a thorough audit. I don't see how you justify accepting incoming traffic when there's an easy way to avoid it. In an audit, I'd be busting you for that Web server. Simple as that.
An approach like the one above is likely to make life easier for several internal groups, including office staff. And quite possibly the ultimate users--power consumers.
[1] I mean every, not most. For example, how do you handle time? I favor an NTP server on the secure LAN taking time inputs from the GPS cloud. I've never worked for an organization that had a spare atomic clock lying around, or I'd have used that, and eliminated one more external data flow.