Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
Software Bug The Military

Software Bug in F-35 Radar Causes Mid-Flight System Reboot 153

Reader Lisandro writes: The F-35 Fighter jet can't seem to catch a break. An advanced AN/APG-81 AESA F35 radar system has been found riddled with a software bug that causes it to degrade and stop working. The solution? Rebooting the system while in the air.

Major General Jeffrey Harrigian, director of the Air Force's F-35 integration office at the Pentagon, was quoted as saying "radar stability - the radar's ability to stay up and running. [...] What would happen is they'd get a signal that says either a radar degrade or a radar fail - "something that would force us to restart the radar." The issue was spotted in late 2015, and thankfully, it was caught during the testing period. The software version "3i" is affected. An update aimed to resolve the bug is expected to be delivered to the US Air Force by the end of March.
This discussion has been archived. No new comments can be posted.

Software Bug in F-35 Radar Causes Mid-Flight System Reboot

Comments Filter:
  • by Z00L00K ( 682162 ) on Friday March 11, 2016 @03:44AM (#51676173) Homepage Journal

    It looks to me that it's a classic memory leak.

    It should have been caught in testing, but of course someone wanted to save money and then it's testing that gets shaved first.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      A serious question here.

      Has anyone experienced memory leaks caused by race conditions and how (un-)common is it? Those would be much harder to catch and might also only appear in real world scenarios.

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        If you're using the standard model of whoever allocates the memory is responsible for freeing it, then it should never happen. Now I understand that sometimes you need to break that paradigm, but a programmer who does this should be very aware of this and careful when they do it. And in any case, again, pointers should always be initialized to NULL, and if reusing a pointer you should check if it's null before assigning to it, and after freeing, it should always be nulled. Even better, use an autopointer

      • by Quince alPillan ( 677281 ) on Friday March 11, 2016 @10:32AM (#51677169)

        Extremely common, actually. It's one of the major pitfalls and difficulties of doing multi-threaded programming and one of the hardest things for programmers new to multi-threaded design to learn how to solve. It can also be extremely difficult to debug, even for experienced programmers.

        Improper garbage collection is another extremely common bug that becomes harder to find and debug with multi-threaded programming, and that can also lead to memory leaks.

        There are time tested techniques to mitigate these issues and strategies to find and squash the bugs, but as you said, they can be extremely hard to reproduce while testing.

    • by WarJolt ( 990309 )

      I find it sad that we still write software for mission critical systems that isn't formally verifiable. You can build a software system where you can prove it won't leak. It amazes me developers forget how to write a proof after they get their degree. This isn't just for fighter jets though. If your cloud goes down your company can lose billions. It makes sense to write software you can formally prove won't fail because the stakes are high.

      • by DamonHD ( 794830 ) <d@hd.org> on Friday March 11, 2016 @04:29AM (#51676245) Homepage

        Have you actually every tried writing a formal proof of correctness for any algorithm at all, let alone a non-trial one dependent on external subsystems and with huge amounts of state?

        Yes, I have tried, and raised funding, and managed in fact to run one layer of our formal modelling language in real time (slowly). But we decided that the proof languages (Z and ML, with a sprinkling of CCS) weren't up to the task, and nor were we.

        Rgds

        Damon

        • by dwsobw ( 2723483 )
          It is possible if you have the resources, have a look at seL4 [sel4.systems] a verified microkernel.
      • by bloodhawk ( 813939 ) on Friday March 11, 2016 @04:42AM (#51676267)
        a formal proof for such a complex system is hideously, insanely ridiculously expensive. proper testing is sufficient. Yes a company could lose billions if their systems went offline as opposed to definitely spending billions to write verifiable software that will be out of date by the time they release it.
        • by Anonymous Coward

          It's possible to write complex systems using modular design: do one thing and do it well.

          Concurrency and latency need be ensured (and debugged) via the interface layer (a dedicated module for interfacing between modules).

          This is why you have a single car area network bus, and why both Ethernet and the Internet have taken off so well. They follow the Unix design philosophy, unlike the F35... otherwise there would be a simple way to restart the radar module without having global side effects.

          • by Waffle Iron ( 339739 ) on Friday March 11, 2016 @11:05AM (#51677425)

            It's possible to write complex systems using modular design: do one thing and do it well.

            And you'll soon discover your simple modules start interacting in ways that you did not anticipate or understand. There are also unlikely to be any tools available to analyze how your set of modules work as a whole.

            Every nontrivial system has emergent behavior. You can't eliminate complexity with hand waving.

            • You can't eliminate complexity with hand waving.

              That's probably why Chuck Moore eliminates complexity by keeping things simple.

            • You can't eliminate complexity with hand waving.

              The purpose of modularization isn't to eliminate complexity but to compartmentalize it. This allows for design and problem resolution by mere mortals.

              • The ability to "compartmentalize" complexity is limited.

                As I said, the complexity still exists in the way the compartments themselves interact. Mere mortals usually still can't deal with the problems.

              • modularization doesn't eliminate complexity though, it removes one complexity and replaces it with a whole other set of complexities and problems.
        • ... as opposed to definitely spending billions to write verifiable software that will be out of date by the time they release it.

          Having their tech be out of date is usually preferable to have it being unreliable, provided that bug patches, etc..., are backported as necessary.

          Spending billions to get the software right on the F-35 would make sense. We are going to make thousands of the things and use them for generations. The software is also likely to be stolen at some point, so it is better for us to take the time and make it as perfect as possible so that even if stolen, other nations do not have an easy time finding and exploit

        • I thought that C++ would make it easier to contain subsystems as implementations of a collection of classes. Beat the shit out of a class and then test the next class. Any patches to the class would wait, or require a separate class to include the base class.

          Of course, they probably wrote the whole think in assembly.

      • by gtall ( 79522 ) on Friday March 11, 2016 @07:31AM (#51676545)

        As others below have mentioned, it is very difficult to formally verify large complex systems. However, it is made even more complex in that there aren't enough research results to cover such a system in its complexity. Also, computer scientists tend to think the world revolves around their code, so if they get that correct, then the system will run correctly. The real world isn't like that, and it is not all captured in software, much of the system is hardware. Trying to capture the correct interaction between hardware and software is very, very hard...and it isn't clear that even if you could that you could verify the result before the universe dies.

      • Mission critical is a much lower burden than safety critical and even safety critical isn't developed with formal methods. Safety critical is developed with a more stringent process but nothing like that. The F-35 has something like 8 or 9 million SLOC. If it were required to develop everything on the F-35 with mathematical proof of correctness you wouldn't have 10% of the capability it has today.
    • by SeaFox ( 739806 ) on Friday March 11, 2016 @04:29AM (#51676247)

      It should have been caught in testing, but of course someone wanted to save money and then it's testing that gets shaved first.

      You mean like it says in TFS?

      The issue was spotted in late 2015, and thankfully, it was caught during the testing period.

      • by stevew ( 4845 ) on Friday March 11, 2016 @08:07AM (#51676645) Journal

        The other detail missing here is that 3i isn't fielded yet. That is something like Block 2F which is only installed in the Marine Corp unit right now. Oh by the way - There is only one unit of Marine F-35s that are "on-duty" right now - the entire rest of the fleet is under test/development.

      • by Z00L00K ( 682162 )

        Well, it should be caught in an early phase of testing, possibly of the unit itself long before it even gets into the full system.

        • In a lot of software development regimes, unit tests are one-shot affairs. If a bug requires continuous operation to manifest, you won't see it until at least subsystem integration, and if the program is pressed for schedule integration burn-in will be shortened and you won't see the bug (if you're unlucky) until the full-up integrated checkout before acceptance testing... the last bastion before it goes to deployment.

          So, on one hand, congratulations to the program on capturing it before fielding, but OTOH,

    • It should have been caught in testing, but of course someone wanted to save money and then it's testing that gets shaved first.

      Too often this is true. Not only do you, as a tester, have to fight for a case of testing with suppliers, you often have to fight off internal forces too. Suppliers hate you for demonizing their product and QA and your bosses hate you for subsequent late deliveries. Everybody hates testers.

      One thing I've found to be a somewhat working solution is to present cost saving estimates directly related to successes in software testing to our management. They speak money and that's what you should speak to them too

    • but of course someone wanted to save money

      No one wants to save money. Defence contracts are "cost plus", so cost overruns lead to higher profits for Lockheed. Congress is happy, because more money goes to the subcontractors in their districts. The Air Force is happy with the overrun, because a bigger project means a promotion for the officer that manages it. The public is also happy, because higher defense spending makes us safer.

  • by Freshly Exhumed ( 105597 ) on Friday March 11, 2016 @03:49AM (#51676183) Homepage

    Oh come on, who here hasn't had to reboot during air to air combat?

    • by Z00L00K ( 682162 )

      It's one thing during flight simulation where a BSOD is just an annoyance, during combat the effect can be a tad more annoying for the pilot.

    • by Feral Nerd ( 3929873 ) on Friday March 11, 2016 @04:40AM (#51676265)

      Oh come on, who here hasn't had to reboot during air to air combat?

      ... a problem that is aggravated by system's insisting on the installation of innumerable update packages on every reboot.

      MISSILE LAUNCH DETECTED!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE APPROACH WARNING!!!
      Installing radar software update 3 of 68....
      MISSILE IMPACT IMMINENT, EJECT! EJECT!
      Installing radar software update 4 of 68....
      For this update you need Microsoft Silverlight, install Silverlight [Y/N]:

      • by sumdumass ( 711423 ) on Friday March 11, 2016 @05:13AM (#51676317) Journal

        You forgot we recommend you upgrade to Windows 10.

        • by rtb61 ( 674572 )

          Now isn't that exactly how M$ makes it money, charging for upgrades for ever. It looks like the F35 is the perfect military industrial complex aircraft, forever requiring upgrades and bug fixes and not just from the US government from every government required to fork over 2% of GDP tribute payment to the US military industrial complex. After all they don't really need high quality weapons, just good enough to attack the terrorists they create. Russian and China as just the bogey men to drive tribute paymen

          • by gtall ( 79522 )

            Yep, Russia meddled and took part of Georgia because the evil U.S. was there. And they decided to steal part of the Ukraine for the same reason. And those islands in the S. China Sea, why Vietnam and the other nations to which they are closer are only doing the bidding of the U.S. hence the need for China to militarize them. N. Korea, yep, those naughty Americans are preventing the Norks from slaughtering the Sorks as they deserve. And those Muslim fanatics, why they'd be fluffy bunny rabbits were it not f

            • by Noble713 ( 3516573 ) on Friday March 11, 2016 @10:20AM (#51677099)
              I know you are being somewhat facetious, but I suspect these positions aren't too far from your actual beliefs, therefore....

              Yep, Russia meddled and took part of Georgia because the evil U.S. was there. And they decided to steal part of the Ukraine for the same reason.

              Because the US State Department spending $5 billion to "influence" the political situation in a country directly on Russia's border couldn't *possibly* provoke a response, right? Crimea is one of Russia's few warm-water ports and an essential link to the Mediterranean. How do you think the US would respond to political instability in Panama, especially if it was caused by another major world power? Oh wait, we already know: https://en.wikipedia.org/wiki/... [wikipedia.org]

              And those islands in the S. China Sea, why Vietnam and the other nations to which they are closer are only doing the bidding of the U.S. hence the need for China to militarize them.

              Something like 80% of the sea traffic going through the SCS is either to or from China. China, which is ~20% of humanity, compared to ~2% combined for Vietnam and the Philippines. Do the Needs of the Many and the Greater Good not apply? And US think tanks have written extensively about strangling raw material imports to China in the event of a conflict. Yeah, no way the Chinese might have a rational self-interest in securing the lifeline to their economy in their own backyard.

              . And Assad of Syria, we just know he was playing secret footsie with Americans before he decided to slaughter his people and chase a few million out of the country.

              Don't you think Assad would rather have a few million additional taxpayers contributing to his economy, even if significant portions of them are the unhappy Sunni majority? He certainly seemed to be getting along fine in 2010. Funny how the provision of funding, foreign fighters, and weapons from Saudi Arabia, Qatar, and Turkey (all Sunni states) roughly coincides with Assad's refusal to allow the planned Qatari/Saudi pipelines across his country. Pipelines to Europe that would undercut his patron Russia's economic interests.

              On a related note, do you express as much disgust at the suppression of popular dissent in Bahrain, or is that ok because King Hamad of Bahrain "is our man"?

          • by Aqualung812 ( 959532 ) on Friday March 11, 2016 @09:01AM (#51676785)

            It looks like the F35 is the perfect military industrial complex aircraft, forever requiring upgrades and bug fixes and not just from the US government from every government required to fork over 2% of GDP tribute payment to the US military industrial complex.

            I'm not a fan of the F35 program, but this is still done in meatspace with aircraft as old as the B52.

            The aircraft manufacturers are constantly pushing new hardware updates, new specs on how each bolt should be tightened, etc. Those don't come for free.

            It is even possible that ongoing costs will be lower with something software-based rather than hardware. It turns out that way often.

            I don't expect it to get cheaper, but if it doesn't it is in spite of software updates, not because of them.

        • Comment removed based on user account deletion
        • You forgot we recommend you upgrade to Windows 10.

          Hehe.... that would have been cool but I mostly regret that last line, it should have been:

          MISSILE APPROACH WARNING!!!
          Installing radar software update 3 of 68....
          MISSILE IMPACT IMMINENT, EJECT! EJECT!
          Installing radar software update 4 of 68....
          Currently updating ejection seat firmware, please try again later.

      • Maybe the F-35 had to update the Flash plugin [youtube.com].
    • Re: (Score:2, Funny)

      by Anonymous Coward

      thank you for calling ITI Advanced Combat Systems

      please press 1 if you are in a combat situation other wise please hold for the next technician
      beep
      please in put your mission number
      beep boop beep booop beeep boop beep boop beep beeep boop boop beep boop beep
      mission verification complete we will now transfer you to a support technician
      hold music

      thank you for calling ITI may name is nahmeed how may i help you

      look budy my radar is froze up and im dodging a mig 35 at the moment how about you fix this thing so

      • by KGIII ( 973947 )

        Hmm... I do not know where but somewhere in there should have been, "Please hold while I do the needful."

    • by Coisiche ( 2000870 ) on Friday March 11, 2016 @06:18AM (#51676417)

      My first thought was alternate movie dialog.

      Maverick: Talk to me Goose... Where's the bogey?

      Goose: Uh... hang on a moment Maverick, we're just going through a reboot... any... minute... now...

    • Oh come on, who here hasn't had to reboot during air to air combat?

      I've had a LAN game of Descent ruined by Windows daylight savings time, does that count?

    • by dywolf ( 2673597 ) on Friday March 11, 2016 @09:33AM (#51676863)

      actually "rebooting", ie, flipping the power switch or circuit breaker, isn't at all uncommon on avionics equipment on military aircraft.
      we aren't talking about typical computers that go through a boot process anyway. this is ruggedized equipment that largely lacks any thing resembling an operating system or RAM or much else a typical Slashdot reader would be familiar with.

      anyone who's spent any time working on military aircraft as a maintainer, particularly the avionics systems, knows that inflight glitches are not at all infrequent. and when they pop up on the Master Caution* or elsewhere, often the first corrective action the pilot takes is to power cycle the specific piece of equipment. most every system is on its own breaker, and pilots are trained in what can and what cannot be power cycled in flight. the majority of the time, that's enough to fix the glitch.

      and typically the first thing that happens when the pilot returns is a rep from each of the main work shops (avionics, flightline, airframes, ordinance, life support) meets him as he is exiting his aircraft, in order to ask if any gripes came up during the flight. this way they can get a jump on it before the pilot even gets back to the maintenance control to write the maintenance order describing the glitch.

      there a thousands of wires, with hundreds of connectors, each connector a cannon plug consisting of several dozen pins, any one of which could have gotten slightly bent (or even broken) upon reconnection, making an imperfect electrical connection or faulty data bus signal (depending on system). Or a wire may fall out of the backend of the pin from a faulty installation of the retainer of the cannon plug. or the plug itself may be not quite fully seated; you'd think it would be easy, but there's a reason we have cannon plug pliers (aka "bi*ch grips"). There's also millions of solder joints and splices that can fatigue from vibration. sometimes a contact simply gets dirty cause oil or grease (we wipe everything constantly, but still happens).

      (*speaking of PITA to maintain: due its nature, being tied into EVERYTHING (hundreds, sometimes thousands, of feet of wiring, depending on aircraft type), the Master Caution Panel (MCP) itself is often the actual point of failure, throwing false indications. one of the first things we frequently did in tracing a gripe was to first eliminate the MCP itself)

    • We were playing Artemis (a starship bridge simulation game on multiple computers), and the captain took us pretty close to one of those devices that eats everything in its path, just to show it to some of the newer players. At that time, the Helm computer crashed, and we couldn't move until it rebooted and reconnected. Fortunately, Engineering was able to divert a lot of power to the forward shields, so we lasted long enough.

  • by wonkey_monkey ( 2592601 ) on Friday March 11, 2016 @04:31AM (#51676251) Homepage

    Software Bug in F-35 Radar Causes Mid-Flight System Reboot

    Alarmist headline.

    First of all, the bug doesn't cause a reboot. It requires a reboot to put the radar back into a useable state.

    Secondly, it is only the radar system that needs rebooting.

    • by rikkards ( 98006 ) on Friday March 11, 2016 @06:27AM (#51676433) Journal

      Third it's already been resolved. My dad worked on the Canadian Maritime Helicopter Project for years. During it the papers had an article talking about some snag that was hit and how horrible it was. I sent my dad the link and his response:
      "Yeah that was my thing and 8 months ago, it was pointed out and in less than a day resolved."

      Media exaggerating? Never...

      • Media exaggerating? Never...

        That said, they are in good company. We're talking about an aircraft that is severely delayed, underperforms compared to spec and with a programme cost of twice what was originally planned.

        Now, that things are going to take longer, perform less than expected and be more expensive than anticipated is almost a given in these projects, but the F-35 breaks a couple of records even given that. They don't call it "The plane that ate the Pentagon" for nothing.

        So who's exaggerating more? My money isn't on the news

      • by umghhh ( 965931 )

        Which one of the resolved versions was that resolved you dad did:

        1. solution found
        2. solution found and coded in
        3. solution found, coded in, and a patch released
        4. solution found, coded in , verified on both maintenance and design branches, patch released, main released scheduled
        5. solution found verified, new package released and deployed?
        6. they actually use it in the field, all instances of the faulty code in the field have been patched and the fault does not reoccur.

        feel free to add other definitions of resolved that you

  • by DrXym ( 126579 ) on Friday March 11, 2016 @04:42AM (#51676269)
    I'm sure it's a very serious bug but does it mean that the software is "riddled" with bugs? For all anyone knows it was an isolated issue that occurred in an atypical circumstance and was subsequently rectified. And it occurred during testing which is the reason that testing even exists as a thing - to find problems.
    • by hey! ( 33014 )

      Well, to a programmer a "bug" is an error in logic. To a user a "bug" is an error in results. A single result may stem from many errors, particularly if we're talking about some kind of resource starvation, which this sounds like. If the starvation was caused by a single line memory leak, well that wouldn't take that long to find and fix. But if it were a faulty approach to managing memory used extensively over several years of development, you'd have a substantial body of logic errors that would take so

  • Didn't we have this same headline for the F-22, back in the day?

    • Didn't we have this same headline for the F-22, back in the day?

      Dunno but didn't that have a thing where it cutoff the oxygen supply to the pilot?

  • by Snufu ( 1049644 ) on Friday March 11, 2016 @05:02AM (#51676297)

    "Yaeh, my jet is plummeting to earth at mach 3. Any suggestions?"

    "Have you tried turning off and on again?"

  • by Daniel Matthews ( 4112743 ) on Friday March 11, 2016 @05:35AM (#51676343)
    A solution would be new code. It sounds like the test pilots are doing a great job of you know, testing.
  • completely normal (Score:3, Interesting)

    by Matthew Anderson ( 4286909 ) on Friday March 11, 2016 @06:13AM (#51676401)
    This happens literally all the time with software updates on jets, anyone who's worked on any other generation fighter/attack aircraft in the "digital age" knows this. The interesting part, is that someone is publicly complaining about it, and making a software version with a bug, known to the public. Every radar system we've produced for 30 years has issues, again, this is completely normal. This article is about sounding some sort of political alarm, it shows that there's some dissent among the ranks, and I can assure you, that any experienced test pilot wouldn't even be remotely surprised to see this type of behavior, but the usual course of action is to document it and train the pilots in the short term, while releasing a new version of software in the long term. Pilots fly with "radar degrade" every single day. This is making a standard issue between contractors and military flight crews, into a public pentagon issue, to either ask congress for more money, rather than holding said contractors accountable for their failure to meet some sort of design goal, or to try to join the "anti F-35" team to advance his own career in some way. Our jets are flying way beyond their designed limitations right now, and the longer it takes to replace them, the worse off all of our military personnel are going to be.
    • If the F35 was just a test plane then it wouldn't be such an issue but they are already in production. It's the new model for the military industrial complex. Spread the work out across the whole country and get it into production as early as possible to make it as politically tricky to cancel and keep as much money coming in as possible.

      The issue with the F35 isn't that these issues are happening with test pilots who are highly trained to expect anything to happen at anytime. The problem is that planes th

  • by ma++i+ude ( 580592 ) on Friday March 11, 2016 @06:25AM (#51676429) Homepage

    The software version "3i" is affected.

    As a general rule, when your version numbering system needs to use complex numbers, something's going wrong with your project.

    • by McWilde ( 643703 )
      To be fair, it's not complex, it's imaginary.
      • To be fair, it's not complex, it's imaginary.

        If you want to be a smartarse about it, get it right. Imaginary numbers are complex numbers whose real part is zero.

    • If you could actually get people to use it both honestly and correctly(which is a big if, I know) complex numbers would actually make for a great version numbering system: You set the imaginary term to reflect the scope of the project, then bump the real term and (hopefully) reduce the imaginary term as subsequent releases implement the various features intended. When the version number has only a real component, you know you are finished. To account for real-world scenarios, version number updates can incr
  • "Th' more they over th' plumbin' th easier it is to stop up th' drain."
    -Montgomery Scott ST3

  • Really? Were Bonnie and Clyde "riddled with a bullet hole"? Come on, editors!

  • If we'd bought them from Verizon it could be years before we get upgraded to the fixed software version. Or we might just have to by the F-36 to get the update.

  • From the Future News desk:

    {Major Autonomous Car Manufacturer} announced today the discovery of a bug in their control software that prompts a reboot of the cars' systems in mid-drive. "driving stability - the cars' ability to stay up and running. [...] What would happen is they'd get a signal that says either a driving degrade or a driving fail - "something that would force us to restart the cars' autonomous control systems."

    This reporter notes that if autonomous cars had a full set of manual controls, instead of just a big red 'STOP' button on the dash, human occupants of the vehicle would have an opportunity to save themselves from a firey death, instead of the surviving family members merely receiving an insurance payout.

  • It'll be totally fixed in the F-36 and it'll also include cool sound effects when the radar target windows minimize and maximize!

  • Get it? Over the Air update? new meaning to the word software crash.

  • is no longer just a figure of speech.

  • This kind of issue is common in the software industry these days. Delivery dates are set way before the engineering team has a chance to make an educated estimate and if there is a time crunch then testing time is cut. Reason: cannot cut development time because you cannot ship what does not exist. Result: software is released that is not properly tested. The flaws are detected during later stages of testing or once in full production use. Conclusion: want reliable systems and software? Then cut features, n

Beware of Programmers who carry screwdrivers. -- Leonard Brandwein

Working...